Sockets opened outside profiling do not close properly inside profiling

[altendky]

Creating and binding a socket outside of a call to memory_profiler.memory_usage() causes it to not close properly inside the call. After calling .close() the port is still claimed and a new socket is unable to be bound to the same port. Without memory_profiler in the middle it works. I haven't gotten to digging into this yet but will share anything I find.

Expand the collapsed sections at the bottom for a full set of commands to recreate and a full terminal session.

For linkage, I initially reported this at CFMTech/pytest-monitor#53.

$ venv/bin/python x.py
Traceback (most recent call last):
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 29, in <module>
    main()
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 18, in main
    memory_profiler.memory_usage((profiled, [sock3]))
  File "/home/altendky/repos/chia-blockchain/tmp/venv/lib/python3.9/site-packages/memory_profiler.py", line 377, in memory_usage
    returned = f(*args, **kw)
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 25, in profiled
    sock2.bind(address)
OSError: [Errno 98] Address already in use

import socket

import memory_profiler


address = ("127.0.0.1", 33125)


def main():
    sock = socket.socket()
    sock.bind(address)

    profiled(sock)

    sock3 = socket.socket()
    sock3.bind(address)

    memory_profiler.memory_usage((profiled, [sock3]))


def profiled(sock):
    sock.close()

    sock2 = socket.socket()
    sock2.bind(address)
    sock2.close()


main()

commands

cat > x.py << EOF
import socket

import memory_profiler


address = ("127.0.0.1", 33125)


def main():
    sock = socket.socket()
    sock.bind(address)

    profiled(sock)

    sock3 = socket.socket()
    sock3.bind(address)

    memory_profiler.memory_usage((profiled, [sock3]))


def profiled(sock):
    sock.close()

    sock2 = socket.socket()
    sock2.bind(address)
    sock2.close()


main()
EOF
cat x.py
python3.9 -m venv venv
venv/bin/python -m pip install --upgrade pip setuptools wheel
venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
venv/bin/python x.py
venv/bin/python --version --version
venv/bin/pip freeze
uname -a
lsb_release -a

full console session

$ cat > x.py << EOF
> import socket
> 
> import memory_profiler
> 
> 
> address = ("127.0.0.1", 33125)
> 
> 
> def main():
>     sock = socket.socket()
>     sock.bind(address)
> 
>     profiled(sock)
> 
>     sock3 = socket.socket()
>     sock3.bind(address)
> 
>     memory_profiler.memory_usage((profiled, [sock3]))
> 
> 
> def profiled(sock):
>     sock.close()
> 
>     sock2 = socket.socket()
>     sock2.bind(address)
>     sock2.close()
> 
> 
> main()
> EOF

$ cat x.py
import socket

import memory_profiler


address = ("127.0.0.1", 33125)


def main():
    sock = socket.socket()
    sock.bind(address)

    profiled(sock)

    sock3 = socket.socket()
    sock3.bind(address)

    memory_profiler.memory_usage((profiled, [sock3]))


def profiled(sock):
    sock.close()

    sock2 = socket.socket()
    sock2.bind(address)
    sock2.close()


main()

$ python3.9 -m venv venv

$ venv/bin/python -m pip install --upgrade pip setuptools wheel
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (56.0.0)
Collecting setuptools
  Using cached setuptools-60.1.0-py3-none-any.whl (952 kB)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-21.3.1 setuptools-60.1.0 wheel-0.37.1

$ venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
Collecting memory-profiler==0.60.0
  Using cached memory_profiler-0.60.0-py3-none-any.whl
Collecting psutil==5.8.0
  Using cached psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB)
Installing collected packages: psutil, memory-profiler
Successfully installed memory-profiler-0.60.0 psutil-5.8.0

$ venv/bin/python x.py
Traceback (most recent call last):
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 29, in <module>
    main()
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 18, in main
    memory_profiler.memory_usage((profiled, [sock3]))
  File "/home/altendky/repos/chia-blockchain/tmp/venv/lib/python3.9/site-packages/memory_profiler.py", line 377, in memory_usage
    returned = f(*args, **kw)
  File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 25, in profiled
    sock2.bind(address)
OSError: [Errno 98] Address already in use

$ venv/bin/python --version --version
Python 3.9.5 (default, Jun  3 2021, 15:18:23) 
[GCC 9.3.0]

$ venv/bin/pip freeze
memory-profiler==0.60.0
psutil==5.8.0

$ uname -a
Linux p1 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

[altendky]

Mmm... multiprocessing is being used with the default start method of fork with which "All resources of the parent are inherited by the child process". Explicitly specifying spawn instead with multiprocessing.set_start_method("spawn") works. spawn is described in part with "unnecessary file descriptors and handles from the parent process will not be inherited".

new commands that work

cat > x.py << EOF
import multiprocessing
import socket

import memory_profiler


address = ("127.0.0.1", 33125)


def main():
    multiprocessing.set_start_method("spawn")

    sock = socket.socket()
    sock.bind(address)

    profiled(sock)

    sock3 = socket.socket()
    sock3.bind(address)

    memory_profiler.memory_usage((profiled, [sock3]))
    print("really made it")


def profiled(sock):
    sock.close()

    sock2 = socket.socket()
    sock2.bind(address)
    sock2.close()


# auaughhghgh  multiprocessing!
if __name__ == "__main__":
    main()
EOF
cat x.py
python3.9 -m venv venv
venv/bin/python -m pip install --upgrade pip setuptools wheel
venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
venv/bin/python x.py
venv/bin/python --version --version
venv/bin/pip freeze
uname -a
lsb_release -a

new working terminal session

$ cat > x.py << EOF
> import multiprocessing
> import socket
> 
> import memory_profiler
> 
> 
> address = ("127.0.0.1", 33125)
> 
> 
> def main():
>     multiprocessing.set_start_method("spawn")
> 
>     sock = socket.socket()
>     sock.bind(address)
> 
>     profiled(sock)
> 
>     sock3 = socket.socket()
>     sock3.bind(address)
> 
>     memory_profiler.memory_usage((profiled, [sock3]))
>     print("really made it")
> 
> 
> def profiled(sock):
>     sock.close()
> 
>     sock2 = socket.socket()
>     sock2.bind(address)
>     sock2.close()
> 
> 
> # auaughhghgh  multiprocessing!
> if __name__ == "__main__":
>     main()
> EOF

$ cat x.py
import multiprocessing
import socket

import memory_profiler


address = ("127.0.0.1", 33125)


def main():
    multiprocessing.set_start_method("spawn")

    sock = socket.socket()
    sock.bind(address)

    profiled(sock)

    sock3 = socket.socket()
    sock3.bind(address)

    memory_profiler.memory_usage((profiled, [sock3]))
    print("really made it")


def profiled(sock):
    sock.close()

    sock2 = socket.socket()
    sock2.bind(address)
    sock2.close()


# auaughhghgh  multiprocessing!
if __name__ == "__main__":
    main()

$ python3.9 -m venv venv

$ venv/bin/python -m pip install --upgrade pip setuptools wheel
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (56.0.0)
Collecting setuptools
  Using cached setuptools-60.1.0-py3-none-any.whl (952 kB)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 56.0.0
    Uninstalling setuptools-56.0.0:
      Successfully uninstalled setuptools-56.0.0
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-21.3.1 setuptools-60.1.0 wheel-0.37.1

$ venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0
Collecting memory-profiler==0.60.0
  Using cached memory_profiler-0.60.0-py3-none-any.whl
Collecting psutil==5.8.0
  Using cached psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB)
Installing collected packages: psutil, memory-profiler
Successfully installed memory-profiler-0.60.0 psutil-5.8.0

$ venv/bin/python x.py
really made it

$ venv/bin/python --version --version
Python 3.9.5 (default, Jun  3 2021, 15:18:23) 
[GCC 9.3.0]

$ venv/bin/pip freeze
memory-profiler==0.60.0
psutil==5.8.0

$ uname -a
Linux p1 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

Ok, so with the cause identified, what next? I'm not exactly sure. Probably some documentation commenting on this with references to each of multiprocessing, spawn, fork, inherit-ed/ing, file descriptors, etc so it is findable by searching. I hesitate to mess with the multiprocessing global state and auto-configure this since code using memory-profiler may choose something else. If we do set the start method then it should only happen if not otherwise configured by the 'outer' code. I'll think it over and see what I can come up with.

[altendky]

Are multiprocessing features really being used? Or could we just use a regular subprocess instead?