[GENERAL] connection file descriptors created with identical number after process fork on mac

[Chris Withers]

Hi All,

I'm writing some multi-process code in Python and trying to make sure I
open a new connection for each process. Here's the really cut down code:

"""
import os, time
import psycopg2
from multiprocessing import Pool


def init():
conn = psycopg2.connect("dbname=...host=localhost")
print os.getpid(), ' child fd:', conn.fileno()

if __name__=='__main__':
pool = Pool(initializer=init)
time.sleep(30)
"""

What's really surpising to me is the output on a mac:

$ python psycopg2_multiprocess.py
44276 child fd: 13
44277 child fd: 13
44278 child fd: 13
44279 child fd: 13

The getpid() output indicates that the connec() call is being made
inside a different process each time, yet the connection appears to
still be using the same fd.

conn.file() is basically (long int)PQsocket(self->pgconn);:
https://github.com/psycopg/psycopg2/blob/master/psycopg/connection_type.c#L898

Is there something I'm missing about file descriptors on Macs or is
something bad happening here?

Chris

[Tom Lane]

Post by Chris Withers
I'm writing some multi-process code in Python and trying to make sure I
...
$ python psycopg2_multiprocess.py
44276 child fd: 13
44277 child fd: 13
44278 child fd: 13
44279 child fd: 13
The getpid() output indicates that the connec() call is being made
inside a different process each time, yet the connection appears to
still be using the same fd.

FD numbers are process-local in all flavors of Unix. The above only
proves that all of these processes had FDs 0..12 open already, which
doesn't seem terribly surprising.

regards, tom lane

[Chris Withers]

Post by Tom Lane

Post by Chris Withers
I'm writing some multi-process code in Python and trying to make sure I
...
$ python psycopg2_multiprocess.py
44276 child fd: 13
44277 child fd: 13
44278 child fd: 13
44279 child fd: 13
The getpid() output indicates that the connec() call is being made
inside a different process each time, yet the connection appears to
still be using the same fd.

FD numbers are process-local in all flavors of Unix. The above only
proves that all of these processes had FDs 0..12 open already, which
doesn't seem terribly surprising.

Thanks, that's certainly good news!

How can I convince myself, from the client side, that I really have got
a new connection and not somehow ended up with one that been passed on
as part of the fork?

cheers,

Chris

[John R Pierce]

Post by Chris Withers
How can I convince myself, from the client side, that I really have
got a new connection and not somehow ended up with one that been
passed on as part of the fork?

$ psql -tc "select pg_backend_pid();"
18635

$ psql -tc "select pg_backend_pid();"
18665

$ psql -tc "select pg_backend_pid();"
18727


now, operating system process ID's do get recycled eventually but not
very fast.