UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2008-12-29 11:53 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Options for robust process cleanup?

Hi guys,

I've got a minor problem and I'm wondering if anybody has any good advice on how to solve it.

Currently, when one of my processes is launched, it access a shared memory segment and writes a little bit of data into the shared memory.  When the process exits, it clears that piece of data from the shared memory area.  At any time, other processes can examine that bit of shared memory and see (more or less) which processes are currently active.  This all uses semaphores for proper data synchronization, and more or less works fine.

The minor problem is this:  it is not always the case that my process exits cleanly.  In particular, it is possible for someone to SIGKILL my process, or for my process to crash before exiting (not that my code would ever do such a thing, ahem ;^)).  In these scenarios, the cleanup-the-shared-memory routine never runs, and my shared memory is not properly cleaned up... the data corresponding to my now-dead process is still there and it shouldn't be.

The question is, is there any good/reliable solution to this problem?  How have other people dealt with this sort of issue?  It seems like it must come up whenever processes are creating shared resources that need to be manually cleaned up when the processes are done using them.

Thanks,
Jeremy

Offline

#2 2008-12-30 03:04 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: Options for robust process cleanup?

The Linux way of doing it is to do a shmctl IPC_RMID on it after doing
the shmat, then it disappears when all users exit. I think the Linux specific
behaviour is that it allows shmat on segments marked for destruction.
But after doing that you can only attach using the shmid, and the key is
gone I think (just like unlinking a file and keeping the fd around).
This is the only reliable way as far as I know. Though having a cronjob
deleting all messy leftovers is reliable too I suppose.

Offline

#3 2008-12-30 05:16 AM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Options for robust process cleanup?

Yes, those things might work, although I don't want the shared memory area to go away, so much as I want the particular bytes in it to be reset to their default state.

(Actually what I really want is a way to specify a routine that the OS would run whenever the process dies... but I guess I'm not going to get that ;^))

Offline

#4 2008-12-30 09:39 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: Options for robust process cleanup?

Offline

#5 2008-12-31 02:36 PM

Clairvoyant1332
Member
Registered: 2005-07-05
Posts: 22
Website

Re: Options for robust process cleanup?

Offline

#6 2008-12-31 03:54 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Options for robust process cleanup?

Offline

#7 2008-12-31 05:03 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: Options for robust process cleanup?

The best way to deal with such a scenario (IMHO), and in a relatively portable manner,
is to just write a simple server that all your processes register with, rather than use
shared memory to store their info...  Then, all you need is to keep open the Unix
domain socket (or you can expand it to use inet sockets, if you want non-local clients
to register too) between client and server, and the server can easily detect the death
of the clients when the socket gets closed, and so know to remove them from its list
of active clients...  All you need is a simple protocol to allow clients to register or to
query the list of registered clients...  And, it can easily be expanded to handle the
storage of any data you like along with the registered client info...  And, it won't really
be any noticably slower than using shared memory, and will eliminate any need for
semaphores or other synchronization issues...  (Or, at least confine them to the internal
code of the server, if eg., you decide to make the server multi-threaded...)

And, if you want to worry about the case of the server process getting killed, well at
some point you have to just declare a sane minimum working environment, and I'd
say requiring that your server be running is a sane requirement for allowing your
clients to run...  But, if you really wanted to be overly helpful, you could have the
first client that fails to connect to the server try to automatically start it...

Offline

#8 2008-12-31 06:07 PM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Options for robust process cleanup?

Hi Rob,

Thanks for the suggestion; I considered a server process as a solution but it really does seem like overkill for what would otherwise be a rather simple system.

In the end I implemented this solution:  each process that writes a record into the shared memory region also writes its process ID into the shared memory region.  Also, each time a process locks the shared memory region, it does a 'ps -Ax' (or the Windows equivalent, EnumProcesses()) to find the list of currently running process IDs, then goes through the shared memory region and erases any entries that aren't in the current process-IDs list.  In that way, "orphaned records" are cleaned up by the next reader of the shared memory region, before any data is extracted from it.

Not terribly efficient, but it seems to work well.  I do wish there was a better (and portable) way under *nix to obtain the current list of process IDs than popen("ps -Ax") and grovelling through its output, though...

Offline

#9 2008-12-31 07:46 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: Options for robust process cleanup?

Offline

#10 2009-01-01 12:41 AM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Options for robust process cleanup?

Offline

#11 2009-01-01 03:09 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: Options for robust process cleanup?

At least Linux tries to avoid reusing old pids, until it wraps around.
Doing otherwise would create too many possibilities for pid races.

Dare I say it? Yeah, why not:

If a "little data" is only a few bytes, you can stuff it in a semaphore. You can
use more than one semaphore if you want to store more than two bytes. Let
processes set it with SEM_UNDO and you'll have the behaviour you want (and
probably one of the most "interesting" semaphore usages). The trickiest part is
to find a new, unused sem_num, but in the worst case you just use another
semaphore set to keep track of slot usage (to complete the madness).
Basically you just use the semaphore set for data storage and its automatic
cleanup property, ignore its semaphore properties, except for atomic updates.

Offline

#12 2009-01-01 07:25 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: Options for robust process cleanup?

Offline

#13 2009-01-02 02:00 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: Options for robust process cleanup?

In 2.6 kernels there's /proc/sys/kernel/pid_max which is by default 32768,
but can be set higher on 64 bits systems, up to 2^22. Dunno why it's limited
to 22 bits and only for 64 bits systems.

If wrap around happens it can cause problems, but at least you don't have
to worry about race conditions too much. They still can happen, but it's
very unlikely: basically only on purpose. ;-)

Offline

Board footer

Powered by FluxBB