UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

  • Index
  • » Threads
  • » TCP sockets as inter-thread signalling mechanism

#26 2008-05-19 05:41 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

i3839;24522 wrote:

If using Linux's epoll you can just add sockets to it from one thread while another
one waits on them. If you make all peer state reachable via epoll_data_t then it all
happens transparent. Something similar can probably be done with BSD's kqueue.

Oh-ho, even though this is getting platform-specific, I like it. :)

Granted, my code is written in Objective-C, so most portability is already out the window. In that case, I need to delve a little further into a kqueue implementation and see if it behaves how I think it will behave.

Thanks.

Offline

#27 2008-05-19 07:08 PM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: TCP sockets as inter-thread signalling mechanism

Oh my...

Again and in reference to the design outline in the first message of this thread... yes - it will work - but ( again ) there are a number of reasons it is probably not the best way to do things :

* you will be eating up to 200% ( more ) sockets / file descriptors i.e. in a regular system you would only have 1 socket for sending information down the wire... in the design you would need that 1 socket, 1 socket for each thread to receive notifications and 1 socket to send notifications...

* if the design is 1 ( thread ) - n ( sockets ) for the outside connections, I don't see how you will be saving much if anything in the arena of complexity... as you still need to have structures to track what thread(s) are handling data from what socket(s)

* if we are talking about running on a system with more than 1 core ( very common these days even at the desktop level ) - the design will actually INCREASE latency significantly...  I already pointed this out but IO ops e.g. selects, reads, writes, etc. are trigger points for the scheduler(s) to do context switches... so creating a system that requires ( any ) additional sockets and subsequently requires more dips into select, read and write by individual threads will hurt efficiency...

* the design is offloading to a sub-system - which due to that sub-systems configurability at the app and system ( config ) level which means you will either be exposing yourself to a lot of unknowns OR have to assure you app knows the lowest common denominator for configuration across all systems on which the code will be deployed and 'manually' sets that config at startup... much more problematic than using something like the uniform and standard pthread condition variable setup...

>> sigh <<

Oh well... on a positive note... stuff like this is how I get a lot of my contract work... lol


Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#28 2008-05-19 07:28 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

mlampkin;24525 wrote:

Oh my...

Again and in reference to the design outline in the first message of this thread... yes - it will work - but ( again ) there are a number of reasons it is probably not the best way to do things :

<snip for great justice>

Agreed, and I apologize for the thread hijack, but I was also a bit stuck trying to eliminate the one control socket pair in my own implementation of 1 I/O thread + x worker threads without having to introduce a timeout to the I/O thread and having it spin when in my case, it will be spinning and doing no real work over 90% of the time. That seems rather wasteful.

i3839 pointed me in the right direction to let me drop the socket pair out of my implementation, if kqueues work the way the man page states anyways.

Offline

#29 2008-05-19 07:36 PM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: TCP sockets as inter-thread signalling mechanism

Oops... I didn't realize your message was 'self contained'... my bad... :)


Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#30 2008-05-19 07:57 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

mlampkin;24528 wrote:

Oops... I didn't realize your message was 'self contained'... my bad... :)


Michael

No complaint from me, I have reimplemented my own socket code over the past couple of days because of the comments such as yours on this thread. Originally I took the route of X worker/select threads, and after some of your comments, I took a look and reimplemented it as 1 select thread + X worker threads as I liked the design better than my original one (since the I/O thread isn't waiting to re-enter select because I am parsing or whatever).

Now, my big complaint with select() is that if you want to have another thread update the fd sets, you can't without a timeout or another descriptor/signal that kicks the thread out of the select() call. I kinda wish epoll or kqueue was in POSIX. :)

Offline

#31 2008-05-19 08:12 PM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: TCP sockets as inter-thread signalling mechanism

Yeah - that is one catch using the select setup... having to force an EINTR / similar against the select to get it to pop back out when you have data to send... and trying to 'cluster' the adds so you didn't interrupt select too much...

Btw - they did add the aio_ stuff in POSIX... and it provides the same sort of functionality...

The caveat being some impls ( Linux - though this may have changed in new releases ) it actually caused a new thread to be created at the kernel level each time a socket was added... so while faster than a regular select or poll - it did still have some extraneous overhead...


Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#32 2008-05-19 10:15 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

Hmm, I may not even have the aio_ functions on my platform. I can't tell.

Either way, kqueue seems to fit my needs a bit better. Data is not constantly streaming back and forth on each socket (rather, it is sent in small 1KB bursts every so often, anywhere from 1 second to minutes between packets, since it is based on the telnet protocol), and so when the connection count gets high, iterating over an event list and using a hash map to get connection state objects for each event will probably win out as being more efficient.

Offline

#33 2008-05-20 05:35 AM

mlampkin
Administrator
From: Sol 3
Registered: 2002-06-12
Posts: 911
Website

Re: TCP sockets as inter-thread signalling mechanism

Ah... was kinda flashing over the kqueue bit... so a BSD platform I assume?

If thats the case... then yeah - aio_ is not available except as an add-on ( module )... and as I understand it there may be / have been problems with it...

Btw - and not quite off topic...

Have you ever looked at the thttpd source code ?  Its a single threaded model BUT if you are looking at alternative to select(...) and cross compatability in general... it provides a great minimalist view of ( some of ) the various alts across platforms...

It may not help you in your particular project... but still might be worth a peek...


Michael


"The only difference between me and a madman is that I'm not mad."

Salvador Dali (1904-1989)

Offline

#34 2008-05-20 05:43 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

mlampkin;24535 wrote:

Ah... was kinda flashing over the kqueue bit... so a BSD platform I assume?

If thats the case... then yeah - aio_ is not available except as an add-on ( module )... and as I understand it there may be / have been problems with it...

OS X, which for all intents and purposes acts very similar to FreeBSD 4.2 or thereabouts. Seems it does have aio_ in OS X 10.5 (possibly earlier), but the sort of design it wants you to use is kinda hard to wrap my head around. :)

mlampkin;24535 wrote:


Btw - and not quite off topic...

Have you ever looked at the thttpd source code ?  Its a single threaded model BUT if you are looking at alternative to select(...) and cross compatability in general... it provides a great minimalist view of ( some of ) the various alts across platforms...

It may not help you in your particular project... but still might be worth a peek...

Michael

I'll check it out, but as this in Objective-C, my desire for portable code isn't gonna get me very far unless I start rewriting parts of it in C++. :)

Offline

#35 2008-05-20 10:29 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,203

Re: TCP sockets as inter-thread signalling mechanism

Objective-C works fine on other platforms too, only problem could be that you're using Apple specific libraries.

It should be easy to adapt C or C++ code to Objective-C too.

Offline

#36 2008-05-20 10:54 PM

Kolenka
Member
Registered: 2008-05-15
Posts: 9

Re: TCP sockets as inter-thread signalling mechanism

i3839;24541 wrote:

Objective-C works fine on other platforms too, only problem could be that you're using Apple specific libraries.

Oh, I don't doubt this will run in GNUStep (eventually, when all the bits needed for it to support Obj-C 2.0 fall into place). I am one of those strange people who constantly thinks about how portable some design is (even commenting out loud when I don't need to), even if I will never actually port the thing.

Right now this is just a server in the same vein as MU*s, and when/if I get time, I'll start on a graphical client that runs on a couple platforms. But the server need not be portable.

There would be some work I need to do (mostly around the SIMD code, since I leverage Apple's vector variable types that hide the Altivec and SSE types), but yeah, I don't think it would be hard, just time consuming. :)

Offline

#37 2008-05-21 01:23 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,203

Re: TCP sockets as inter-thread signalling mechanism

Then looking at thttpd source code is even better cause that one is actually ported to multiple platforms, e.g. using epoll on linux and kqueue on BSD.

Offline

  • Index
  • » Threads
  • » TCP sockets as inter-thread signalling mechanism

Board footer

Powered by FluxBB