UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2008-10-28 02:13 AM

ryujin
Member
Registered: 2008-10-28
Posts: 12

Re: epoll usage questions

I have read the in Linux epoll is the way to go for scalable concurrent server implementation and I have succesfully implemented a sample program butI still have some small questions on epoll usage I hope someone here can answer.

1. Questions about epoll_wait():

1.1 Does maxevents parameter determine the maximun
      number of concurrent clients?? If I want to handle
      200,000 clients at any given time do I need to set this
       variable to 200000??

1.2 Do I have to allocate the memory area pointed by
      the *events argument. I tried with and without
      allocating it and so far it works with both cases
      but I have no definitive answer to this. If I do have
      to allocate it do I need to allocate enough memory
      to store maxevents*sizeof(struct epoll_event)??
      In my case with 200000 concurrent clients would
      this be overkill??

2. Question about timeouts
   
     I can set a timeout in epoll_wait but I can not find any use
     for this. I register 200,000 client sockets to epoll file descriptor
     but I cannot timeout each socket individually. And epoll_wait
     returns zero so there is no list of timed out sockets I can use
     to remove timed out sockets.

     2.1 How exactly should the epoll_wait timeout parameter be used?

     2.2 How do I close sockets that have been idle for some time?

any answers/tips/resources are welcome...

Horacio

Offline

#2 2008-10-28 07:47 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: epoll usage questions

Offline

#3 2008-10-28 08:30 AM

ryujin
Member
Registered: 2008-10-28
Posts: 12

Re: epoll usage questions

Very complete answer, thanks a lot...  now I understand a little better epoll.

This should be included in the epoll man page as this is not explained there.

Horacio

Offline

#4 2008-10-28 12:25 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: epoll usage questions

Also, on the idle sockets concept, let me suggest you setsockopt(SO_KEEPALIVE),
and since you're on Linux you can also set per-socket keep-alive timeouts via
TCP_KEEP{CNT,IDLE,INTVL} sockopts...  (See "man 7 socket" and "man 7 tcp"...)
This way, the kernel will keep traffic flowing on idle connections, allowing you to
quickly detect connections that have silently gone dead somehow...

Offline

#5 2008-10-29 12:50 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: epoll usage questions

Yeah, if you only want to detect lost connections quicker, use those options Rob
mentioned. Try to avoid per socket timers if you can, they're slightly messy.

Offline

#6 2008-10-29 04:13 AM

ryujin
Member
Registered: 2008-10-28
Posts: 12

Re: epoll usage questions

Thanks for the tip, I set the KEEP_ALIVE option to all my sockets but really did not know why, just copied from a book.

I understand this socket option works in Linux but what about BSD (Mac, FreeBSD, etc..) and what about Windows??  are there corresponding mechanisms in these platforms to detect down clients?


regards,
Horacio

Offline

#7 2008-10-29 06:54 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: epoll usage questions

No idea...

Offline

#8 2008-10-29 01:13 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: epoll usage questions

Well, SO_KEEPALIVE is pretty much universal, and I'd expect every system with any
kind of sockets implementation to support it...  However, what is NOT universal is the
per-socket keep-alive timeouts...  On most systems, SO_KEEPALIVE's timeouts are
fixed or only configurable on a system-wide basis where everything is affected...
Which tends to make it a lot less useful...  For one thing, the standard default timeout
is outrageously long (generally 2 hours of idle time before it even starts to probe)...
And, most apps will generally want to detect problems FAR sooner than that...  So, if
your system supports any kind of configurable keep-alive timeouts (preferably
per-socket like Linux, but even system-wide is better than nothing), I highly recommend
setting them to something saner than the defaults...

Offline

#9 2008-10-31 06:11 AM

ryujin
Member
Registered: 2008-10-28
Posts: 12

Re: epoll usage questions

while(loop->fd > 0) {
    nfds = epoll_wait(loop->fd,loop->events, loop->count, loop->timeout);

    if(nfds == 0) {
        printf("suj_evloop_start(): epoll_wait timeout\n");
        continue;
     }

     for(i = 0; i < nfds; i++) {
        event = (suj_event *)loop->events[i].data.ptr;
        event->callback(event);
    }
}

Offline

#10 2008-10-31 01:52 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: epoll usage questions

Offline

#11 2008-11-02 04:01 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: epoll usage questions

Preventing DoS is quite difficult. But as Rob pointed out, adding a timeout
doesn't fix the problem. If they can make 50K connections, they can also
keep 50K connections active.

What would make it harder to DoS your server is limiting the number
of connections per host. To do that you need to keep track all peer's
addresses and check the count when a new connections comes in.

A quick and easy way to do that would be a static hashtable which is big
enough, and just ignore hash collitions. E.g. a 4K array of connection count
numbers. If you keep the limit below 255 connections per address you can
make it a char array (meaning you can make the hashtable big without wasting
much memory). Store the hashvalue in suj_event or whatever per socket
structure you have, and decerement the count when the socket is closed.

Just use a doubly (circular, embedded) linked list to link all your suj_events
together,  with the listen socket as the head node, and you can always find
all other suj_events from one of them and easily remove and add nodes
withoutmuch trouble. Making the listen socket structure statically allocated
means you don't have to keep a "head" pointer up to date, but still can
find all nodes.

---

The code you posted won't really work. That because you need to check
each N seconds, not only when epoll_wait timed-out. This because the
sockets are idependent, and all of them can be active, except for one.
In that case epoll_wait will never time out. So you'd want to check the
time and compare it to the last time you did a sweep. Other than that
the code is in the right direction.

Offline

#12 2008-11-02 05:37 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: epoll usage questions

Offline

#13 2008-11-03 03:22 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: epoll usage questions

True, but if they throw a distributed DoS at you, it are the serious guys.
Without anything a slightly annoyed person could DoS your server with
a laptop. ;-)

Offline

#14 2013-06-19 08:40 PM

ankur
Guest

Re: epoll usage questions

Working with epoll, I have made a multi client server it will accept the connection of new clients, read the data from client and write some configuration data to.

I have the following issues:

Working with read operation for the clients i am able to get only one read notification. If client further send data to server (i mean second or third time) i did not get any notification of EPOLLIN and my server waiting in epoll_wait call. Secondly I have to send data to client asynchronously for that I have to generate the EPOLLOUT event but i ab not able to generate that event as my server is waiting for karnal notification in epoll_wait process.

#15 2013-06-20 10:54 AM

Nope
Administrator
From: Germany
Registered: 2004-01-24
Posts: 385
Website

Re: epoll usage questions

We'd need a bit more info. Perhaps you could post some code snippets of your epoll usage?

But perhaps it would already be enough to change the epoll settings to use level instead of edge triggered behaviour....

Offline

#16 2013-06-20 12:48 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: epoll usage questions

Yeah, it sounds like you might be using EPOLLET...  Don't do that unless you're prepared to fully clear out the socket receive queue before ever calling epoll_wait() again...

Alternatively, if you're not using EPOLLET, then you may be reading more than you intended to in a single read()/recv() call...  I don't know your network protocol, but I've seen lots of people make the big mistake of assuming multiple write()/send() calls on one end would translate into the same number of read()/recv() calls on the other end...  TCP doesn't work that way!  It's a stream protocol, not a datagram one...  If you want to send discrete messages within the stream, it's up to your protocol to separate them somehow...  The most common methods are either a delimiter/terminator character/string at the end of each message, or a fixed-length header at the front of each message which gives the full length of the message...  But, if you just rely on trying to read()/recv() into a big buffer, you're quite likely to read multiple separate messages from the other side at once...  (Or, only part of a message, or some number of whole messages and one partial at the end...)

As for EPOLLOUT, are you specifying that as one of the events to watch for in the epoll_ctl() call for the socket?  If so, it may also be related to use of EPOLLET, if you're using it...  You'll only get a notification once when it changes state: from non-writable (ie: send queue full) to writable...  If it stays writable, you won't get notified again, if you're using edge-triggered semantics...  Drop the EPOLLET, and you'll basically constantly get barraged with EPOLLOUT events, since sockets are almost always writable, except in the rare case when their send queue fills up...  So, in general, I don't consider testing for writability like that to be very useful...  It's basically only of use for testing right up-front before you try to write, or when coupled with non-blocking I/O for waiting for it to become writable after you've received an EAGAIN/EWOULDBLOCK failure...  But, if you constantly test for it in the same epoll_wait() where you test for readability, you're basically going to be constantly and immediately returning from that epoll_wait(), because it's pretty much always going to be writable...

Offline

Board footer

Powered by FluxBB