UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2014-08-09 08:24 AM

tahmoures
Member
Registered: 2014-08-09
Posts: 6

Raw socket sniffer miss packets

Hi all,

I write a sniffer to sniff data from LAN through raw socket in Linux.
As I ran wireshork and my code simultaneously I found out that there some packets that my program miss to receive and also there are some other packets that I received in my program but the wireshark miss them.
The data rate is about 60 Mbps.

Do you have any idea about it?

Thanks.

Last edited by tahmoures (2014-08-09 08:25 AM)

Offline

#2 2014-08-09 03:17 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Raw socket sniffer miss packets

Are you really using a "raw socket" (AF_INET, SOCK_RAW)?  Or, is it actually a packet socket (AF_PACKET)?  Wireshark will be using libpcap which will be using a packet socket on Linux...  Those will see a lot more traffic than a plain old raw IP socket will...

Assuming you're really using a packet socket as well, the issue might just be that the traffic flow is too much and packets are getting dropped before they can be read by one or the other program...  With you both competing to read the same packets at the same time, one of you is going to end up sleeping while the other runs and reads the packets; by the time the sleeper's next timeslice comes around, its receive buffer may have been filled, in which case some packets will get dropped...  You might try increasing SO_RCVBUF...

Or, is either program using a BPF filter to weed out some types of packets?  If you're only interested in seeing certain kinds, setting a filter is probably a very good thing and will help with the overflow situation by keeping your traffic down to only what you care about seeing...

Offline

#3 2014-08-10 05:16 AM

tahmoures
Member
Registered: 2014-08-09
Posts: 6

Re: Raw socket sniffer miss packets

RobSeace wrote:

Are you really using a "raw socket" (AF_INET, SOCK_RAW)?  Or, is it actually a packet socket (AF_PACKET)?  Wireshark will be using libpcap which will be using a packet socket on Linux...  Those will see a lot more traffic than a plain old raw IP socket will...

Assuming you're really using a packet socket as well, the issue might just be that the traffic flow is too much and packets are getting dropped before they can be read by one or the other program...  With you both competing to read the same packets at the same time, one of you is going to end up sleeping while the other runs and reads the packets; by the time the sleeper's next timeslice comes around, its receive buffer may have been filled, in which case some packets will get dropped...  You might try increasing SO_RCVBUF...

Or, is either program using a BPF filter to weed out some types of packets?  If you're only interested in seeing certain kinds, setting a filter is probably a very good thing and will help with the overflow situation by keeping your traffic down to only what you care about seeing...


Thanks for your response,


Actually I use (AF_PACKET). about the connection I should say that it is a point to point connection and I also filter the MAC address in order to bypass the arp packets.

About the SO_RCVBUF, due to the fact that the max size of the ethernet packet is about 1500 byte, dose it make difference  to increase it?

According to the point that you mentioned that wireshark and my program compete with each other, I say the data calssified in 3 group;
1. the data that both the wireshark and my program receivce
2. the data that my program receive and wireshark miss
3. the data that wireshark receive and my program miss.

Offline

#4 2014-08-10 02:38 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Raw socket sniffer miss packets

About the SO_RCVBUF, due to the fact that the max size of the ethernet packet is about 1500 byte, dose it make difference  to increase it?

It might...  The OS will buffer up more than just one packet at a time in your receive queue!  If you give it more space, it'll be able to buffer more packets...  But, if the traffic flow is fast enough, you may still end up filling the receive buffer and dropping some packets, especially if you run two competing sniffers at the same time on the same machine...

Are you doing any significant processing for each incoming packet, or just displaying info about it then immediately returning to try to read another packet?  If doing anything that takes a lot of time, you probably want to hand that off to another thread/process to do, so you can return to reading more packets ASAP...  But, if you're just doing a simple display of the packet, it probably won't buy you much, if anything, to hand it off like that...

Also, if your version of Linux is new enough to have recvmmsg() (note the 2 "m"s!), you might benefit from using that to read multiple packets at once in a single syscall...

Offline

#5 2014-08-11 09:29 AM

tahmoures
Member
Registered: 2014-08-09
Posts: 6

Re: Raw socket sniffer miss packets

RobSeace wrote:

About the SO_RCVBUF, due to the fact that the max size of the ethernet packet is about 1500 byte, dose it make difference  to increase it?

It might...  The OS will buffer up more than just one packet at a time in your receive queue!  If you give it more space, it'll be able to buffer more packets...  But, if the traffic flow is fast enough, you may still end up filling the receive buffer and dropping some packets, especially if you run two competing sniffers at the same time on the same machine...

Are you doing any significant processing for each incoming packet, or just displaying info about it then immediately returning to try to read another packet?  If doing anything that takes a lot of time, you probably want to hand that off to another thread/process to do, so you can return to reading more packets ASAP...  But, if you're just doing a simple display of the packet, it probably won't buy you much, if anything, to hand it off like that...

Also, if your version of Linux is new enough to have recvmmsg() (note the 2 "m"s!), you might benefit from using that to read multiple packets at once in a single syscall...

I increased the but it makes no difference.

I actually do nothing about the received data, just try to display them.
I also test the same thing with 2 computer which connected point to point with udp socket and once again I missed data at the rate 40Mbps.

Offline

#6 2014-08-11 12:42 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Raw socket sniffer miss packets

It may just be you can't keep up with traffic flow of that speed...  If my quick rough calculations are correct, even assuming full MTU sized packets, 40Mbps would mean around 3500 packets per second...  Meaning your sniffer needs to read and display 3500 packets every single second...  If it fails to keep up with that rate, it will eventually fill up its receive queue and packets will get dropped...

Other stuff you can try: raise your priority (ie: lower your nice value)...  Maybe even sched_setscheduler(SCHED_FIFO), or similar...  Also, if you have that previously mentioned recvmmsg(), it may very well help you read the incoming packets faster...  But, ultimately if you really can't stand missing any packets, it may be necessary to split things up as previously mentioned, and have one dedicated thread/process that JUST reads the packets as fast as possible (and runs at a high priority), adding them to some sort of in-memory buffer/pool/list, which another thread/process (running at much lower priority) reads the packets from and displays them...  Ideally, you'd want a totally lockless design so that the packet reader can just throw things onto the list of packets without any delays and get back to reading more packets...

Offline

#7 2014-08-12 07:21 AM

tahmoures
Member
Registered: 2014-08-09
Posts: 6

Re: Raw socket sniffer miss packets

Thanks so much.

One other things that I already have problem with is that I need to reach 700 Mega bit per second rate through UDP in the Gigabit mode. But unfortunately now I can just reach to 190 Mega bit per second.
What should I do about this?

Thanks,

Offline

#8 2014-08-12 12:43 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Raw socket sniffer miss packets

One other things that I already have problem with is that I need to reach 700 Mega bit per second rate through UDP in the Gigabit mode. But unfortunately now I can just reach to 190 Mega bit per second.
What should I do about this?

Is your problem at the writing side?  Ie: you just can't send the data fast enough?  Or, are you sending it, but it's getting dropped or not read fast enough on the receiving side?

It's my understanding that to get even close to the theoretical bandwidth with GigE, you need to use jumbo frames...  Ie: MTU settings much higher than 1500...  I believe 9000 is a popular jumbo frame setting...  But, it needs to be supported by both end machines and any switches you have in between...  And, you need good, short cable runs, with no interference nearby...

But, if your sending app is reading data off a hard drive to send, THAT might be your limiting factor, since most hard drives are going to be pretty slow in comparison...  Similarly, if the receiving app is writing data to a hard drive, it's going to be speed-limited by how fast it can write the data...

Offline

#9 2014-08-13 07:07 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Raw socket sniffer miss packets

I didn't know about recvmmsg() , thanks Rob! A bit embarrassing that they messed up the time-out handling though...

I'm pretty sure that you don't need jumbo frames to get close to theoretical speed on gigabit ethernet, just good hardware (NIC, CPU and switch) and good drivers. See e.g. https://www.myricom.com/solutions/10-gi … mance.html, where they get pretty much the same with or without jumbo frames.

Also, firewalls may slow down things too, so make sure they're disabled or only have a few simple rules.

At what packet size do you need to reach 700 Mbit/s?

Offline

#10 2014-08-13 07:45 AM

tahmoures
Member
Registered: 2014-08-09
Posts: 6

Re: Raw socket sniffer miss packets

RobSeace wrote:

One other things that I already have problem with is that I need to reach 700 Mega bit per second rate through UDP in the Gigabit mode. But unfortunately now I can just reach to 190 Mega bit per second.
What should I do about this?

Is your problem at the writing side?  Ie: you just can't send the data fast enough?  Or, are you sending it, but it's getting dropped or not read fast enough on the receiving side?

It's my understanding that to get even close to the theoretical bandwidth with GigE, you need to use jumbo frames...  Ie: MTU settings much higher than 1500...  I believe 9000 is a popular jumbo frame setting...  But, it needs to be supported by both end machines and any switches you have in between...  And, you need good, short cable runs, with no interference nearby...

But, if your sending app is reading data off a hard drive to send, THAT might be your limiting factor, since most hard drives are going to be pretty slow in comparison...  Similarly, if the receiving app is writing data to a hard drive, it's going to be speed-limited by how fast it can write the data...

as i checked I found that in the sender side it cannot send faster than 11 mega byte per second.

I also checked the jumbo packets with the mtu 9000 on both side but it makes no difference.

Offline

#11 2014-08-13 12:54 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Raw socket sniffer miss packets

i3839 wrote:

I didn't know about recvmmsg() , thanks Rob! A bit embarrassing that they messed up the time-out handling though...

Yeah, I think they eventually fixed it, but it's still somewhat strange semantics...  Apparently, it doesn't even begin to kick in until the first message is received, then just limits how long to wait for additional messages before returning...  Which, I suppose, seems somewhat reasonable...  But, really, anyone feeling a need to use such a function presumably is dealing with a very fast flow of incoming packets, so it seems almost pointless, since at any given time there is likely to be a bunch of packets sitting in the receive queue already that it can just immediately return, without needing to screw about with any timeouts...  I'd imagine just doing a standard user-space select()/poll()/epoll* timeout to wait for the socket to be readable, then trying recvmmsg() in non-blocking mode to grab as many as possible and return immediately, would serve most caller's needs... *shrug*

tahmoures wrote:

as i checked I found that in the sender side it cannot send faster than 11 mega byte per second.

Did things slow down?  Previously you said you got to 190 Mbps, I thought?  11 MBps would be only 88 Mbps...

Also, you didn't address where the data you're sending is coming from...  If it's being read off a hard drive, might that be the limiting factor?

Also, what size datagrams are you sending?  Based on MTU size?  Or much larger, just letting them fragment?

Also, see the link i3839 posted...  There they link to another page detailing LOTS of system configuration and optimization that should be done to get the best performance...  One of the many things mentioned is that they strongly recommend disabling SELinux, so if you've got that enabled that may be killing your performance...  Apparently, disabling TCP timestamps helps a lot as well, but since you're using UDP that's probably of no use to you...

Offline

Board footer

Powered by FluxBB