UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

  • Index
  • » C
  • » recv()'ing unkown data size

#1 2007-06-20 05:01 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Hi all,

I'm thinking, and thinking, and thinking.
Even after I red the article on this website, I can't imagine how to create a function to recv() an unknown data size.

I want it in 1 function, so I can build an function something like srecv(handler).
only one parameter, wich is the fd. (no recv() lengths or something).

Is there somebody who knows how to recv() an unkown data size? I know that all te messages end with: "\r\n".

Best Regards,
Niek van der Steen

Offline

#2 2007-06-20 08:16 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

Offline

#3 2007-06-20 08:20 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

I just re-read, and see you say you don't even want to pass a buffer or max length
to the function...  In that case, I assume you want it to just dynamically allocate a
big enough buffer for you?  That's doable, but a little tricky...  You might be better
off using glibc's getline() or getdelim() (in combination with the above-mentioned
stdio fdopen() trick), if you have access to them...

Offline

#4 2007-06-20 11:37 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: recv()'ing unkown data size

I'd write a receive function which does the socket stuff, handling all errors and EAGAIN, and reads as much data as possible into a fixed size buffer. Then it parses that buffer for whole "messages", and call the message handling function which handles the message itself.

If there's a partial message left, store it (e.g. malloc + memcpy). Repeat the whole thing later when more data can be received. If you wrote the code so it does recv() in a loop, then adding the handling for partial messages is probably easy.

So basically I think the "problem" that needs to be solved is the handling of partial messages.

Personally I don't see much advantage of using MSG_PEEK, only one that pops up is that it could avoids calling malloc(), saving some memory handling. But if the receive buffer isn't big enough to receive the biggest possible message then MSG_PEEK won't work. Disadvantage of MSG_PEEK is that you always do extra work (recving all data twice), while with the above method only extra work is done in the (hopefully rare) case that a partial message is received. But simplicity of code is worth more than a small performance improvement (if any).

Offline

#5 2007-06-21 01:21 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

But, if you're going to do your own stdio-style buffering, why not just use stdio
instead?  There's really little point in reinventing wheels (especially ones as old
and well-refined by this time as stdio is)...  I just see NO logic at all in making your
own stdio-like buffering system...

However, the big advantage to the MSG_PEEK approach is that not all I/O needs to
flow through stdio (or your stdio-alike buffering system)...  If you start buffering, then
you can NEVER do raw read()/recv() on the socket again...  But, with the MSG_PEEK
approach, it never actually takes anything out of the socket buffers for extra buffering,
so there's no problem with mixing and matching your I/O methods as you like...

As for the buffer size: that's going to be problematic, no matter WHAT method you
go with, unless you want to just dynamically grow the buffer as needed (which you
can do with any approach, MSG_PEEK included)...  But, if you've designed the app
protocol, then presumably you should have SOME idea of the max possible message
size you should have to deal with, and can set some sane upper limit and use a
fixed buffer of that size to hold any message...  I don't really think it should be a
real issue, in real-world practice...

As for MSG_PEEK being less efficient, well maybe...  The main reason I came up
with that approach (and, I'm sure I wasn't the first to do so, but it did come to me
on my own one day) was to replace the standard-at-the-time alternative method
of reading delimited input (without forcing all future I/O through a stdio-like buffering
layer, that is), which was reading a single char at a time until you hit the delim...
So, as you can see, the MSG_PEEK approach GREATLY improved on the
efficiency of THAT approach... ;-)  Is stdio-style buffering more efficient?  Yes,
probably...  But, it has its down-sides, too...

Of course, I think we all agree that the BEST approach to handling variable-length
messages over TCP is to simply prepend a fixed-size message header giving the
full message length...  That's very easy and efficient to handle, and no need to
worry about handling silly delimiters at all... ;-)

Offline

#6 2007-06-21 02:40 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: recv()'ing unkown data size

Yeah, fixed size message headers is really so much better than the rest.

As for not using fgets: more control, don't have to trust glibc, less overhead, and of course NIH. I'm not sure how it interacts with non-blocking sockets and zero length recv()'s either, so from my point of view, not worth the risk.

Offline

#7 2007-06-21 02:40 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Sooooooo.........:

recv()'ing first with MSG_PEEK, wich returns the amount of characters waiting in the buffer? If it is zero, or -1 I return that value.

If it is higher as 0, I recv() it with "0" as flags?

edit:
The amount of data waiting after a MSG_PEEK, where is it stored? In the buffer var? Or the return value?

Offline

#8 2007-06-21 09:14 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

I can't test it right now, so I ask it.

Would this work properly?:

char function srev(int fd) {
    int amount;
    char buf;
    char buffer;
    amount = recv(fd, buf, 0, MSG_PEEK);
    if(amount < 1) {
        return amount;
    } else {
        recv(fd, buffer, amount, 0);
        return buffer;
    }
}

Offline

#9 2007-06-21 09:37 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

Offline

#10 2007-06-21 09:49 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Okay. I will do that.

What do you guys recommend me for a max length?
I'm gonna use it for the MSN protocol (do not shoot me, please =P )

Offline

#11 2007-06-21 09:50 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

Offline

#12 2007-06-21 10:00 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

Offline

#13 2007-06-22 01:41 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: recv()'ing unkown data size

Offline

#14 2007-06-22 08:58 AM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Offline

#15 2007-06-22 09:32 AM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

okay, I just re-read your post. And saw the function again, But. I don't get the defenition of all the parameters.?

Offline

#16 2007-06-22 01:09 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

And, what don't you get, exactly?  It's only got 4 arguments: sockfd is obviously the
socket FD; buf is the buffer to receive the message into; bufsz is the size of that
buffer (ie: the max length message you can ever possibly receive); and delim is the
delimiter string that divides your messages...  (For "\r\n"-delimited input, I would
recommend just assuming a delimiter of "\n", which covers you for sloppy implementations
which only send LF without the CR, as well...  Then, you can just trim any trailing
series of "\r" and/or "\n" from the end of the message before processing...)

Offline

#17 2007-06-22 01:22 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,839
Website

Re: recv()'ing unkown data size

Offline

#18 2007-06-22 01:27 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Okay! =D

This will defenitly (I don't know how to write that word :P ) help me out! thnq!

Offline

#19 2007-06-22 01:58 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

hmm..

When I compile that header file ( I placed all the functions in a header file ) I got these messages. Very strange:

pascal# make test
cc -O2 -fno-strict-aliasing -pipe   test.c  -o test
In file included from test.c:17:
sock.h:27: error: syntax error before '&' token
sock.h: In function `ssend':
sock.h:28: error: `fd' undeclared (first use in this function)
sock.h:28: error: (Each undeclared identifier is reported only once
sock.h:28: error: for each function it appears in.)
sock.h:28: error: `msg' undeclared (first use in this function)
test.c: In function `main':
test.c:22: warning: passing arg 1 of `sockopen' makes integer from pointer without a cast

This is my code:

/*
 * sock.h
 * Header file
 * gemaakt voor makkelijk aanroepen van sockets
 *
 * TELAVIA
 * (c) June 2007
 *
 *  _____    _             _       
 * |_   _|  | |           (_)     
 *   | | ___| | __ ___   ___  __ _
 *   | |/ _ \ |/ _` \ \ / / |/ _` |
 *   | |  __/ | (_| |\ V /| | (_| |
 *   \_/\___|_|\__,_| \_/ |_|\__,_|  
 */

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <netdb.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <sys/socket.h>

int ssend(int fd, char &msg) {
    send(fd, &msg, strlen(&msg), 0);
}

int sockopen(char hostnaam, int portnummerke) {  // Functie voor het openen van een sok (maat 45)
    int sockfd, aant_bytes;  
    struct hostent *he;
    struct sockaddr_in hun_adres;

    if ((he = gethostbyname(&hostnaam)) == NULL) {
        perror("gethostbyname");
        exit(1);
    }

    if ((sockfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        perror("socket");
        exit(1);
    }

    hun_adres.sin_family = AF_INET;
    hun_adres.sin_port = htons(portnummerke);
    hun_adres.sin_addr = *((struct in_addr *)he->h_addr);
    memset(&(hun_adres.sin_zero), '\0', 8);

    if (connect(sockfd, (struct sockaddr *)&hun_adres, sizeof(struct sockaddr)) == -1) {
        perror("connect");
        exit(1);
    }

    return sockfd;
}

int srecv(int sockfd, void *buf) {
    int ret, len, i, j, bufsz;
    char *bufp, *p, *delim;

    bufsz = 8000;
    delim = "\n";

    if (delim == NULL) delim = "";
    len = strlen (delim);
    if (len < 1) len = 1;

    for (ret = 0, bufp = buf; ret < bufsz; ) {
        do {
            i = recv (sockfd, bufp, bufsz - ret, MSG_PEEK);
        } while ((i < 0) && (errno == EINTR));
        if (i < 0) {
            ret = i; break;
        } else if (i == 0) {
            break;
        }
        for (p = memchr (bufp, *delim, i); p; ) {
            if ((len > 1) && memcmp (p, delim, len)) {
                p++; j = i - (p - bufp);
                if (j > 0) p = memchr (p, *delim, j);
                else p = NULL;
            } else {
                p += len; i = p - bufp; break;
            }
        }
        do {
            i = recv (sockfd, bufp, i, 0);
        } while ((i < 0) && (errno == EINTR));
        if (i < 0) {
            ret = i; break;
        } else if (i == 0) {
            break;
        } else {
            bufp += i; ret += i;
            if (p) break;
        }
    }
    return (ret);
}

Offline

#20 2007-06-22 02:15 PM

biologz
Administrator
From: Puking on the pavement
Registered: 2005-11-02
Posts: 396

Re: recv()'ing unkown data size

Hi,

i've noticed the following error in your ssend function:

int ssend(int fd, char &msg) {
    send(fd, &msg, strlen(&msg), 0);
}

the prototype is bad, it should be:

int ssend(int fd, char *msg)

I think that's what cause all the errors, try and tell me.


gethostbyintuition() is still a dream of mine

                                                 -- quoted from bash

Offline

#21 2007-06-22 02:17 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Yeah, a lot of errors are solved right now!

But I still got these:

pascal# make test
cc -O2 -fno-strict-aliasing -pipe   test.c  -o test
In file included from test.c:17:
sock.h: In function `ssend':
sock.h:28: warning: passing arg 1 of `strlen' from incompatible pointer type
test.c: In function `main':
test.c:22: warning: passing arg 1 of `sockopen' makes integer from pointer without a cast

Offline

#22 2007-06-22 02:22 PM

biologz
Administrator
From: Puking on the pavement
Registered: 2005-11-02
Posts: 396

Re: recv()'ing unkown data size

Yes it should be:

strlen(msg)

and not strlen(&msg) because strlen expact a char *.


gethostbyintuition() is still a dream of mine

                                                 -- quoted from bash

Offline

#23 2007-06-22 02:24 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Oh dear. I'm really Day-Dreaming or something. In PHP I never have these problems..

But why it says that error in the sockopen() function ("blabla makes integer from pointer without a cast").
It isn't an integer!?

Offline

#24 2007-06-22 02:29 PM

biologz
Administrator
From: Puking on the pavement
Registered: 2005-11-02
Posts: 396

Re: recv()'ing unkown data size

Well, i guess hostnaam should be a char * because if it's a char, there is only one character...

So you should change that in your prototype:

int sockopen(char *hostnaam, int portnummerke)

And give to the function a pointer to a char, it should work. Don't forget to read the manpages of the functions you use.


gethostbyintuition() is still a dream of mine

                                                 -- quoted from bash

Offline

#25 2007-06-22 02:30 PM

niek
Member
Registered: 2006-12-24
Posts: 109

Re: recv()'ing unkown data size

Yeah, that was what I thought. But when I change it, I got this:

pascal# make test
cc -O2 -fno-strict-aliasing -pipe   test.c  -o test
In file included from test.c:17:
sock.h: In function `sockopen':
sock.h:36: warning: passing arg 1 of `gethostbyname' from incompatible pointer type

Offline

  • Index
  • » C
  • » recv()'ing unkown data size

Board footer

Powered by FluxBB