UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2010-02-20 10:02 PM

RipRage
Member
From: England
Registered: 2010-01-06
Posts: 146

Re: Fread() in chunks

Hi experts
i am trying to read a file in chunks using fread() and malloc() i just cant seem to fiqure it out. i only know how to suck a file in all in one go. if anyone could please explain and show a quick example on how to do this correctly i would be most grateful! many thanks.

#include <stdio.h>
#include <stdlib.h>

#define filename "C:\\Image.jpg"

int main(void)
{
	FILE *pf;
	char *buf;
	unsigned long fsize;
	size_t bytes_read;

	pf = fopen(filename, "rb");
	if (pf == NULL) {
		printf("File not found!\n");
		return 1;
	}
	else {
		printf("Found file %s\n", filename);

		fseek(pf, 0, SEEK_END);
		fsize = ftell(pf);
		rewind(pf);

		printf("File contains %ld bytes!\n", fsize);

		buf = (char*) malloc (sizeof(char)*fsize);
		if (buf == NULL) {
			printf("Memory error!\n");
			return 2;
		}
		else {
			printf("Allocating memory!\n");

			bytes_read = fread(buf, 1, fsize, pf);
			if (bytes_read != fsize) {
				printf("Reading error!\n");
				return 3;
			}
			
			printf("The whole dame file is in memory!, greedy guts!\n");
			fclose(pf);
			free(buf);
		}
	}

	return 0;
}

Offline

#2 2010-02-21 03:26 AM

jfriesne
Administrator
From: California
Registered: 2005-07-06
Posts: 348
Website

Re: Fread() in chunks

The typical way to do it is to wrap fread() into a function that will loop until it has read in all the data required.   Something like this (warning:  I only eyeballed this code, I didn't test it, so it's possible it might contain a bug):

// Tries to read (numBytesToRead) bytes from (pf) into (buf)
// Returns (numBytesToRead) if it read (numBytesToRead) bytes, or -1 if it failed
size_t fread_fully(FILE * pf, char * buf, size_t numBytesToRead)
{
   size_t bytesRead = 0;
   while(bytesRead < numBytesToRead)
   {
      size_t nextReadSize = fread(buf+bytesRead, 1, bufSize-bytesRead, pf);
      if (nextReadSize <= 0) return -1;  // error or early EOF!
      bytesRead += nextReadSize;
   }
   return numBytesToRead;
}

Then just have the code you posted above fread_fully() instead of fread().

Offline

#3 2010-02-21 10:27 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,231

Re: Fread() in chunks

Jeremy, fread(3) already reads as much as you tell it to, except when
encountering EOF or an error. It's not like read(2) on sockets.

That said, I've no idea what you want to achieve RipRage. If you want to
get the whole file in memory then you can as well do it with one fread call
instead of in chunks. As the end result is the same, I assume you want to
read the file in chunks, do some processing on a chunk, then discard it and
read the next one. Or something like that?

If that is the case then just use a fixed size buffer (e.g. 128KB) to fread into,
no need to get the file size. After each fread process the data before looping
again, and break out of the loop when fread returns 0 (or stop after handling
the first chunk smaller than the bufsize. Check for errors though, it might not
be EOF).

So the reading part is simpler, but the data processing part is harder,
depending on what you actually want to do.

Offline

#4 2010-02-21 11:31 AM

RipRage
Member
From: England
Registered: 2010-01-06
Posts: 146

Re: Fread() in chunks

Ok thanks folks! And I am sorry I should of made it more clear, the main reason for this is I want to be able to cast a loop were I can use fread() and send() to read chunks of a file and fire it out of a socket till eof. But Im not sure how to do that, I only know how to suck the whole file in and blast it out, witch is ok for small files but if i was using a large file then it's not very pratical to store the whole thing in RAM.

Offline

#5 2010-02-21 01:39 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,231

Re: Fread() in chunks

In that case just alternate fread and send, while handling short sends.

Offline

#6 2010-02-21 02:02 PM

RipRage
Member
From: England
Registered: 2010-01-06
Posts: 146

Re: Fread() in chunks

Ok I'll have a play with it, I'm still fairly new to c so I'm trying to take in as much as possible so thank you for your patience and you all have tought me alot! What would be the correct expression to break from the loop it should be fread() break, write()/send() break. back to fread() and so on till EOF. i dot know why but i just cant seem to nail loops i never know what expression to use to break, and i find extreamly fustrating.

Offline

#7 2010-02-21 07:03 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Fread() in chunks

What would be the correct expression to break from the loop it should be fread() break, write()/send() break. back to fread() and so on till EOF.

Yeah, basically...  Here's a rough example:

int file2socket (FILE *fp, int sockfd)
{
    char buf[8*1024];
    size_t len;
    int ret;

    for (ret = 0;;) {
        len = fread (buf, 1, sizeof (buf), fp);
        if (len == 0) {
            ret = feof (fp);
            break;
        }
        if (!sendall (sockfd, buf, len)) break;
    }

    return (ret);
}

Where sendall() is a looping write()/send() function, of the sort I posted previously...
The buffer size to use can be argued over, but in general the larger the better, at
least up to a point...  And, keep it a multiple of your page size, for maximum
efficiency...  (Of course, for even more efficiency, you can avoid copying the file
data into user-space at all via mmap()+write()/send() or sendfile(), as I previously
mentioned in another thread...)

Offline

#8 2010-02-21 07:08 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Fread() in chunks

Oh, and for the best network efficiency, you should leave Nagle enabled and/or
set TCP_CORK during the fread()+sendall() loop, as well...  That will enable packing
all the file data into as few network packets as possible...

Offline

Board footer

Powered by FluxBB