UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2005-08-03 01:52 PM

enjoy
Member
Registered: 2005-03-13
Posts: 20

Re: mmap() for large files?

Hi,

I am using mmap() function to map some big binary file (image file) in my project.mmap() is working fine to some extend.But when i try to map some big files, it is not fine so. see my program,
...
#include <sys/mman.h>
.....
int fd;
unsigned char *left;
unsigned char *right;

fd=open("file1.Img",O_RDONLY);
left=(unsigned char*)mmap(NULL,(35000*35000),PROT_READ,MAP_PRIVATE,fd,0);
if(left==MAP_FAILED){
           cout<<"Left map error";exit(-1);
}
close(fd);

fd=open("file2.Img",O_RDONLY);
right=(unsigned char*)mmap(NULL,(35000*35000),PROT_READ,MAP_PRIVATE,fd,0);
if(left==MAP_FAILED){
           cout<<"Left map error";exit(-1);
}
close(fd);
.........

this is how i am mapping.here only the first mmap(left) is happening fine. The second one is not happening.
1)Is it because of memory insufficiency?
or what is the real problem here?
2)what i have to do for mapping both the image files(binary image files)?
3)is mmap()  consistent?
4)but both the mapping is fine for 30000*30000 bytes of data.why so?

Note: my system is having 1000MB main memory  and 2GB swap space to use.

regards
Enjoy

Offline

#2 2005-08-03 05:47 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,847
Website

Re: mmap() for large files?

That's a pretty huge amount of memory to be mapping...  I suspect you're just
running out of room in your process address space for more mappings of such
huge size...  Depending on kernel configuration, on an x86 system, you may
have as little as 1G of addressable user-mode process space...  Typically, it's
3G, though...  But, remember, there's already lots of other stuff there (program
code, all shared libs used, your stack, heap, etc.)...  So, it wouldn't surprise me
if you were running into that limit...  However, your main system memory is only
3G total (1G real, 2G swap), as well, so that's pretty tight for the amount of data
you're mapping here, too...  (35000*35000 = ~1.2G)  And, I think mmap() will
reserve swap space in advance for MAP_PRIVATE mappings, so you'd never be
able to exceed the size of your total swap space alone...  (Adding MAP_NORESERVE
might bypass that restriction...)

But, the proper thing to do is check the value of errno (call perror() or print out
"strerror(errno)") after the mmap() fails, and that will tell you why it's failing, rather
than having me just guess at the cause...  (See "man mmap" for more detailed
descriptions of the failure conditions...)

Offline

#3 2005-08-04 01:02 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: mmap() for large files?

Offline

#4 2005-08-08 06:10 AM

enjoy
Member
Registered: 2005-03-13
Posts: 20

Re: mmap() for large files?

Hi,

I already questioned that i have been facing problem in mmap()ing large files.
when i checked with error number, the result is that 'cannot allocate memory'(i have 1GB main memory + 2 GB swap space).But when i tried with 64-bit machine the large file mmap()ing is happening well.

but even in 64-bit machine, 'Bus error' is coming. what is this bus error? why it is coming?
how i am mmap()ing is,

........
unsigned char *ptr;

ptr=(unsigned char*)mmap(NULL,(35000*33000),PROT_READ,MAP_SHARED,file_des,0);
.....
...

eventhough the MAP_FAILED is not happening,at the mid of some processing on ptr(only read), 'Bus error' is comming.
is it because of that the required bytes is not given in terms of multiples of page size?

i want to learn more about mmap()ing large files? can you give some links regarding this? is there any possibility to use shared memory concept? would shared memory techinic be second to none for mmap()ing?
get me details about using shared memory techinic for mapping.

in the previous thread , you suggested about code tags. what is that code tags? get me details about code  tag!!!


regards,
Enjoy

Offline

#5 2005-08-12 03:12 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,239

Re: mmap() for large files?

Offline

#6 2008-06-14 02:58 PM

mohit3884
Member
Registered: 2008-06-14
Posts: 2

Re: mmap() for large files?

Hii
I am implementing the shared memory of size abt 5 MB. But whenevr i try to read or write the shared memory beyond abt 4kb, its showing bus error .(I am also facing the same prob as faced by "enjoy".)
I gone through the replies but was not able to solve my problem. Kindly help

@smile: were u able to solve the bus error problem. if yes ,please share the solution

Offline

#7 2008-06-14 06:44 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,847
Website

Re: mmap() for large files?

And, are you sure you're not going past the end of the mapped file, as i3839 notes
a probable cause of SIGBUS?  If so, can you post your code?

Offline

#8 2008-06-16 07:12 AM

mohit3884
Member
Registered: 2008-06-14
Posts: 2

Re: mmap() for large files?

Yahh m  not going past the end of the mapped file.

Offline

Board footer

Powered by FluxBB