UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

#1 2009-05-12 09:04 AM

jordi.escola
Member
Registered: 2009-05-12
Posts: 4

Re: Thread LEAK

Hello.. i have this sample code...

I have a problem with this.. the memory used by the process is not liberated when the threads ends.. And the memory (RES) is growing as new tthreads are opened,

#include <stdio.h>
#include <iostream>
#include <pthread.h>
static void* cliente(void *datos);

int main()
{
int ret;
pthread_t mythread[4];
printf("BEGIN: press key\r\n");
getchar();
while(ret<4)
{
int error=pthread_create((pthread_t *)&mythread[ret],NULL,cliente,NULL);
pthread_detach(mythread[ret]);

ret++;
}

printf("END: press key\r\n");
getchar();

}
static void* cliente(void *datos)
{
char *buff=(char *)malloc(100000000);
sleep(15);
printf("end thread..\r\n");
free(buff);
buff=NULL;
pthread_exit(NULL);
}


Memory

STEP (BEGIN) VIRT:13560  RES:976
STEP (during creation pthreads)  VIRT:434m  RES:1076
STEP (END, All the threads finished correctly) VIRT: 44424  RES:1100

Why in the last step has not memory (VIRT and RES)  been liberated??

in pmap

0000000000400000       4       -       -       - r-x--  thread1
0000000000600000       4       -       -       - rw---  thread1
0000000015969000     132       -       -       - rw---    [ anon ] <--- this value increase
00000000413e9000       4       -       -       - -----    [ anon ]
00000000413ea000   10240       -       -       - rw---    [ anon ]
0000000041dea000       4       -       -       - -----    [ anon ]
0000000041deb000   10240       -       -       - rw---    [ anon ]
00000000427eb000       4       -       -       - -----    [ anon ]
00000000427ec000   10240       -       -       - rw---    [ anon ]
0000003602200000     112       -       -       - r-x--  ld-2.5.so
000000360241b000       4       -       -       - r----  ld-2.5.so
000000360241c000       4       -       -       - rw---  ld-2.5.so
0000003603200000    1328       -       -       - r-x--  libc-2.5.so
000000360334c000    2048       -       -       - -----  libc-2.5.so
000000360354c000      16       -       -       - r----  libc-2.5.so
0000003603550000       4       -       -       - rw---  libc-2.5.so
0000003603551000      20       -       -       - rw---    [ anon ]
0000003603600000     520       -       -       - r-x--  libm-2.5.so
0000003603682000    2044       -       -       - -----  libm-2.5.so
0000003603881000       4       -       -       - r----  libm-2.5.so
0000003603882000       4       -       -       - rw---  libm-2.5.so
0000003603e00000      88       -       -       - r-x--  libpthread-2.5.so
0000003603e16000    2044       -       -       - -----  libpthread-2.5.so
0000003604015000       4       -       -       - r----  libpthread-2.5.so
0000003604016000       4       -       -       - rw---  libpthread-2.5.so
0000003604017000      16       -       -       - rw---    [ anon ]
000000360dc00000      52       -       -       - r-x--  libgcc_s-4.1.2-20080825.so.1
000000360dc0d000    2048       -       -       - -----  libgcc_s-4.1.2-20080825.so.1
000000360de0d000       4       -       -       - rw---  libgcc_s-4.1.2-20080825.so.1
000000360f400000     920       -       -       - r-x--  libstdc++.so.6.0.8
000000360f4e6000    2044       -       -       - -----  libstdc++.so.6.0.8
000000360f6e5000      24       -       -       - r----  libstdc++.so.6.0.8
000000360f6eb000      12       -       -       - rw---  libstdc++.so.6.0.8
000000360f6ee000      72       -       -       - rw---    [ anon ]
00002ae60dc7a000      12       -       -       - rw---    [ anon ]
00002ae60dc99000      16       -       -       - rw---    [ anon ]
00007fff9ce1a000      84       -       -       - rw---    [ stack ]
ffffffffff600000    8192       -       -       - -----    [ anon ]
----------------  ------  ------  ------  ------


Any body can helpme????

Offline

#2 2009-05-12 11:03 AM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Thread LEAK

For one thing, "ret" isn't initialized to zero, so that can give unexpected behaviour.

Give Valgrind a try, it's a great memory debugger.

That said, you don't have a memory leak I think. What causes not all memory
to be freed are glibc's malloc implementation and heap fragmentation, but
it can be something else as well. When you mmap a library you'll get high VIRT
but low RES. When you start using that library its pages are read from the
file on-demand and copied to memory, and you end up with higher RES. The
same happens for executables and other mapped files.

If you're on Linux look at /proc/$PID/smaps and maps to see what that
memory area is exactly. I suspect it's glibc's/libpthread's RW mapping for
its own bookkeeping, in this case all the thread stuff. That or it's the heap.

Memory usage is a complicated thing affected by many details, using RES
or VIRT to detect if you've a memory leak or not isn't a good method, it
only works if you have a big leak and when they steadily keep increasing
over time (though that can also be caused a by heap fragmentation). The
numbers you posted aren't out of the ordinary and I wouldn't worry about it.

If you repeat your test again after the first one your memory usage should
end up about the same as it ended up before. But comparing cold cache
with the memory usage afterwards will never work. To hunt down real
memory leaks use Valgrind (or read the code).

Offline

#3 2009-05-12 12:13 PM

jordi.escola
Member
Registered: 2009-05-12
Posts: 4

Re: Thread LEAK

I use valgrind too.. and mtrace..

But my problem is that  the memory is not liberated totally, and with every new thread that is created new memory is accumulated.

TIA

Offline

#4 2009-05-12 03:17 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Thread LEAK

I use valgrind too..

Well, if "valgrind --leak-check=full" doesn't find any leak, you almost certainly don't
have one...

But my problem is that the memory is not liberated totally, and with every new thread that is created new memory is accumulated.

Really?  Because, your posted results don't really show that...  You show a very
mild increase in memory at the end over what you had at the start, and that could
very easily be explained by mmap()'d pages being paged in, as i3839 said, along
with possibly small malloc()'d blocks of memory being held in your heap for later
reuse by your malloc() implementation...  If this is Linux/glibc, you can possibly return
some of that heap memory back to the system via malloc_trim()...  But, in general,
you shouldn't need to bother doing so...  In real code, you're likely to end up reusing
the same freed memory again in the future anyway, so there's not much point in
returning it back to the system...

If malloc_trim() cuts down your final memory usage totals, it's likely from memory
allocated by libpthreads or something, and NOT from your malloc(), though...  Those
big 100M malloc()'s are likely using mmap(), and the memory usage seems to be
disappearing pretty much in full by the end of your run...  Your final stats are really
only trivially elevated from the starting stats...  What exactly makes you think this is
such a big deal??

You claim every new thread increases the final totals and they just keep climbing,
but that could be explained by internal malloc()'d memory in libpthread for each new
thread you create...  Since you create them all up front and then they all die together
later, there's no opportunity for reuse of that memory...  Try this: after all the threads
exit, have your main() create another group of the same number of threads, and
see what the situation looks like after THEY exit...  I'm betting it'll be pretty much
identical to your current ending situation...  Ie: no further increase in memory...

Offline

#5 2009-05-12 04:21 PM

jordi.escola
Member
Registered: 2009-05-12
Posts: 4

Re: Thread LEAK

Begin program ....  13560   984

End program 109964  1312

If you modify the code and increase to 4000 threads the final memory(report) increase 300kb. and not liberated, virtual memory aprox 100 Mb.

The memory should return to her first (984) value  or not?

My problem is in a big project that i have... and the memory increase up to 1 > 2 GB because we received to much connections every time..

This sample is a little scale in order to find a solution...

Offline

#6 2009-05-12 09:47 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Thread LEAK

Begin program .... 13560 984

End program 109964 1312

Are those numbers from what I said to do: a second loop of threads after the first
ones had completely exited?  Or, just from increasing the simultaneous thread
count?  Because, modifying your program to do what I said (add a second loop of
threads that only get created after the others exit), I see no extra increase beyond
the memory used after the first batch...

The memory should return to her first (984) value or not?

No, almost certainly not...  Like i3839 said, some of that will grow and stay as pages
are swapped in...  Some will also grow from malloc()'d heap memory that doesn't get
returned to the system when free()'d, but stays around for later reuse...

Here's my modified version of your code:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>

static void* cliente(void *datos);

int main()
{
    int ret;
    pthread_t mythread[4];
    char buf[1024];

    sprintf (buf, "ps -p %d u", getpid());
    system (buf);
    printf("BEGIN: press key\r\n");
    fgets(buf,sizeof(buf),stdin);

    for (ret = 0; ret<4; ret++) {
        pthread_create((pthread_t *)&mythread[ret],NULL,cliente,NULL);
        pthread_detach(mythread[ret]);
    }

    sleep (1);
    sprintf (buf, "ps -p %d u", getpid());
    system (buf);
    printf("END: press key\r\n");
    fgets(buf,sizeof(buf),stdin);
    sprintf (buf, "ps -p %d u", getpid());
    system (buf);

    for (ret = 0; ret<4; ret++) {
        pthread_create((pthread_t *)&mythread[ret],NULL,cliente,NULL);
        pthread_detach(mythread[ret]);
    }

    sleep (1);
    sprintf (buf, "ps -p %d u", getpid());
    system (buf);
    printf("END2: press key\r\n");
    fgets(buf,sizeof(buf),stdin);
    sprintf (buf, "ps -p %d u", getpid());
    system (buf);

    return (0);
}

static void* cliente(void *datos)
{
    char *buff=(char *)malloc(100000000);
    sleep(5);
    printf("end thread..\r\n");
    free(buff);
    buff=NULL;
    pthread_exit(NULL);
}

And, here's output from a run...

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0   5800   444 pts/3    S+   17:45   0:00 ./crap
BEGIN: press key

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0 437556   592 pts/3    Sl+  17:45   0:00 ./crap
END: press key
end thread..
end thread..
end thread..
end thread..

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0  38776   628 pts/3    S+   17:45   0:00 ./crap
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0 439660   652 pts/3    Sl+  17:45   0:00 ./crap
END2: press key
end thread..
end thread..
end thread..
end thread..

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0  38776   628 pts/3    S+   17:45   0:00 ./crap

Offline

#7 2009-05-13 09:14 AM

jordi.escola
Member
Registered: 2009-05-12
Posts: 4

Re: Thread LEAK

correct... in your code, the RSS memory increase aprox 200 kb... if you increase the number of threads the  memory used  increasingly and it would not be liberated

Offline

#8 2009-05-13 01:43 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Thread LEAK

Start to actually read what we write for a change...

After the first run it's according to Rob's figures

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0  38776   628 pts/3    S+   17:45   0:00 ./crap

After the second run it's, oh surprise:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ras      16842  0.0  0.0  38776   628 pts/3    S+   17:45   0:00 ./crap

The other numbers are either when the test is running and not all threads
finished yet, or in the cache cold case, which you're focussing on too much.

Offline

#9 2009-05-13 01:56 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Thread LEAK

in your code, the RSS memory increase aprox 200 kb...

Yes, but the key point I was making was that that increase is stable and NOT a leak...
As you can see from the second set of thread creation: it does NOT increase that
RSS size up any higher than the first set of threads did...  The final RSS size at the
very end is exactly the same as it is after the first set of threads exit...

if you increase the number of threads the memory used increasingly and it would not be liberated

Yes, the RSS does increase a bit more for each simultaneous thread running...  And,
that's perfectly fine and not a problem, and certainly not a "leak" of any kind...  Where
it is coming from is probably libpthread overhead of some sort...  However, as demonstrated,
this value is stable: if you recreate another group of threads after the first group exit,
the end result is identical, and does not further increase...  So, it's not a leak...

The VSZ is the one that really climbs a bunch for each new thread...  But, that's
presumably just due to each thread's separate stack needing virtual address space,
and once given out, you retain that address space even after the thread exits...  But,
just as with RSS, this value remains stable and doesn't further increase if you create
another group of threads afterwards...  So, the space is being properly reused, and
there is no leak...

Offline

#10 2009-05-13 09:23 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Thread LEAK

The VSZ is that big because each thread allocates 100MB.

Offline

#11 2009-05-13 10:49 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: Thread LEAK

The VSZ is that big because each thread allocates 100MB.

No, I didn't mean the run-time VSZ while all the threads are alive...  I mean the
post-exit VSZ jumps a fairly large amount over starting size...  Which I assume must
be due to the thread stacks or some other similar per-thread data...  The 100M
malloc()'d memory dissappears from the final post-exit VSZ, because I assume
malloc() is using mmap() for such a huge amount, so it gets returned immediately...

But, in any case, the important point is there's no leak, anyway: once either value
stabalizes after creating a certain number of simultaneous threads, it doesn't climb
any higher than that (unless you exceed that previous simultaneous thread count)...

Offline

#12 2009-05-13 11:38 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: Thread LEAK

I was mind boggled when I saw that huge VSZ, until I remembered the malloc.
Yeah, 40MB is pretty bad, but we're talking about glibc here. ;-)

With 16KB stacks I get:

$ ./t                                                                       USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
indan     1522  0.0  0.0   1604   420 pts/0    S+   00:33   0:00 ./t
BEGIN: press key

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
indan     1522  0.0  0.0 393472   540 pts/0    Sl+  00:33   0:00 ./t
END: press key
end thread..
end thread..
end thread..
end thread..

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
indan     1522  0.0  0.0   2884   612 pts/0    S+   00:33   0:00 ./t
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
indan     1522  0.0  0.0 409924   644 pts/0    Sl+  00:33   0:00 ./t
END2: press key
end thread..
end thread..
end thread..
end thread..

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
indan     1522  0.0  0.0  19284   644 pts/0    S+   00:33   0:00 ./t


Interestingly enough the second number is very low, but the final one is still the same as before I reduced the stack.

Offline

Board footer

Powered by FluxBB