UNIX Socket FAQ

A forum for questions and answers about network programming on Linux and all other Unix-like systems

You are not logged in.

  • Index
  • » C
  • » [SOLVED] some guidance on debugging a segfault originating in printf?

#1 2011-08-16 07:58 PM

thinking
Member
Registered: 2005-09-15
Posts: 103

[SOLVED] some guidance on debugging a segfault originating in printf?

hi all

whats the problem:
a test application written in c segfaults on printf() call's
the application is used to test a library i've written for distributed computing purpose
in current state, not that special, just sending a few udp broadcasts and doing tcp connects

more details on the segfault behaviour:
the prog is compiled like this

gcc -g -ggdb -g3 -ggdb3 -Wall -Wextra -Wpadded -O0 -I../../external -I../../src -o main ./main.c

running with gdb shows the following backtrace after the segfault:

Program terminated with signal 11, Segmentation fault.
#0  _IO_vfprintf_internal (s=0xbf9cff4c, format=0x2 <Address 0x2 out of bounds>, ap=0xbf9cff7a "A\037w") at vfprintf.c:1288
1288	vfprintf.c: No such file or directory.
	in vfprintf.c
(gdb) bt
#0  _IO_vfprintf_internal (s=0xbf9cff4c, format=0x2 <Address 0x2 out of bounds>, ap=0xbf9cff7a "A\037w") at vfprintf.c:1288
#1  0x006818a0 in __printf (format=0x8058d4d "\ncommand: ") at printf.c:35
#2  0x08058009 in main (argc=5, argv=0xbf9d04f4) at ./main.c:197

let's have a look at main.c:196 - main.c:198

while(buffer[0]!='q' && run){
        printf("\ncommand: ");
        if(scanf("%s", buffer)<=0) continue;

what else can be said:
if i remove every printf in the whole app AND library it doesnt segfault but the while loop above ends after a few loops cause run is switched to 0
the weird thing - i removed the SIGTERM handler which set's run to 0, so it shouldnt break until i enter q, which in fact i didnt
that's why i assume some application space corruption im not expirienced with

after adding some printf in the library it segfaults again after a few loops on printf, nowhere else
the printf can also be whatever you want

i also tried valgrind and it doesnt detect any memory leaks or wrong assignments or such
it really doesnt print anything remarkable until the segfault

what i think:
my opinion is that i maybe do something wrong with a few pointers and cause something to break
im using a few function pointers for library callbacks and a double linked list which does a few complicated pointer arithmetic operations
it seems that the problem is also tightly coupled to the test application and library im working on

i tried to break it down to a test case but i dont the the error to occur on smaller pieces

questions:
since i don't think this is a problem where someone can answer "its caused by ...."
my question is
"given a correctly working libc, ubuntu installation (natty) and no hardware failure, which i just assume, what needs to be done to cause printf to segfault like this altough it is called with a static string like printf("test");"?

i think this goes into something like corrupting heap or stack which im not that much familiar with - maybe there are some resources on how stack/heap corruption is done?

thanks for any help on this

[EDIT]
from http://www.learncpp.com/cpp-tutorial/79 … -the-heap/
i got a simple example for a stack overflow

int main()
{
    char nStack[100000000];
    printf("test");
    return 0;
}

which causes printf to segfault

on my ubuntu installation i found that a size of 8384000
results in a segfault pretty much the same way as i described above
so it seems that i corrupt my function stack and after reading a few articles on this topic i think i was using too much network buffer variables in combination
with a bunch of function calls using local struct's
altough corrupting nearly 8MB of stack seems hard to believe, anyway ill try to reduce the memory footprint as a first step
[/EDIT]

Last edited by thinking (2011-08-31 03:28 PM)

Offline

#2 2011-08-17 12:53 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: [SOLVED] some guidance on debugging a segfault originating in printf?

I doubt it's a pure stack overflow of that sort (ie: just using too much stack space)...  Much more likely is dynamic heap corruption...  Though, I'm not sure why valgrind wouldn't spot it for you...  It's always been great at catching such things for me...  Perhaps it is stack corruption of some sort, such as overflowing a stack buffer and trashing other important stuff also on the stack (like function return addresses)...  It's hard to say without seeing more code...

However, just from the 3 lines you posted, I don't like the look of the third one...  A scanf("%s") is unbounded, so could easily overflow your buffer there...  You'd be far better off using something like fgets() or getline() for input...

Offline

#3 2011-08-17 03:56 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: [SOLVED] some guidance on debugging a segfault originating in printf?

From the lines posted, I would guess that 'buffer' isn't always NUL terminated.
(or line terminated, for that matter.) Make sure the receiver always terminates
the buffer with a NUL before processing the data.

Stack size you can see in /proc/$PID/smaps, for instance. So you could check
that when it's hanging in gdb after a crash.

Took me forever to find out that X crashed with no error messages because
it used more stack than I allowed with ulimit -s.

Offline

#4 2011-08-17 04:07 PM

i3839
Oddministrator
From: Amsterdam
Registered: 2003-06-07
Posts: 2,230

Re: [SOLVED] some guidance on debugging a segfault originating in printf?

I got confused by that scanf, thougth 'buffer' was the input instead of stdin.
That's indeed a strange way of using scanf(), I'd do what Rob suggests and
use something better fitting that's also not unbound.

What I meant is the buffer where the UDP packet is received into. But if the
UDP data aren't strings then never mind about NUL termination.

Offline

#5 2011-08-31 03:22 PM

thinking
Member
Registered: 2005-09-15
Posts: 103

Re: [SOLVED] some guidance on debugging a segfault originating in printf?

it seems i have solved the problem, although i have no idea what exactly went wrong

in general i did two things:
1) adding code to monitor the stack size during runtime
2) rearranging my source

1)
as it seemd that stack corruption plays a role on this i added a local variable in main like this

char *stackstart = NULL;

int main(){
    int someotherlocalvar;
    char bottom_of_stack;
    stackstart = (char*)&bottom_of_stack;
...

in whatever function call i do i get the size of the current stack using this

void somefunction(int someparam){
    char current_stack_position;
    printf("the current stack size is: %d bytes\n", (int)(stackstart-(&current_stack_position)) );
...

there are a bunch of things to note on this code:
o) its not very portable, for example the stack size on my linux box grows from top to bottom
    thats why i do stackstart-current_stack_position
o) this method is not very accurate
    for example gcc reorder's local variables on stack (i think cause of optimization or alignment purpose), thus the bottom_of_stack variable can be located after OR before someotherlocalvar dependin on the architecture youre using - so in general the stackpointer wont be at position 0 on the stack
o) this is just a rough example of what i did and it actually worked for me - i would not recommend using this in production code
    have a look at http://stackoverflow.com/questions/5382 … -size-in-c for another example

using this method i noticed that my global variables are garbaged after a few function calls after the first loop
since i couldn't find any further reason for this i got to step 2 which solved the problem

2)
on project start for fast development i made the decision that i dont split the code in .c and .h files but only use .h files
with functions directly in them
i also had #ifndef/#define/#endif constructs on top/end of every file
in case of a small lib this method is nice to see code changes pretty fast, so not having to do make clean; make and such
as the code grows it gets clumsy
to be honest i made some bad expirience with g++ compiliation with templates and mid sized projects, but this also just reflects my lack of knowledge on compilation units

i dont know what was the problem but i think gcc caused it during linking in combination with some global debugging variables i was using, could also be a gcc bug - hard to track at this point
so i rearranged everything to autotools and a minimal makefile for my own testing purpose
that's it - the code is working now without any change of function internals
just cleanly separating in .c,.h files did it

@robseace and i3839
the reason i used scanf, was that its 'just' for testing and i'm the only user currently
but you're right, the scanf isnt very nice for obvious reason and also shouldn't be used for testing - there are better functions for this

thx for your thoughts on this

Last edited by thinking (2011-08-31 03:26 PM)

Offline

#6 2011-08-31 08:07 PM

RobSeace
Administrator
From: Boston, MA
Registered: 2002-06-12
Posts: 3,829
Website

Re: [SOLVED] some guidance on debugging a segfault originating in printf?

for example gcc reorder's local variables on stack (i think cause of optimization or alignment purpose), thus the bottom_of_stack variable can be located after OR before someotherlocalvar dependin on the architecture youre using - so in general the stackpointer wont be at position 0 on the stack

Plus, there's stack used in libc's pre-main() code, and main()'s argc and argv (and envp) will be on the stack...  Like i3839 said, "/proc/self/*maps" is a much more accurate method of figuring out stack usage on Linux...

Offline

  • Index
  • » C
  • » [SOLVED] some guidance on debugging a segfault originating in printf?

Board footer

Powered by FluxBB