A Simple Lab Guide of Using GDB in Embedded Linux System by cawan (cawan[at]ieee.org or chuiyewleong[at]hotmail.

com) on 27/11/2012 GDB is a useful debugging tool for linux developer as well as linux hacker in understanding the binary running in the system. So, it is a need for embedded linux developer to master the skill of GDB too. However, due to the nature of embedded system which is highly customized in term of processor platform, it is a little bit tricky in applying the basic GDB skill in embedded platform. First, it is important to cross compile the version of GDB which is compatible to the embedded platform. Well, this has been discussed in the previous paper in this blog, "How To Cross Compile GDB and Run in Embedded System". Second, it is always a necessity to understand the architecture internal of the RISC processor, such as MIPS, ARM, PPC, and etc. Besides, for each RISC platform, it is crucial to know the registers, instruction sets, addressing modes, operation modes, calling conventions, status control, and much much more in detail. Third, it is also a need to understand how linux works such as the boot process, file system, runtime, shared library, system call, and etc. In addition, it is an advantage to know more about the linux kernel. So, in order use GDB for relevant jobs in proper, it is important to have some level of fundamental knowledges in the aforementioned topics. On the other hand, the GDB is also a precious tool to assist in understanding the system internal in further details. Yes, it is bidirectional. Besides, it is really recommended to perform local debugging with GDB, instead of remote debugging with GDBServer. Well, the reasons are from a number of aspects, but we will show in examples in our discussion later. So, without wasting any more time, let us start our demonstration of GDB in embedded linux system right now. We are having a not responding process and we need to figure out the root cause of the problem. Our embedded platform is MIPS. tango3[gdb]# ./gdb GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "mips-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Setting up the environment for debugging gdb. No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] /root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file: No breakpoint number 0. (gdb) attach 766 Attaching to process 766 Reading symbols from /bin/busybox...(no debugging symbols found)...done. Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1 0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) Well, the pid of the not responding process is 766 and we attach into it now. From the message output, it seems the program counter is stucked at nanosleep() in

/lib/libc.so.6. Let's do a backtrace. (gdb) bt #0 0x2ac050a8 in nanosleep () from /lib/libc.so.6 #1 0x2ac04d8c in sleep () from /lib/libc.so.6 #2 0x0045b2bc in ?? () Yes, the nanosleep() is being called by sleep(). How to know how long it will sleep ? We know that sleep() will accept a single parameter in the unit of second. But, what is the situation when the sleep() calls nanosleep() ? Let us check the function prototype of nanosleep(). int nanosleep(const struct timespec *req, struct timespec *rem); struct timespec { time_t tv_sec; long tv_nsec; };

/* seconds */ /* nanoseconds */

Well, the nanosleep() accepts 2 parameters in the format of timespec structure. Now, we need to know a little bit about the calling convention in MIPS environment. In general, the first 4 paramenters will be stored in register $a0 to $a3, and the rest will be in stack. So, while nanosleep() accepts 2 parameters, then they are should be in $a0 and $a1. Let us check now. (gdb) info reg zero R0 00000000 t0 R8 00000031 s0 R16 000003e8 t8 R24 000000be status 00001c13 fcsr 00000000

at fffffff8 t1 2ab67540 s1 00020000 t9 2ac05084 lo 0002af19 fir 01739300

v0 00000204 t2 00000000 s2 00000000 k0 00000000 hi 000001e5 restart 00000204

v1 00000000 t3 7fb73878 s3 7fb73800 k1 00000000 badvaddr 2ac9a570

a0 7fb737f8 t4 ffffffff s4 7fb737f8 gp 2acc9950 cause 50808020

a1 7fb737f8 t5 7fb737d0 s5 7fb73880 sp 7fb737e0 pc 2ac050a8

a2 00000000 t6 2aab1b84 s6 80000000 s8 2ac05084

a3 00000001 t7 00000000 s7 7fffffff ra 2ac04d8c

Fine, both $a0 and $a1 share a single parameter, 7fb737f8. So, we should dump the address and it should be the timespec structure. Let's verify. (gdb) x/16x $a0 0x7fb737f8: 0x000003d3 0x344b26ae 0x7fb73808: 0x00000000 0x00000000 0x7fb73818: 0x00000000 0x00000000 0x7fb73828: 0x00000000 0x00000000 (gdb) x/16d $a0 0x7fb737f8: 979 877340334 131072 0x7fb73808: 0 0 0 0 0x7fb73818: 0 0 0 0 0x7fb73828: 0 0 0 0

0x00020000 0x00000000 0x00000000 0x00000000 0

0x00000000 0x00000000 0x00000000 0x00000000

Yes, it seems the not responding process will sleep for 979 seconds or 877340334 in nano seconds. However, in fact, the not responding process is created by a simple command line. tango3[gdb]# sleep 1000 & 766 So, we suppose to get 1000 seconds in timespec structure, but why we get 979 seconds now ? Let's check again.

(gdb) quit A debugging session is active. Inferior 1 [process 766] will be detached. Quit anyway? (y or n) y Detaching from program: /bin/busybox, process 766 tango3[gdb]# ps PID USER TIME 1 root 0:01 2 root 0:00 3 root 0:00 4 root 0:00 5 root 0:00 41 root 0:00 42 root 0:00 43 root 0:00 46 root 0:00 48 root 0:00 49 root 0:00 61 root 0:00 62 root 0:00 63 root 0:00 64 root 0:00 65 root 0:00 658 root 0:00 662 root 0:00 699 root 0:00 717 root 0:00 724 root 0:00 731 root 0:00 738 root 0:00 739 root 0:00 740 root 0:00 751 root 0:00 773 root 0:00 tango3[gdb]#

COMMAND init [kthreadd] [ksoftirqd/0] [events/0] [khelper] [kblockd/0] [ata/0] [ata_aux] [khubd] [kseriod] [kmmcd] [pdflush] [pdflush] [kswapd0] [kprefetchd] [aio/0] [scsi_eh_0] [scsi_eh_1] [sigmblockd] [sigm_write_cach] /sbin/hotplug2 --no-coldplug --persistent --override --ma /sbin/watchdog -t 20 -T 30 /dev/watchdog -sh /sbin/syslogd -n -m 0 /sbin/klogd -n [rpciod/0] ps

Well, it seems the sleep timer has been expired and the process is returned. We should take an important note here, unlike debuggers in windows environment, while GDB is detached from a process, the process will not be closed. Let us verify now. tango3[gdb]# sleep 1000 & 774 tango3[gdb]# ps PID USER TIME COMMAND 1 root 0:01 init 2 root 0:00 [kthreadd] 3 root 0:00 [ksoftirqd/0] 4 root 0:00 [events/0] 5 root 0:00 [khelper] 41 root 0:00 [kblockd/0] 42 root 0:00 [ata/0] 43 root 0:00 [ata_aux] 46 root 0:00 [khubd] 48 root 0:00 [kseriod] 49 root 0:00 [kmmcd] 61 root 0:00 [pdflush] 62 root 0:00 [pdflush] 63 root 0:00 [kswapd0] 64 root 0:00 [kprefetchd] 65 root 0:00 [aio/0]

658 root 0:00 [scsi_eh_0] 662 root 0:00 [scsi_eh_1] 699 root 0:00 [sigmblockd] 717 root 0:00 [sigm_write_cach] 724 root 0:00 /sbin/hotplug2 --no-coldplug --persistent --override --ma 731 root 0:00 /sbin/watchdog -t 20 -T 30 /dev/watchdog 738 root 0:00 -sh 739 root 0:00 /sbin/syslogd -n -m 0 740 root 0:00 /sbin/klogd -n 751 root 0:00 [rpciod/0] 774 root 0:00 sleep 1000 786 root 0:00 ps tango3[gdb]# ./gdb GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "mips-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Setting up the environment for debugging gdb. No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] /root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file: No breakpoint number 0. (gdb) attach 774 Attaching to process 774 Reading symbols from /bin/busybox...(no debugging symbols found)...done. Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1 0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) detach Detaching from program: /bin/busybox, process 774 (gdb) quit tango3[gdb]# ps PID USER TIME COMMAND 1 root 0:01 init 2 root 0:00 [kthreadd] 3 root 0:00 [ksoftirqd/0] 4 root 0:00 [events/0] 5 root 0:00 [khelper] 41 root 0:00 [kblockd/0] 42 root 0:00 [ata/0] 43 root 0:00 [ata_aux] 46 root 0:00 [khubd] 48 root 0:00 [kseriod] 49 root 0:00 [kmmcd] 61 root 0:00 [pdflush] 62 root 0:00 [pdflush] 63 root 0:00 [kswapd0] 64 root 0:00 [kprefetchd] 65 root 0:00 [aio/0] 658 root 0:00 [scsi_eh_0]

662 root 699 root 717 root 724 root 731 root 738 root 739 root 740 root 751 root 774 root 791 root tango3[gdb]#

0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00 0:00

[scsi_eh_1] [sigmblockd] [sigm_write_cach] /sbin/hotplug2 --no-coldplug --persistent --override --ma /sbin/watchdog -t 20 -T 30 /dev/watchdog -sh /sbin/syslogd -n -m 0 /sbin/klogd -n [rpciod/0] sleep 1000 ps

Well, the process being attached will not exit when it is detached from GDB. So, by attach and detach the sleeping process for several times, we should realize something meaningful. tango3[gdb]# sleep 1000 & 795 tango3[gdb]# ./gdb GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "mips-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Setting up the environment for debugging gdb. No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] /root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file: No breakpoint number 0. (gdb) attach 795 Attaching to process 795 Reading symbols from /bin/busybox...(no debugging symbols found)...done. Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1 0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) x/16d $a0 0x7fc837f8: 985 817127037 131072 0 0x7fc83808: 0 0 0 0 0x7fc83818: 0 0 0 0 0x7fc83828: 0 0 0 0 (gdb) detach Detaching from program: /bin/busybox, process 795 (gdb) attach 795 Attaching to program: /bin/busybox, process 795 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1

0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) x/16d $a0 0x7fc837f8: 942 6108777 131072 0 0x7fc83808: 0 0 0 0 0x7fc83818: 0 0 0 0 0x7fc83828: 0 0 0 0 (gdb) detach Detaching from program: /bin/busybox, process 795 (gdb) attach 795 Attaching to program: /bin/busybox, process 795 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1 0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) x/16d $a0 0x7fc837f8: 911 935667926 131072 0 0x7fc83808: 0 0 0 0 0x7fc83818: 0 0 0 0 0x7fc83828: 0 0 0 0 (gdb) Nice, it seems the values in timespec structure keep deducting even though the process itself is being debugged. So, we can say that once the counter in timespec reach to 0, the process will automatically exit once we detach the process from GDB. Let's verify. (gdb) detach Detaching from program: /bin/busybox, process 795 (gdb) attach 795 Attaching to program: /bin/busybox, process 795 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/ld.so.1 0x2ac050a8 in nanosleep () from /lib/libc.so.6 (gdb) x/16d $a0 0x7fc837f8: 0 652534296 131072 0 0x7fc83808: 0 0 0 0 0x7fc83818: 0 0 0 0 0x7fc83828: 0 0 0 0 (gdb) detach Detaching from program: /bin/busybox, process 795 (gdb) attach 795 Attaching to program: /bin/busybox, process 795 ptrace: No such process. (gdb) Yes, we are in the right way. Now, let's have a look to source level debugging with GDB. Before that, let us create a simple cawan.c first. Host:# cat cawan.c #include<stdio.h> main() { int flag=0; char *string="cawan"; if(flag==0) printf("hello world %s\n",string); else

printf("You should not see this !\n"); } Host:# mips-linux-gnu-gcc -EL -g -o cawan cawan.c Host:# ls cawan* cawan cawan.c Host:# Well, we have cross compiled the cawan.c with debugging symbols, and the executable is cawan. Let us run it in MIPS environment with GDB. tango3[gdb]# ./gdb GNU gdb (GDB) 7.1 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "mips-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Setting up the environment for debugging gdb. No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] No symbol table is loaded. Use the "file" command. Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal] /root/cawan/mips-4.3/bin/gdb-7.1/gdb/.gdbinit:8: Error in sourced command file: No breakpoint number 0. (gdb) file cawan Reading symbols from /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan...done. (gdb) list During symbol reading, DW_AT_name missing from DW_TAG_base_type. (gdb) list 1 #include<stdio.h> 2 main() 3 { 4 int flag=0; 5 char *string="cawan"; 6 if(flag==0) 7 printf("hello world %s\n",string); 8 else 9 printf("You should not see this !\n"); 10 } (gdb) run Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan hello world cawan Program exited with code 022. (gdb) Good, the executable runs as usual. Let us set a breakpoint at line 7. (gdb) b 7 Breakpoint 1 at 0x40068c: file cawan.c, line 7. (gdb) run Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan Breakpoint 1, main () at cawan.c:7 7 printf("hello world %s\n",string); (gdb) p string During symbol reading, incomplete CFI data; unspecified registers (e.g., zero) at 0x400670.

$1 = 0x400850 "cawan" (gdb) n hello world cawan 10 } (gdb) Nice, once the breakpoint be hit, we can check the value of "string", and it is definitely equal to "cawan" as what we have defined in proper. Now, we make another breakpoint at line 6, and run the executable again. (gdb) b 6 Breakpoint 2 at 0x400680: file cawan.c, line 6. (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /root/cawan/mips-4.3/bin/gdb-7.1/gdb/cawan Breakpoint 2, main () at cawan.c:6 6 if(flag==0) (gdb) p flag During symbol reading, incomplete CFI data; unspecified registers (e.g., zero) at 0x400670. $3 = 0 (gdb) set variable flag = 1 (gdb) n 9 printf("You should not see this !\n"); (gdb) n You should not see this ! 10 } (gdb) Excellent, we can simply change the value of "flag" and let the executable runs in different path, which is not the normal track. Well, that's all about our discussion of GDB in performing local debugging. Then how about remote debugging ? Let's try and we expect it works as usual. tango3[gdb]# ./gdbserver/gdbserver :8888 ./cawan Process ./cawan created; pid = 756 Listening on port 8888 Well, the gdbserver should be running at the MIPS platform now. Let's connect it from the host machine with mips-linux-gnu-gdb. Host:# mips-linux-gnu-gdb ./cawan GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu". For bug reporting instructions, please see: <https://support.codesourcery.com/GNUToolchain/>... (gdb) target remote 192.168.10.2:8888 Remote debugging using 192.168.10.2:8888 warning: Architecture rejected target-supplied description warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x2aaa8810 in ?? () (gdb) list 1 #include<stdio.h>

2 main() 3 { 4 int flag=0; 5 char *string="cawan"; 6 if(flag==0) 7 printf("hello world %s\n",string); 8 else 9 printf("You should not see this !\n"); 10 } (gdb) c Continuing. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld.so.1: No such file or directory. Program exited with code 022. (gdb) Nice, it seems the executable has been run in MIPS environment. Let's check the MIPS console. Remote debugging from host 192.168.10.1 hello world cawan Child exited with status 18 GDBserver exiting tango3[gdbserver]# Good, the executable has been run once. In order to allow the target executable to restart without restarting the GDBServer, we should use "target extended-remote" instead of "target remote". So, let us do the same thing as what we did in local debugging section. First, run the GDBServer at MIPS environment. tango3[gdbserver]# ./gdbserver :8888 ./cawan Process ./cawan created; pid = 762 Listening on port 8888 Then, at host machine, connect to MIPS environment with mips-linux-gnu-gdb. Host:# mips-linux-gnu-gdb ./cawan GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu". For bug reporting instructions, please see: <https://support.codesourcery.com/GNUToolchain/>... (gdb) target extended-remote 192.168.10.2:8888 Remote debugging using 192.168.10.2:8888 warning: Architecture rejected target-supplied description warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. 0x2aaa8810 in ?? () (gdb) list 1 #include<stdio.h> 2 main() 3 { 4 int flag=0;

5 char *string="cawan"; 6 if(flag==0) 7 printf("hello world %s\n",string); 8 else 9 printf("You should not see this !\n"); 10 } (gdb) b 7 Breakpoint 1 at 0x40068c: file cawan.c, line 7. (gdb) c Continuing. Program received signal SIGINT, Interrupt. 0x2aaa8810 in ?? () (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan warning: Architecture rejected target-supplied description warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld.so.1: No such file or directory. [Switching to Thread 763] Breakpoint 1, main () at cawan.c:7 7 printf("hello world %s\n",string); (gdb) p string $1 = 0x400850 "cawan" (gdb) b 6 Breakpoint 2 at 0x400680: file cawan.c, line 6. (gdb) c Continuing. Program exited with code 022. (gdb) run Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan warning: Architecture rejected target-supplied description warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld.so.1: No such file or directory. [Switching to Thread 764] Breakpoint 2, main () at cawan.c:6 6 if(flag==0) (gdb) p flag $2 = 0 (gdb) set variable flag = 1 (gdb) n 9 printf("You should not see this !\n"); (gdb) n 10 } (gdb) Let's verify the message output at MIPS console.

Remote debugging from host 192.168.10.1 Killing all inferiors Process ./cawan created; pid = 763 hello world cawan Child exited with status 18 Process ./cawan created; pid = 764 You should not see this ! Well, it seems we can do the same thing as in local debugging mode. However, if we do something unknown by the GDBServer, the GDB at the host machine will get into infinity loop... and there are quite a number of such booby traps. Let's show an example when we try to step into printf(). (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: /home/smp383/mips-4.3/bin/testgdb/gdb-7.0/gdb/gdbserver/cawan warning: Architecture rejected target-supplied description warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code. warning: .dynamic section for "/lib/libc.so.6" is not at the expected address (wrong library or version mismatch?) Error while mapping shared library sections: /lib/ld.so.1: No such file or directory. [Switching to Thread 765] Breakpoint 2, main () at cawan.c:6 6 if(flag==0) (gdb) n Breakpoint 1, main () at cawan.c:7 7 printf("hello world %s\n",string); (gdb) s warning: GDB can't find the start of the function at 0x2aabe06c. GDB is unable to find the start of the function at 0x2aabe06c and thus can't determine the size of that function's stack frame. This means that GDB may be unable to access that stack frame, or the frames below it. This problem is most likely caused by an invalid program counter or stack pointer. However, if you think GDB should simply search farther back from 0x2aabe06c for code which looks like the beginning of a function, you can increase the range of the search using the `set heuristic-fence-post' command. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c. warning: GDB can't find the start of the function at 0x2aabe06b. warning: GDB can't find the start of the function at 0x2aabe06c.

Yes, it is quite annoying whenever we are concentrating to troubleshoot our binary, or designing our shellcode for embedded linux system. So, again, it is really recommended to use local debugging method, instead of the remote debugging, under the embedded platform. Besides, it is important to note that the version of cross compiler being used should match to the dedicated version of GDB. For our example here, the cross compiler that we are using is CodeSourcery v4.3, and it should work together with GDB v7.0. Otherwise, if we use CodeSourcery v4.3 to compile GDB v7.1, we will get the error as similar to this, tango3[gdbserver]# ./gdbserver :8888 ./cawan Process ./cawan created; pid = 757 Listening on port 8888 Host:# mips-linux-gnu-gdb ./cawan GNU gdb (Sourcery G++ Lite 4.3-51) 6.8.50.20080821-cvs Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=mips-linux-gnu". For bug reporting instructions, please see: <https://support.codesourcery.com/GNUToolchain/>... (gdb) target remote 192.168.10.2:8888 Remote debugging using 192.168.10.2:8888 warning: Architecture rejected target-supplied description Malformed packet(b) (missing colon): ore:0; Packet: 'T051d:30bba07f;25:1088aa2a;thread:2f5;core:0;' (gdb) So, the latest version of a tool doesn't mean it will always work in embedded system. Instead, the matching of the version number among each others should always be alert.