GDB Insight of Threads

Looking at thread state inside gdb

While helping a friend debug a multi-threaded application hang on linux, using gdb. She was facing difficulty in pointing out exactly which were the threads that were in blocked state. Getting thread info only tells us what the thread ids are and the routine they are executing.
(gdb) info threads
12 Thread 4398053392560 (LWP 7234) 0x0000000010005740 in xxxx ()
11 Thread 4398054441136 (LWP 7235) 0x0000000010005740 in xxxx ()
10 Thread 4398055489712 (LWP 7237) 0x0000000010005748 in xxxx ()
9 Thread 4398063878320 (LWP 8263) 0x0000000010000768 in yyy ()
8 Thread 4398062829744 (LWP 8267) 0x0000000010003624 in zzz ()
7 Thread 4398061781168 (LWP 8270) 0x0000000010003624 in zzz ()
6 Thread 4398060732592 (LWP 8273) 0x0000000010001ce4 in ddd ()
5 Thread 4398059684016 (LWP 8276) 0x0000000010000750 in fff ()
4 Thread 4398058635440 (LWP 8277) 0x000000001001e128 in ttt ()
3 Thread 4398057586864 (LWP 8278) 0x0000000010003624 in zzz ()
2 Thread 4398056538288 (LWP 8281) 0x000000001001e140 in rrrr ()
1 Thread 269578240 (LWP 28151) 0x000000001000cba4 in www ()
She complained that in another proprietary debugger she has worked with in the past, she could easily also get the thread status information as well. Found that on gdb, we need to use the following commands to get the status information of the threads:
(gdb) help info proc
Show /proc process information about any running process.
Specify any process id, or use the program being debugged by default.
Specify any of the following keywords for detailed info:
mappings -- list of mapped memory regions.
stat -- list a bunch of random process info.
status -- list a different bunch of random process info.
all -- list all available /proc info.

(gdb) info proc 7234 stat
process 7234
cmdline = '/usr/bin/xyz'
warning: unable to read link '/proc/7234/cwd'
warning: unable to read link '/proc/7234/exe'
Process: 7234
Exec file: xyz
State: S
Parent process: 3230
Process group: 7234
Session id: 7234
TTY: 0
TTY owner process group: -1
Flags: 0x402040
Minor faults (no memory page): 96
Minor faults, children: 0
Major faults (memory page faults): 0
Major faults, children: 0
utime: 0
stime: 0
utime, children: 0
stime, children: 0
jiffies remaining in current time slice: 20
'nice' value: 0
jiffies until next timeout: 1
jiffies until next SIGALRM: 0
start time (jiffies since system boot): 4371
Virtual memory size: ...
Resident set size: ....
rlim: ......
Start of text: 0xb8013000
End of text: 0xb8025ca4
Start of stack: 0xbf925630

(gdb) info proc 7234 status
process 7234
cmdline = '/usr/bin/xyz'
warning: unable to read link '/proc/7234/cwd'
warning: unable to read link '/proc/7234/exe'
Name: xyz
State: S (sleeping)
Tgid: 7234
Pid: 7234
PPid: 3230
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 109 1000
FDSize: 32
Groups: 4 20 24 46 106 117 118 123 124 1000
VmPeak: 4784 kB
VmSize: 4784 kB
VmLck: 0 kB
VmHWM: 636 kB
VmRSS: 636 kB
VmData: 380 kB
VmStk: 84 kB
VmExe: 76 kB
VmLib: 4020 kB
VmPTE: 20 kB
Threads: 1
SigQ: 0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001002
SigCgt: 0000000180004001
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed: 00000000,00000003
Cpus_allowed_list: 0-1
Mems_allowed: 1
Mems_allowed_list: 0
voluntary_ctxt_switches: 627
nonvoluntary_ctxt_switches: 0

So as we can see, we can get a whole lot of information about the threads and not just the state. There are other options supported by the info proc command as shown by the help. gdb picks up the above information from the /proc/<pid> interface. It helps to be able to look at all the pieces of debug information from the same interface itself.

Debugging Individual Threads

#include <stdio.h>
#include <pthread.h>
#include <unistd.h>

pthread_t thread;

void* thread3 (void* d)
{
  int count3 = 0;

  while(count3 < 1000){
    sleep(10);
    printf("Thread 3: %d\n", count3++);
  }
  return NULL;
}

void* thread2 (void* d)
{
  int count2 = 0;

  while(count2 < 1000){
    printf("Thread 2: %d\n", count2++);
  }
  return NULL;
}

int main (){

  pthread_create (&thread, NULL, thread2, NULL);
  pthread_create (&thread, NULL, thread3, NULL);
  
  //Thread 1
  int count1 = 0;

  while(count1 < 1000){
    printf("Thread 1: %d\n", count1++);
  }

  pthread_join(thread,NULL);
  return 0;
}

Compile this program in order to examine it under GDB.

gcc -g three-threads.c -o three-threads  -lpthread
gdb ./three-threads

First set breakpoints on all thread functions; thread1, thread2, and main.

(gdb) break thread3
Breakpoint 1 at 0x4006c0: file three-threads.c, line 9.
(gdb) break thread2
Breakpoint 2 at 0x40070c: file three-threads.c, line 20.
(gdb) break main
Breakpoint 3 at 0x40074a: file three-threads.c, line 30.

Then run the program.

(gdb) run
[...]
Breakpoint 3, main () at three-threads.c:30
30   pthread_create (&thread, NULL, thread2, NULL);
[...]
(gdb) info threads
* 1 Thread 0x7ffff7fd5720 (LWP 4620)  main () at three-threads.c:30
(gdb) 

Note that the command info threads provides a summary of the program's threads and some details about their current state. In this case there is only one thread that has been created so far.

Continue execution some more.

(gdb) next
[New Thread 0x7ffff7fd3710 (LWP 4687)]
31   pthread_create (&thread, NULL, thread3, NULL);
(gdb) 
Breakpoint 2, thread2 (d=0x0) at three-threads.c:20
20   int count2 = 0;
next
[New Thread 0x7ffff75d2710 (LWP 4688)]
34   int count1 = 0;
(gdb) 
Breakpoint 1, thread3 (d=0x0) at three-threads.c:9
9   int count3 = 0;
info threads
  3 Thread 0x7ffff75d2710 (LWP 4688)  thread3 (d=0x0) at three-threads.c:9
  2 Thread 0x7ffff7fd3710 (LWP 4687)  thread2 (d=0x0) at three-threads.c:20
* 1 Thread 0x7ffff7fd5720 (LWP 4620)  main () at three-threads.c:34

Here, two more threads are created. The star indicates the thread currently under focus. Also, the newly created threads have hit the breakpoint set for them in their initialization functions. Namely, thread2() and thread3().

To begin real thread debugging, use the thread <thread number> command to switch the focus to another thread.

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff7fd3710 (LWP 4687))]#0  thread2 (d=0x0)
    at three-threads.c:20
20   int count2 = 0;
(gdb) list
15   return NULL;
16 }
17 
18 void* thread2 (void* d)
19 {
20   int count2 = 0;
21 
22   while(count2 < 1000){
23     printf("Thread 2: %d\n", count2++);
24   }

Thread 2 stopped at line 20 in its function thread2().

(gdb) next
22   while(count2 < 1000){
(gdb) print count2
$1 = 0
(gdb) next
23     printf("Thread 2: %d\n", count2++);
(gdb) next
Thread 2: 0
22   while(count2 < 1000){
(gdb) next
23     printf("Thread 2: %d\n", count2++);
(gdb) print count2
$2 = 1
(gdb) info threads
  3 Thread 0x7ffff75d2710 (LWP 4688)  thread3 (d=0x0) at three-threads.c:9
* 2 Thread 0x7ffff7fd3710 (LWP 4687)  thread2 (d=0x0) at three-threads.c:23
  1 Thread 0x7ffff7fd5720 (LWP 4620)  main () at three-threads.c:34
(gdb) 

Above, a few lines of thread2 printed the counter count2 and left thread 2 at line 23 as is seen by the output of 'info threads'.

Now thread3.

(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff75d2710 (LWP 4688))]#0  thread3 (d=0x0)
    at three-threads.c:9
9   int count3 = 0;
(gdb) list
4 
5 pthread_t thread;
6 
7 void* thread3 (void* d)
8 {
9   int count3 = 0;
10 
11   while(count3 < 1000){
12     sleep(10);
13     printf("Thread 3: %d\n", count3++);
(gdb) 

Thread three is a little different in that it has a sleep statement and executes slowly. Think of it as a representation of an uninteresting IO thread. Because this thread is uninteresting, continue its execution uninterrupted, using the continue.

(gdb) continue &
(gdb) Thread 3: 0
Thread 3: 1
Thread 3: 2
Thread 3: 3

Take note of the & at the end of the continue. This allows the GDB prompt to return so other commands can be executed. Using the interrupt, execution can be stopped should thread 3 become interesting again.

(gdb) interrupt
[Thread 0x7ffff75d2710 (LWP 4688)] #3 stopped.
0x000000343f4a6a6d in nanosleep () at ../sysdeps/unix/syscall-template.S:82
82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)

It is also possible to go back to the original main thread and examine it some more.

(gdb) thread 1
[Switching to thread 1 (Thread 0x7ffff7fd5720 (LWP 4620))]#0  main ()
    at three-threads.c:34
34   int count1 = 0;
(gdb) next
36   while(count1 < 1000){
(gdb) next
37     printf("Thread 1: %d\n", count1++);
(gdb) next
Thread 1: 0
36   while(count1 < 1000){
(gdb) next
37     printf("Thread 1: %d\n", count1++);
(gdb) next
Thread 1: 1
36   while(count1 < 1000){
(gdb) next
37     printf("Thread 1: %d\n", count1++);
(gdb) next
Thread 1: 2
36   while(count1 < 1000){
(gdb) print count1 
$3 = 3
(gdb) info threads 
  3 Thread 0x7ffff75d2710 (LWP 4688)  0x000000343f4a6a6d in nanosleep ()
    at ../sysdeps/unix/syscall-template.S:82
  2 Thread 0x7ffff7fd3710 (LWP 4687)  thread2 (d=0x0) at three-threads.c:23
* 1 Thread 0x7ffff7fd5720 (LWP 4620)  main () at three-threads.c:36
(gdb) 

As can be seen from the output of info threads, the other threads are where they were left, unaffected by the debugging of thread 1.

Software Academy

GDB CheatSheet

Running

#gdb     <program> [core dump]
                     Start GDB (with optional core dump).
# gdb    --args <program> <args…>
                    Start GDB and pass arguments
# gdb    --pid <pid>
                    Start GDB and attach to process.
set args <args...>
                     Set arguments to pass to program to be debugged.
run                Run the program to be debugged.
kill                Kill the running program.

Breakpoints

break <where>
                   Set a new breakpoint.
delete <breakpoint#>
                   Remove a breakpoint.
clear
                   Delete all breakpoints.
enable <breakpoint#>
                   Enable a disabled breakpoint.
disable <breakpoint#>
                   Disable a breakpoint.

Watchpoints

watch <where>
                    Set a new watchpoint.
delete/enable/disable <watchpoint#>
                    Like breakpoints.

<where>

function_name
                       Break/watch the named function.
line_number
                       Break/watch the line number in the current source file.
file:line_number
                       Break/watch the line number in the named source file.

Conditions

break/watch <where> if <condition>
                        Break/watch at the given location if the condition is met.
                        Conditions may be almost any C expression that evaluate to true or false.
condition <breakpoint#> <condition>
                        Set/change the condition of an existing break- or watchpoint.

Examining the stack

backtrace
where
                        Show call stack.
backtrace full
where full
                        Show call stack, also print the local variables in each frame.
frame <frame#>
                        Select the stack frame to operate on.

Stepping

step
                       Go to next instruction (source line), diving into function.
next
                       Go to next instruction (source line) but donʻt dive into functions.
finish
                       Continue until the current function returns.
continue
                        Continue normal execution.

Variables and memory

print/format <what>
                            Print content of variable/memory location/register.
display/format <what>
                            Like „print“, but print the information after each stepping instruction.
undisplay <display#>
                            Remove the „display“ with the given number.
enable display <display#>
disable display <display#>
                           En- or disable the „display“ with the given number.
x/nfu <address>
                           Print memory.
                           n: How many units to print (default 1).
                           f: Format character (like „print“).
                           u: Unit.
                          Unit is one of:
                                       b: Byte,
                                       h: Half-word (two bytes)
                                       w: Word (four bytes)
                                       g: Giant word (eight bytes)).

Format

a                     Pointer.
c                     Read as integer, print as character.


d                     Integer, signed decimal.
f                      Floating point number.
o                     Integer, print as octal.
s                     Try to treat as C string.
t                      Integer, print as binary (t = „two“).
u                     Integer, unsigned decimal.
x                     Integer, print as hexadecimal.

<what>

expression
                       Almost any C expression, including function calls (must be prefixed with a
                       cast to tell GDB the return value type).
file_name::variable_name
                       Content of the variable defined in the named file (static variables).
function::variable_name
                       Content of the variable defined in the named function (if on the stack).
{type}address
                       Content at address, interpreted as being of the C type type.
$register
                       Content of named register. Interesting registers are $esp (stack pointer), $ebp
                       (frame pointer) and $eip (instruction pointer).

Threads

thread <thread#>
                     Chose thread to operate on.

Manipulating the program

set var <variable_name>=<value>
                       Change the content of a variable to the given value.
return <expression>
                        Force the current function to return immediately,
                        passing the given value.

Sources

directory <directory>
                      Add directory to the list of directories
                       that is searched for sources.
list
list <filename>:<function>
list <filename>:<line_number>
list <first>,<last>
                       Shows the current or given source context.
                       The filename may be omitted. If
                        last is omitted the context starting at
                        start is printed instead of centered around it.
set listsize <count>
                        Show many lines to show in „list“.

Signals

handle <signal> <options>
                            Set how to handle signles. Options are:
                                 (no)print: (Donʻt) print a message when signals occurs.
                                 (no)stop: (Donʻt) stop the program when signals occurs.
                                 (no)pass: (Donʻt) pass the signal to the program.

Informations

disassemble
disassemble <where>
                                 Disassemble the current function or given location.
info args
                                 Print the arguments to the function of the current stack frame.
info breakpoints
                                 Print informations about the break- and watchpoints.
info display
                                 Print informations about the „displays“.
info locals
                                 Print the local variables in the currently selected stack frame.
info sharedlibrary
                                 List loaded shared libraries.
info signals
                                 List all signals and how they are currently handled.
info threads
                                List all threads.
show directories
                                Print all directories in which GDB searches for source files.
show listsize
                                Print how many are shown in the „list“ command.
whatis variable_name
                                Print type of named variable.