Strace is a tool that allows you to peer into the execution of an program, and view it’s interaction with the system and kernel. It is included with most modern *nix distributions and is a wonderful tool to troubleshoot errors.

The most common usage is to attach strace to a running process with the “-p” flag, and specify the PID of the running program:

root@cpdemo [~]# strace -p 28882
Process 28882 attached - interrupt to quit
time([1150494282])                      = 1150494282
stat64(“/var/log/exim_mainlog”, {st_mode=S_IFREG|0640, st_size=73629, …}) = 0
time([1150494282])                      = 1150494282
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0}, {2, 0})               = 0
time([1150494284])                      = 1150494284
stat64(“/var/log/exim_mainlog”, {st_mode=S_IFREG|0640, st_size=73629, …}) = 0
time([1150494284])                      = 1150494284
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({2, 0},  <unfinished …>
Process 28882 detached
root@cpdemo [~]#

The above example details what the output from an strace of a properly running “eximstats” binary would produce.

For binaries such as Apache that fork off child processes, you may also you the “-f” flag to follow those children as well. Most of the time you will want to you the “-o” flag to specify an output file so you may go back through later and see if there are any errors. While some of this may be daunting, much of the time if there is an error, it will be clearly marked with “ERROR” or something similar.