strace for Web Developers: Seeing What Your App Really Asks the Kernel
When your profiler shows nothing and your logs are silent, strace shows you every syscall with timings. Here is a practical subset worth knowing.
Most application debugging happens at the level of your language. Stack traces, profilers, logs. That works until the problem is in the space between your process and the operating system, at which point all of those go quiet at exactly the same time.
strace shows you every system call a process makes. It is the tool for the case where your code is doing nothing and taking a long time to do it.
The one command worth memorising
strace -f -T -p <pid>
-ffollow forked children and threads, essential for anything with a worker pool-Tprint the time spent in each syscall, which is the whole point-pattach to a running process
The output is a firehose. Filtering is what makes it usable:
strace -f -T -p <pid> -e trace=network,openat 2>&1 | grep -v EAGAIN
Tested on Linux 6.8 with strace 6.8. On macOS the equivalent is dtruss, which needs SIP disabled and is generally more painful; on a Mac I run the process in a Linux container instead.
Why it finds things profilers cannot
A CPU profiler samples on-CPU stacks. A thread blocked in a syscall is not on CPU, so it is invisible.
This produces a specific and confusing situation: latency is up, the flame graph is unchanged, and total CPU per request has not moved. That combination means the time is being spent waiting, and waiting means a lock, disk, or the network.
That is exactly the situation where strace earns its place. A 400ms latency spike I spent an hour on came down to ten minutes with strace once I accepted the profiler was telling me something rather than nothing. The output showed a poll timing out after five seconds on a UDP socket to port 53, and that was the whole answer.
The syscalls worth recognising
You do not need to know all 350. About fifteen cover most application behaviour.
Files
openat(AT_FDCWD, "/app/config.yaml", O_RDONLY) = 3 <0.000042>
read(3, "server:\n port: 8080\n", 4096) = 21 <0.000011>
close(3) = 0 <0.000008>
The number after = is the return value, and for openat it is the file descriptor. -1 ENOENT means the file does not exist, which is how you find configuration your app is looking for in a path you did not expect.
Network
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 7 <0.000023>
connect(7, {sa_family=AF_INET, sin_port=htons(5432), ...}) = -1 EINPROGRESS <0.000067>
poll([{fd=7, events=POLLOUT}], 1, 5000) = 1 <0.041203>
sendto(7, "...", 60, 0, NULL, 0) = 60 <0.000038>
recvfrom(7, "...", 2048, 0, NULL, NULL) = 284 <0.402118>
EINPROGRESS on a connect is normal for non blocking sockets. The interesting number is the time on poll and recvfrom, because that is real waiting.
Locks and waiting
futex(0x7f2c1c0, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 <2.104882>
A futex with a long duration is lock contention. If your output is dominated by futex waits, you have a concurrency problem rather than an I/O problem.
Memory
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c...
brk(0x55a8f2c00000) = 0x55a8f2c00000
Frequent large mmap calls mean the allocator is repeatedly asking the kernel for memory, which often indicates a leak or fragmentation.
Practical recipes
Which files is it opening, and which are missing
strace -f -e trace=openat -p <pid> 2>&1 | grep ENOENT
Answers "why is it not picking up my config file" in about five seconds. The app is looking somewhere you did not expect, and this shows you exactly where.
What is it talking to
strace -f -e trace=connect,sendto,recvfrom -T -p <pid> 2>&1 | grep -v EAGAIN
Every outbound connection with timings. This is how you discover that your service is making a call you did not know about, or that a library is phoning home.
Filtering out EAGAIN matters because non blocking sockets generate enormous volumes of it and it is almost always noise.
Where is the time going
strace -f -c -p <pid>
# then Ctrl-C after 30 seconds
-c gives you a summary table instead of a stream:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
71.42 12.408291 4136 3000 poll
18.03 3.132447 52 60240 recvfrom
6.11 1.061308 35 30120 sendto
2.44 0.423992 14 30280 1204 futex
This is the fastest way to characterise a process. Seventy percent of wall time in poll means waiting on I/O. Seventy percent in futex means lock contention. High counts with small durations mean syscall overhead, which is its own problem.
Catching a startup failure
You cannot attach to a process that dies in 200 milliseconds. Launch under strace instead:
strace -f -o /tmp/trace.log ./myapp
tail -50 /tmp/trace.log
The last few syscalls before exit usually name the file, socket, or permission that failed. This is my go-to for a container that crashes immediately with no useful log output.
Inside a container
You need the capability:
docker run --cap-add=SYS_PTRACE ...
Or from outside, attach to the process in the container's namespaces:
docker run -it --rm --pid=container:myapp --cap-add=SYS_PTRACE \
nicolaka/netshoot strace -f -T -p 1
In Kubernetes, an ephemeral debug container with --target shares the PID namespace, which is the general pattern for debugging containers with no tooling in them:
kubectl debug -it mypod --image=nicolaka/netshoot --target=app
strace -f -T -p 1
The cost, and when not to use it
strace is not free. It uses ptrace, which means every traced syscall traps into the kernel twice, once on entry and once on exit, with a context switch to the tracer each time.
The overhead is substantial: a syscall heavy process can slow down by an order of magnitude. On a busy production service that is enough to cause timeouts and cascading failures.
Rules I follow:
One pod out of many, briefly. Never trace every instance. Take one out of the load balancer if you can.
Filter aggressively. -e trace=network traces far fewer calls than the default, and the overhead scales with what you trace.
Use -c for characterisation. The summary mode is cheaper than streaming and usually tells you enough to know where to look next.
Prefer eBPF tools in production. bpftrace, bcc, and the various *snoop tools do the same job with dramatically lower overhead because they run in the kernel instead of trapping to userspace. opensnoop, tcpconnect, and execsnoop are direct replacements for common strace recipes.
If you are on a modern kernel and have the tooling available, bpftrace is the better default for production and strace remains easier for a quick look at one process.
The mental model to keep
Your application is a sequence of syscalls with computation between them. Everything the process does that affects the outside world goes through that interface: files, sockets, memory, processes, time.
When your language level tools cannot see the problem, it is usually because the problem lives at that boundary. strace is how you look at it directly, and the reason it feels like a superpower the first few times is that most of us spend our careers one or two abstraction layers above where the interesting failure actually is.
