How to Read a Flame Graph in Sixty Seconds

The three rules

Width is time. A box twice as wide consumed twice as much CPU. This is the only thing that matters.

Vertical is call depth. A box sitting on another box means it was called by it. Height means nothing about cost.

Left to right is alphabetical, not chronological. This trips up everyone once. A flame graph is not a timeline.

That is genuinely most of it. Find the widest box near the top that you recognise, and you have found where the time goes.

What it actually represents

A profiler samples the call stack at some frequency, typically 99 or 999 times a second. Each sample is a snapshot of what was executing.

A flame graph aggregates identical stacks and draws each as a box whose width is proportional to how many samples contained it. So a function appearing in 40 percent of samples occupies 40 percent of the width.

The important consequence: the graph shows where CPU time was spent, not what happened when. Two functions side by side may have interleaved throughout the run.

Sampling at 99 Hz rather than 100 is deliberate, to avoid accidentally synchronising with something running at exactly 100 Hz and producing systematically biased samples.

Reading one

Start at the bottom. That is your entry point, usually main or an event loop, and it spans the whole width because everything descends from it.

Work upward. Each layer is a call. Where a box splits into several boxes above it, that function called several things.

Plateaus at the top are where the work happens. A wide flat box with nothing above it means the CPU was executing that function itself, not something it called. Those are your leaf functions and they are where optimisation actually applies.

A tall narrow spike is deep recursion or a long call chain that is not costing much. Interesting structurally, irrelevant for performance.

The question to ask: what is the widest thing I did not expect?

If 60 percent of the width is JSON serialisation and you did not think you were serialising much, that is the finding. If 60 percent is your core computation, the profile is telling you the code is doing what it should and you need an algorithmic change rather than a tweak.

The variants

Icicle graph is the same thing upside down, with the root at the top. Chrome DevTools and several language profilers use this. Same rules, inverted.

Differential flame graph compares two profiles, colouring by increase or decrease. This is the most useful variant for a regression: you profile before and after, and the red boxes are what got slower. Much faster than reading two graphs side by side.

Off-CPU flame graph shows where threads were blocked rather than running. This is the one people do not know about and it answers a question the normal graph cannot.

That distinction matters enormously. A standard CPU profile shows nothing when your service is slow because it is waiting on the network, since a blocked thread is not on CPU and does not get sampled.

I once spent an hour on a latency problem where the flame graph was identical between healthy and unhealthy pods. That identity was the clue: CPU unchanged with latency up means the time is off-CPU, which means waiting, which means a lock, disk, or network.

If your profile looks normal and your service is slow, you need an off-CPU profile or a syscall trace.

Generating them

Linux, any language:

perf record -F 99 -g -p <pid> -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

Or with the modern tooling:

perf record -F 99 -g -p <pid> -- sleep 30
perf script report flamegraph

Node:

node --prof app.js
node --prof-process isolate-*.log > processed.txt

# or better
npx 0x app.js
npx clinic flame -- node app.js

Python:

py-spy record -o profile.svg --pid 1234 --duration 30
py-spy top --pid 1234

py-spy attaches to a running process without modifying it, which makes it excellent for production. It needs SYS_PTRACE in a container.

Go has it built in:

go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

The -http flag opens a browser with a flame graph view, plus a call graph and source annotation. Go's tooling here is the best of any language and it is worth knowing even if you do not write Go, because it sets the expectation.

Java: async-profiler, which handles both CPU and allocation profiling.

Common misreadings

Assuming left to right is time order. Worth stating twice because everyone does it once.

Optimising a wide box that is a framework. If your web framework's request handling is 15 percent of the profile, that is normal and you are not going to fix it. Look for your own code.

Ignoring the many small boxes. A hundred narrow boxes that are all the same function called from different places can collectively dominate. Some tools let you merge by function name, which reveals this.

Profiling the wrong thing. A profile of a process that is mostly idle tells you about the idle path. Generate load, or profile during the actual slow period.

Profiling for too short a period. Thirty seconds of samples at 99 Hz is about 3000 samples, which is enough for a reasonable picture. Three seconds is not.

Missing symbols. Boxes labelled with hex addresses mean the profiler could not resolve function names. You need debug symbols, or --frame-pointer builds, or for JIT languages a symbol map. An unreadable profile is usually a build configuration problem rather than a profiler problem.

What to do with the finding

A flame graph tells you where time goes. It does not tell you why, and the gap between those is where the actual work is.

If the widest box is a database driver, the finding is not "the driver is slow". It is that you are making a lot of database calls, and the next question is whether they are N+1.

If it is JSON serialisation, the question is whether you are serialising more than you need, not whether you should switch libraries.

If it is your own hot loop, you have an algorithmic question.

The mistake I see most is treating the widest box as the thing to optimise. Usually it is a symptom, and the useful move is to ask why that function is being called as much as it is.

Continuous profiling

Worth mentioning because it changes the workflow.

Tools like Pyroscope, Parca, and the cloud providers' equivalents run a low overhead profiler continuously in production and store the history. Overhead is typically low single digit percent.

The benefit is not that you can profile on demand. It is that when something regresses, you can compare against last week automatically. A differential flame graph between two deploys turns "something got slower" into "this function got slower" without needing to reproduce anything.

That is the same argument as tracking build times or query counts over time: slow degradation is invisible unless something is recording it, and by the time it is obvious the cause is buried under months of commits.