How Breakpoints Actually Work: INT3, ptrace, and the Debugger Contract

You set a breakpoint by clicking in a gutter and the program stops there. It feels like the debugger is watching your code, supervising execution, checking at each instruction whether it has arrived at the interesting line.

It is not, and it could not be. Watching every instruction would make execution thousands of times slower. What actually happens is stranger and much more elegant: the debugger edits your program's machine code while it is running.

Understanding this explains a lot of otherwise mysterious debugger behaviour, including why conditional breakpoints can be slow, why optimised builds put breakpoints on the wrong line, and why you sometimes cannot inspect a variable that is obviously in scope.

The one byte trick

On x86 there is an instruction called INT3. It is one byte, 0xCC, and its entire purpose is to raise a breakpoint trap. It exists specifically so debuggers can do this.

To set a breakpoint at an address, the debugger reads the byte currently there, saves it, and writes 0xCC over it.

That is the whole mechanism. The program then runs at full native speed with no supervision and no per instruction checking. The CPU executes normally until it reaches that address, executes INT3, traps into the kernel, and the kernel delivers SIGTRAP to the process, stopping it and notifying the debugger.

Concretely, say the original code is:

0x401136:  48 89 e5     mov  %rsp,%rbp
0x401139:  89 7d fc     mov  %edi,-0x4(%rbp)

Set a breakpoint at 0x401139 and memory becomes:

0x401136:  48 89 e5     mov  %rsp,%rbp
0x401139:  CC 7d fc     int3 ; (garbage)

The debugger saved 0x89. Note it only replaced the first byte of a three byte instruction. The rest is now nonsense, and it never executes because INT3 traps first.

Resuming is the fiddly part

To continue, the debugger cannot simply resume, because the 0xCC is still there and the real instruction has not run. So it:

1. Writes the saved byte back
2. Rewinds the instruction pointer by one, since the CPU incremented past the INT3 and now points mid instruction
3. Single steps one instruction, so the restored instruction executes
4. Writes 0xCC back so the breakpoint works next time
5. Continues

Step two is a classic source of debugger bugs. On x86 the trap is reported after the instruction, so the off by one correction is mandatory. Steps three and four are why breakpoints in a hot loop hurt, since each hit costs several context switches between the process and the debugger even if you resume immediately.

Who is allowed to do this

Writing to another process's memory is not something the OS permits casually. On Linux the mechanism is the ptrace system call.

ptrace(PTRACE_ATTACH, pid, NULL, NULL);
ptrace(PTRACE_PEEKTEXT, pid, addr, NULL);       // read a word
ptrace(PTRACE_POKETEXT, pid, addr, data);       // write a word
ptrace(PTRACE_CONT, pid, NULL, NULL);           // resume

ptrace establishes a tracer and tracee relationship. Once attached, the tracee stops on every signal it receives and the kernel notifies the tracer through waitpid. The debugger's main loop is essentially:

loop:
    waitpid()              -> tracee stopped, why?
    if SIGTRAP at a known breakpoint address:
        tell the user, wait for a command
    else:
        pass the signal through

This is also why debugging often needs elevated permissions. Most distributions set ptrace_scope to 1, which allows attaching only to direct descendants. Attaching to an already running process by PID typically needs root or the CAP_SYS_PTRACE capability. Inside a container you need --cap-add=SYS_PTRACE, and if you have ever wondered why gdb -p fails in Docker with no useful message, that is why.

macOS uses Mach exception ports instead and requires code signing entitlements, which is why attaching to system binaries on macOS is often impossible even as root.

From a line number to an address

The debugger works with addresses. You clicked on line 42 of main.c. The bridge between those is debug information, usually DWARF, embedded in the binary when you compile with -g.

DWARF contains a line number table mapping machine addresses to file, line, and column. It is encoded as a state machine rather than a flat table, but conceptually:

Address     File     Line  Col   is_stmt
0x401136    main.c   41    1     yes
0x401139    main.c   42    5     yes
0x401142    main.c   42    18    no
0x40114b    main.c   43    5     yes

To place a breakpoint on line 42, the debugger finds the lowest address whose line is 42 and which is marked is_stmt, meaning a statement boundary where the machine state genuinely corresponds to "about to execute line 42".

Two consequences you have definitely experienced.

Breakpoints move. Set one on a blank line or a comment and the debugger silently relocates it to the next line with code, because no address maps to your line. Set one on a line the optimiser deleted and it moves somewhere surprising.

Optimised builds behave badly. With -O2 the compiler inlines, reorders, and merges. One address can map to several source lines from different functions, and one source line can map to a dozen scattered addresses. The line table records this faithfully, so stepping jumps around and variables read as optimised out, because the value only ever lived in a register that has since been reused. The debugger is telling the truth. The code you are running is not the code you wrote.

Hardware breakpoints and watchpoints

Software breakpoints require writing to memory, which fails for code in ROM and cannot detect data access at all. You cannot put an INT3 on a read.

x86 provides four debug registers, DR0 through DR3, holding addresses, plus DR7 controlling what each watches: execute, write, or read and write, at 1, 2, 4, or 8 byte granularity. The CPU compares every memory access against these registers in hardware with no overhead until one matches.

This is what a watch expression in GDB uses, and it is dramatically faster than the alternative, which is single stepping the whole program and comparing the value after each instruction. That is what a software watchpoint means and it can be a thousand times slower. If GDB tells you it is using a software watchpoint, you have run out of debug registers or asked to watch a region too large for one.

Four is the hard limit. Watch a fifth thing and something has to give.

Conditional breakpoints and why they can be slow

break process_order if order_id == 4471

The naive implementation is a normal INT3, and each time it fires the debugger reads order_id from the tracee and decides whether to stop.

The cost is the round trip. Every hit means a trap into the kernel, a signal, a waitpid wake up, several ptrace reads, a byte restore, a single step, a byte rewrite, and a resume. Call it tens of microseconds. If that breakpoint sits in a function called a million times before your condition is true, you have added tens of seconds, and that is why a conditional breakpoint in a hot path can make a program appear to hang.

Two better options exist. Ignore counts are evaluated without expression evaluation so they are cheaper, though they still pay the trap cost. And GDB can compile a simple condition into bytecode and, with an in process agent, evaluate it inside the tracee with no context switch unless it is true. A big win where available, limited to simple expressions.

Logpoints, meaning breakpoints that print and continue, have the same cost profile. In V8 and other managed runtimes they can be much cheaper because the runtime patches bytecode directly rather than trapping to a separate process.

Managed runtimes work differently

Everything above is native code. In V8, the JVM, or CPython, the debugger is part of the runtime.

V8 and Node expose the Chrome DevTools Protocol. A breakpoint tells V8 to deoptimise the containing function from optimised machine code back to bytecode and insert a check. This is why performance changes when the inspector is attached: functions containing breakpoints stop being JIT optimised. It is also why breakpoints behave oddly in code that has been inlined.

CPython used a per line trace callback through sys.settrace, which fired on every line of every function and made debugged code roughly a hundred times slower. Python 3.12's sys.monitoring fixed this properly with per event, per location callbacks, so a breakpoint costs nothing where it is not set. That is why debuggers got dramatically faster in recent Python versions.

The JVM uses JVMTI, with the JIT deoptimising affected methods much like V8.

The theme is that managed runtimes have a supported hook so nobody patches machine code, but you still pay by losing optimisation in the affected function.

Why knowing this is useful

Several things become predictable once you have the model.

A breakpoint in a tight loop is expensive. Put the condition on an outer, less frequent function, or use an ignore count.

You get four watchpoints. If you need more, use hardware watchpoints for the two that matter and instrument the rest in code.

Optimised out is not a bug. Rebuild with -Og, which keeps most optimisations while preserving debuggability.

ptrace is exclusive. One tracer per process, which is why you cannot attach two debuggers and why strace and gdb fight over the same target.

Attaching inside a container needs SYS_PTRACE.

A debugger changes the program. Not just timing, but the actual bytes in memory or the optimisation tier of a function. For a race condition that difference is often the entire reason it stops reproducing.

That last one deserves emphasis, because it demystifies the heisenbug. When you attach a debugger you deoptimise functions, add microseconds of trap overhead at each breakpoint, and serialise thread scheduling around signal delivery. A race depending on two threads arriving within nanoseconds of each other will not survive that. The bug is not being shy. You genuinely changed the conditions of the experiment.

Which is a large part of the argument for print debugging in concurrent code.