The strace Output That Showed the File Was Never Opened

I would like to say the path was the problem, because that would be a shorter post, but it started at the ignored.

Strip the syscall trace as the ground truth down and one property makes it decisive: the trace shows what the process actually asked the kernel for, so it settles the argument between what the file says and what the program did.

I have since met the same mechanics in plenty of process whose file access can be observed at the syscall level.

Why the logs could not answer

The log said the config was loaded, and the loaded was the assumption, and the assumption was the not the evidence, and the not evidence was the default's fallback, and the fallback was the silent, and the silent was the no error, and the no error was the mystery, because the program did not report the missing file, it reported the loaded config, and the loaded was the default.

The file was the read by the human, and the human's read was the correct, and the correct was the irrelevant, because the question was the process's read, and the process's read was the syscall, and the syscall was the trace, and the trace was the answer, and the answer was the path's difference.

The layers are the program and the library and the kernel, and the three are the different views, and the different views are the disagreement, and the disagreement is the hour, and the hour is the speculation, and the speculation is the cost, and the cost is the reason the trace comes before the fourth reading of the config.

What the trace showed

The trace was the openat call, and the call was the path, and the path was the etc slash app slash config dot yml, and the yml was the expected, and the expected was the not the actual, and the actual was the yaml, and the yaml was the extension's difference, and the difference was the ENOENT, and the ENOENT was the no such file, and the no such file was the fallback, and the fallback was the default, and the default was the ignored config.

The one character was the answer, and the answer was the invisible in the log, and the invisible was the program's silence, and the silence was the fallback's design, and the design was the reasonable, and the reasonable was the incident's cause, because the silent fallback hid the typo for a week.

The trace was the one command, and the command was the strace with the file filter, and the filter was the readable, and the readable was the diagnosis, and the diagnosis was the ten seconds, and the ten seconds was the week's end, and the end was the fix, and the fix was the extension.

This is the same principle as reading the process state rather than the dashboard's summary, same mechanism, different costume.

How to run the trace well

Filter to the syscalls you care about. The trace filter is the file or the network, and the filter is the noise's removal, and the removal is the readable, and the readable is the diagnosis, and the diagnosis is the fix, and the fix is the expression, and the expression is the discipline, because the unfiltered trace is the unreadable.

Follow the child processes. The daemon forks, and the forks is the children, and the children is the actual work, and the work is the trace's target, and the target is the follow flag, and the flag is the discipline, because the parent's trace shows the fork and not the open.

Trace the failing call, not the successful ones. The ENOENT is the answer, and the answer is the error's line, and the line is the diagnosis, and the diagnosis is the fix, and the fix is the grep for the error, and the grep is the discipline, because the successful calls are the noise.

Compare against the expected path. The expected is the config's documentation, and the documentation is the comparison, and the comparison is the difference, and the difference is the fix, and the fix is the one character, and the character is the discipline, because the trace alone shows the attempt and the comparison shows the mistake.

Remember the trace shows the secrets. The trace is the arguments, and the arguments are the passwords and the tokens, and the tokens are the sensitive, and the sensitive is the handling, and the handling is the deletion, and the deletion is the discipline, because the trace file is the credential dump.

What I now do

When a program's behaviour does not match its configuration, the syscall trace shows what the process actually requested from the kernel. Filter to the relevant syscalls, follow child processes, look for the failing call rather than the successful ones, compare the attempted path against the expected one, and delete the trace afterwards.

The config file was correct and the program ignored it, and the log said the configuration had loaded, because the program asked the kernel for a path ending in dot yaml while the file on disk ended in dot yml, and the failed open fell back to defaults without reporting it. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud