The Feature Flag I Flipped Three Hours Before Anyone Noticed

The flag was supposed to enable a new pricing display for five percent of traffic. I flipped it at 9:04 in the morning, watched the dashboard for twenty minutes, saw nothing, and assumed the rollout was just slow to propagate.

At 12:20 a colleague restarted a pod to pick up an unrelated config change. At 12:21, traffic started showing the new pricing. The flag had been on the whole time. Nothing had been reading it.

The bug was one line of module level code, and it is one of the most common ways a feature flag silently fails to be a feature flag.

This was Node 22.14 with a small config client, but the exact same defect exists in every language and I have now found it in Go, Python and Java codebases.

The symptom, precisely

A flag change in the flag service took effect on some instances and not others. Specifically, it took effect on instances that started after the change, and never on instances that were already running. There was no polling interval to wait out. Hours later, old instances were still serving the old value.

The hypotheses that died

Propagation delay in the flag service

The first suspect was the flag backend itself. Maybe the change took minutes to reach the edge, maybe there was a CDN in front of the flag API. I queried the flag API directly from an affected instance and got the new value back in twelve milliseconds.

It died because the backend was serving the new value to anyone who asked. The instance was simply never asking.

A caching layer in the flag client

Next I assumed the SDK cached values with a TTL and I just had to wait it out. I read the SDK source. It polled every thirty seconds and updated an in memory map. It looked completely correct.

It died because the map it updated was not the map the application read.

That sentence took me embarrassingly long to arrive at.

The breakthrough

The application code looked like this, in a config module:

const client = new FlagClient(process.env.FLAG_KEY);
const SHOW_NEW_PRICING = client.getBoolean("show-new-pricing", false);

export function pricingEnabled() {
  return SHOW_NEW_PRICING;
}

getBoolean was called once, at module load. It returned the value as it was at boot and stored it in a constant. The SDK kept polling happily and updating its own map, but the application had copied the value out at startup and never looked again.

The "flag" was really a boot time constant wearing a flag's clothes. Every instance froze the value at the moment it started. New instances got the new value. Old instances kept the old one, forever, until restarted.

This is the config equivalent of the cache key bug in cache invalidation and stale data: the source of truth is fine, and a copy of it, made once, is what gets served.

Why the type system did not save us

The interesting part is that this looks type correct. getBoolean returns a boolean. SHOW_NEW_PRICING is a boolean. Nothing at compile time distinguishes "a boolean captured now" from "a boolean evaluated on each call". The difference is temporal, and types rarely model time.

In Go the same bug is a package level var showNewPricing = client.Bool(...). In Python it is a module level constant. In Java it is a static final initialised from a config call. The pattern is language neutral because it is a mistake about when, not about what.

What I changed

Read the flag at the decision point

The mechanical fix:

export function pricingEnabled(userId) {
  return client.getBoolean("show-new-pricing", false, { userId });
}

Now the value is read when the decision is made, so it is always current, and it can also be evaluated per user, which is the actual point of a percentage rollout. The boot time constant disappeared.

Made the dangerous shape hard to write

I banned module level captures of config values in review, and added a lint that flags any top level call into the flag client. It is a heuristic and it has produced exactly two false positives in a year, which is a good trade for catching a class of bug that otherwise only surfaces in production.

Added a change detector test

I wrote a test that starts the app, flips a flag in a stubbed flag backend, and asserts the app sees the new value within one poll interval without a restart. That test is the whole bug, compressed to forty seconds. It now runs in CI on every change to the config layer.

This is the pattern I keep coming back to for config: the only way to know your config is live is to change it and watch it apply. Everything else is a belief.

What I would do differently

I would have noticed the shape of the symptom sooner. "Works on new instances, not on old ones" is a sentence that has exactly one common cause: something was captured at boot. I went through two plausible infrastructure explanations before reading the code, and the code had the answer in it from the start.

I also would have questioned any flag that is read exactly once at startup. There are legitimate boot time settings, things like database URLs, where a restart is the correct semantic. But a rollout flag that only applies after a restart is not a rollout flag. If the semantic of your change is "restart required", that is a config value, and it should be labelled and deployed like one, with a rollout plan, not flipped in a dashboard.

The rule

A feature flag is a function, not a value. If you can assign it to a constant at boot, you have turned it into a config value and lost everything that made it a flag: liveness, targeting and instant rollback.

The same "captured once" defect shows up everywhere state crosses a boundary at startup, including the CI runner that reused a six day old container in flaky CI shared runner state. The runtime changes. The copies do not.

Check your flag reads today. Grep for the client being called at top level. If you find one, that flag has never been a flag.