The Egress Rule That Let the Backup Phone Home

The audit found the egress rule before an attacker did, which is the good version of this story, and the rule was a wildcard, added to unblock a backup job whose destination had changed, and left in place because the job worked and nobody's Tuesday depended on removing it. The wildcard was not an attack. It was an opening, and an opening's cost is paid by whoever finds it, and the finding is a matter of time, which is why the allowlist's exceptions are the perimeter, and the perimeter had an exception shaped like everything.

This is the autopsy of an egress rule as a vulnerability, and of the specific dynamics that make egress the most neglected boundary: it is configured under time pressure, it is verified by the job working, and it is logged on every use by a log nobody reads.

This was a workload with a default deny egress, per the service is running, listening and completely unreachable, and the shared property is the one worth fixing.

Why egress is the forgotten boundary

The inbound boundary gets the attention, the firewalls, the WAFs, the reviews, because the inbound threat is the one the industry narrates. The egress boundary is configured by the people who need the job to work, under the pressure of a failing job, and its failures are silent, because a job that can reach everything is not visibly broken, it is visibly working, and the working is the verification, and the verification is the vulnerability, because the check that passes is "can the backup reach its destination", never "can the backup reach only its destination".

The difference between those two checks is the wildcard, and the wildcard is the answer to the first check and the failure of the second, and the second check is the one that was never run.

The dynamics of the wildcard

The wildcard arrived through the standard path, which is worth naming because the path is the defect. The job failed on a Tuesday, the destination had moved to a new host, the operator, under the pressure of the failing backup, which is the one job whose failure is noticed, widened the rule to make it work, and the widening was logged as a fix, and the fix was verified by the job succeeding, and the success closed the ticket, and the ticket's closure was the end of the rule's review, forever.

The wildcard's cost is not the backup's traffic, which is legitimate. It is the exfiltration channel it creates for anything that compromises the workload, because a compromised job, or a compromised dependency inside the job, inherits the wildcard, and the wildcard is the ability to send the workload's data anywhere, which is exfiltration as a feature of the perimeter, and the SSRF lesson from the webhook URL field that could reach your metadata endpoint, and the shared property is the one worth fixing.

The logging that was unread

The rule was logged on every use, because the egress filter logs its allows, and the log contained, for a month, every connection the wildcard permitted, which is the exfiltration audit trail, pre written, and unread, because the log of allowed traffic is the least read log there is, since allow is the boring outcome, and the boring outcome is the audit's entire evidence. The incident's evidence was present from day one, in a log whose reader did not exist, which is the observability gap from nobody reads your dashboards, which is why the two fixes look alike.

The fixes that shrink the hole

The allowlist is per destination, and the destination is a set, not a pattern. The backup's destination is a host, or a small set, and the rule names them, and the naming is verified against the job's actual connections, which the log provides, so the rule is derived from the observed traffic, which is the allowlist built from evidence, and the wildcard is the allowlist built from impatience.

The exception has an expiry. The Tuesday fix, when it must be wide, is wide with a deadline, and the expiry is the review, because the expiring rule forces the question, is this still needed, at a scheduled time, with nobody under pressure, which is the only time the question gets an honest answer.

The allow log gets a reader. The egress allow log is alerted on novelty, a destination never seen before, a volume outside the baseline, because the exfiltration is novel by definition, and the novelty alert is the reader the log never had, which converts the pre written evidence into a question asked at the moment of the finding.

The backup's data is encrypted before egress. The exfiltration channel's value is the plaintext, and the backup that encrypts before leaving, with keys the workload does not hold, makes the wildcard a channel for ciphertext, which is the blast radius reduced at the data layer, the same removal principle as the agent held a token that could delete what it was asked to read, applied to the data instead of the credential.

What I would do differently

I would have treated the egress rule as a security artefact with the same review as an inbound rule, because the wildcard's direction does not change its nature, it is a hole in the perimeter, and the perimeter does not care which way the traffic flows, only that the hole is shaped like everything.

I would also have made the Tuesday fix expensive to leave in place, by making the wide rule visible, a dashboard of the current egress rules with their widths and ages, so the month old wildcard is a red row, because the wildcard's power is its invisibility, and the row is the visibility.

What I now do

An egress allowlist is a perimeter, and its exceptions are its holes, and the wildcard is a hole shaped like everything, added under pressure, verified by the job working, and logged on every use by a log nobody reads. Name the destinations from observed traffic, expire the exceptions, alert on novelty in the allow log, and encrypt before egress.

The backup phoned home for a month, politely, on schedule, and the only witness was the allow log. That is the whole pattern, and it is the part worth remembering.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud