The Connection String That Broke on a Special Character

The rotation was routine, the new password was strong, and the service failed to connect with an error that named a host nobody recognised, which is the signature of a parse, not a network, because the network never saw the host you intended. The password contained an at sign, and the connection string, assembled as a URL, is a URL, and a URL has grammar, and the grammar says the at sign separates the credentials from the host, so the parser read the password's at sign as the boundary and split the authority at the wrong place, sending the connection to a host made of the password's tail.

This is the autopsy of a string that is two languages at once, a secret and a URL, and of the parse that had to guess which, and guessed, and was wrong, and the fix is the discipline of never letting a secret live inside a grammar it can break.

This was Postgres 16.3 with a URL style connection string, and the failure generalises to any DSN that embeds credentials in a parsed format, which is most of them.

The quick version

A URL connection string has the shape scheme then user then colon then password then at then host, and the parser scans for the at sign to find the host. A password containing an at sign, unencoded, moves the boundary, so everything after the at sign, which is part of the password, is read as the host, and the real host is discarded or misread, and the user and the password prefix are read as credentials for a host that does not exist. The connection then fails with a name resolution error or a refused connection to a nonsense host, and the error message prints the nonsense host, which is the parse's confession, if you know to read it as one.

The same string worked before the rotation because the old password had no reserved character, which is the cruel part: the format's correctness depended on the secret's content, which is a contract between the parser and the password generator, and the generator, doing its job by adding symbols, broke the contract.

Why the test suite missed it

The connection string was tested with the old password, and the parse was tested with a fixture password that had no symbols, so the parse's assumption, no reserved characters, was never exercised, which is the fixture blindness from the export that always returned nine hundred ninety nine rows, and the shared property is the one worth fixing.

It is also why the failure arrived at rotation, not at deploy, because the string was only reassembled with the new secret at rotation, so the latent parse bug had a trigger date, which is the shape of every secret dependent bug, dormant until the secret changes.

The causes, ranked

1. A secret embedded unencoded in a parsed grammar

The primary defect. The password was interpolated raw into the URL, so the password's bytes were subject to the URL's grammar, and any reserved character, at, colon, slash, hash, becomes syntax. The secret must be percent encoded before insertion, which is the encoding that turns syntax back into data, the same data versus structure discipline as SQL injection inside the ORM you trusted, at the URL layer.

2. A generator and a parser with no shared contract

The password generator emits from a symbol set that includes reserved characters, and the connection string consumer assumes none, and nothing between them enforces encoding, so the two components disagree about the alphabet, and the disagreement is a bug with a trigger date.

3. An error message that prints the parse, not the intent

The failure printed the nonsense host, which is correct output from a wrong parse, and the on call read it as a DNS problem and chased resolution for an hour, because the error describes the parser's reality, per error messages are a user interface, and the parser's reality was not the operator's intent.

The fix

The immediate fix encoded the password before assembling the string, and the connection recovered, and the postmortem's durable fixes were three.

First, the connection string is assembled by a function that takes the components as separate arguments and encodes each, so the raw interpolation, the bug's vector, no longer exists as a code path, and the encoding is the library's, not a hand rolled replace, because hand rolled encoding misses a character per quarter.

Second, the parse is tested against the generator's alphabet: a property test that generates passwords from the full symbol set, assembles and re parses, and asserts the parsed host and user equal the inputs, which is the contract between generator and parser made executable, and it fails on the at sign in milliseconds.

Third, the error path logs the intended host alongside the parsed host, so the next parse failure prints both realities, the intent and the parse, and the divergence is visible in the first log line, which is the diagnosis handed over instead of chased.

What I would do differently

I would have treated the connection string as an interface between two components, the generator and the parser, with an explicit encoding contract, because any string that carries a secret through a grammar is an interface, and interfaces need contracts, and the contract here was a hope that the password stayed tame, which is a contract written by the optimistic.

I would also have preferred the non URL form where the driver supports it, host, port, user and password as separate parameters, because the form that never concatenates the secret into a grammar is the form that cannot be broken by the secret's content, and the absence of the parse is the only parse that never fails.

The rule

A connection string that embeds a secret in a parsed grammar is two languages sharing one byte stream, and any reserved character in the secret becomes syntax and moves the parse, so the secret must be encoded before insertion, or better, never concatenated at all.

The password did its job by being strong, and the string did its job by being a URL, and the incident was the collision of two correct behaviours, which is the most expensive class of bug there is, and the cheapest to prevent, because the prevention is one encoding call, or one parameter form, standing between the secret and the grammar.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud