The Migration That Could Not Run in a Blue-Green Deploy

It ended at the incompatible schema. It started at the deploy. Every hand-off in between did exactly what it was told, and this is the one that lied.

The migration that breaks the two-version window has a single weak property, and here it is: blue-green assumes both versions run against the same schema, so a change the old version cannot read removes the rollback, and the rollback is the whole point of blue-green.

I have since met the same mechanics in plenty of zero-downtime strategy that runs two versions concurrently.

Why the two versions must share the schema

The blue-green is the two environments, and the two are the old and the new, and the old is the running, and the new is the candidate, and the candidate is the switch, and the switch is the traffic's move, and the move is the reversible, and the reversible is the rollback, and the rollback is the switch back, and the back is the old version, and the old version is the schema's reader, and the reader is the requirement.

The requirement is the schema's compatibility, and the compatibility is the both versions' ability, and the ability is the read and the write, and the two are the columns, and the columns are the migration's subject, and the subject is the drop, and the drop is the old version's failure, and the failure is the rollback's absence, and the absence is the incident.

The migration runs before the switch, and the before is the both versions' live, and the live is the old's read, and the read is the dropped column, and the dropped is the error, and the error is the immediate, and the immediate is the incident's start, and the start is the migration's timing, and the timing is the fix's subject.

Why the rollback became a restore

The switch back was the plan, and the plan was the old version, and the old version was the schema's expectation, and the expectation was the column, and the column was the gone, and the gone was the error, and the error was the old version's broken, and the broken was the rollback's absence, and the absence was the restore, and the restore was the backup, and the backup was the hours, and the hours was the data's loss window, and the window was the incident's cost.

The restore is the worse, because the restore is the data's rollback, and the data's rollback is the writes' loss, and the loss is the customers', and the customers' is the incident, and the incident is the reason the schema migration must be reversible without the data's loss, and the reversible is the expand-contract, and the expand-contract is the fix.

This is the same coupling as the prompt and the model that shipped together and could not be separated, which is why the two fixes look alike.

What actually fixed it

Expand, migrate, contract, in three deploys. The expand adds the new column, and the new is the both versions' tolerance, and the tolerance is the switch, and the switch is the new version's write to both, and the both is the backfill, and the backfill is the contract, and the contract is the old column's drop, and the drop is the third deploy, and the third is the safe, because no version needs the dropped column any more.

Never drop or rename in the same deploy as the code change. The drop is the incompatible, and the incompatible is the rollback's absence, and the absence is the incident, and the incident is the prevention's target, and the target is the separation, and the separation is the two deploys, and the two is the discipline.

Add the column as nullable or with a default. The nullable is the old version's insert, and the insert is the working, and the working is the compatibility, and the compatibility is the fix, and the fix is the constraint's absence, and the absence is the temporary, and the temporary is the contract's later, and the later is the third deploy.

Test the rollback, not only the deploy. The rollback is the switch back, and the back is the old version's run, and the run is the schema's test, and the test is the failure's detection, and the detection is the fix, and the fix is the drill, and the drill is the discipline, because the untested rollback is the assumed rollback, and the assumed is the restore.

Keep the migration and the code versioned together. The migration is the schema's version, and the version is the code's expectation, and the expectation is the compatibility's check, and the check is the fix, and the fix is the record, and the record is the diagnosis, and the diagnosis is the incident's speed.

What I now do

Blue-green runs two versions against one schema, so a migration that drops or renames what the old version reads removes the rollback and turns it into a restore. Expand in one deploy, migrate the code in the second, contract in the third, add new columns as nullable, and test the rollback rather than only the deploy.

The deploy was blue-green and the rollback was supposed to be a switch, but the migration dropped a column the old version still selected, so switching back produced errors and the only way home was a database restore. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud