The Prompt and the Model That Could Not Be Separated

I have chased this from the fix end and from the drop end, and the truth was in the hand-off, as it usually is.

Here is the shape of the coupled AI deploy, and the one property that makes it expensive at rollback time: the prompt and the model are separate variables with a shared effect, and a deploy that changes both cannot be reverted selectively, so the rollback is all or nothing.

I have since met the same mechanics in plenty of feature where the prompt and the model version are deployed as one unit.

Why they shipped together

The prompt was rewritten for the new model, and the rewrite was the adaptation, and the adaptation was the reason the two moved together, because the new model responded better to the new structure, and the better was the measurement, and the measurement was the evaluation, and the evaluation used both, and the both was the tested combination, and the combination was the deploy, and the deploy was the unit, and the unit was the coupling.

The coupling was reasonable at deploy time, because the combination was tested, and the tested is the safe, and the safe is the ship, and the ship is the value, and the value is the improvement, and the improvement required both, so the both shipped, and the shipping was correct, and the correct was the problem later, because the correctness at deploy time is the coupling at rollback time, and the rollback time is the incident, and the incident is when the coupling costs.

The coupling is invisible until the rollback, because the deploy is the forward direction, and the forward is the tested path, and the tested path is the safe, and the safe is the confidence, and the confidence is the ship, and the ship is the coupling's hiding place, and the hiding place is the reason the coupling is not designed against.

Why the rollback failed

The quality dropped, and the drop was the prompt, and the prompt was the suspect, and the suspect was the revert, and the revert was the deploy, and the deploy was the unit, and the unit was the both, so the revert undid the model too, and the model's undo was the downgrade, and the downgrade was the regression, and the regression was the second incident, and the second was the confusion, because the quality got worse in a different way, and the different way was the model, and the model was not the suspect, and the not suspect was the misdiagnosis, and the misdiagnosis was the hour, and the hour was the incident's duration.

The rollback's failure was the granularity, because the revert is the whole deploy, and the whole is the both, and the both is the over correction, and the over correction is the new problem, and the new problem is the second incident, and the second is the cost, and the cost is the coupling, and the coupling is the fix's target.

This is the same shape as the model upgrade that changed the output schema nobody pinned, and the shared property is the one worth fixing.

The fix

Version the prompt separately. The prompt lives in its own artefact with its own version, and the version is the identifier, and the identifier is the rollback's unit, and the unit is the selective revert, and the selective is the fix, and the fix is the separation, and the separation is the discipline from prompt deploys, versioning and rollback, and the common root is the property, not the platform.

Pin the model version in configuration. The model identifier is configuration, and the configuration is the separate variable, and the separate is the independent rollback, and the independent is the fix, and the fix is the pin, and the pin is the control, and the control is the discipline, because the floating alias cannot be rolled back, and the not rollback is the coupling.

Deploy them in sequence, not together. The model ships first, and the model is evaluated, and the evaluated is the baseline, and the baseline is the prompt's deploy, and the prompt is evaluated against the baseline, and the against is the attribution, and the attribution is the fix, because the sequential deploy means each change has its own measurement, and the measurement is the rollback's evidence.

Keep the previous prompt compatible with the new model. The prompt is written to work on both models during the transition, and the both is the window, and the window is the independent rollback, and the independent is the fix, and the fix is the compatibility, and the compatibility is the work, and the work is the cost of the separation, and the cost is worth the rollback.

Record which combination is live. The running prompt version and model version are logged with every request, and the logged is the attribution, and the attribution is the diagnosis, and the diagnosis is the fix, and the fix is the record, and the record is the incident's first question answered, and the answered is the speed, and the speed is the duration's reduction.

The rule I keep

The prompt and the model are separate variables with a shared effect, so a deploy that changes both cannot be reverted selectively, and the rollback becomes all or nothing. Version the prompt as its own artefact, pin the model in configuration, deploy in sequence with a measurement between, and log the live combination on every request.

The quality dropped and the rollback fixed the prompt by also downgrading the model, and the downgrade was a second incident. The lesson I keep is the property, not the incident.

Related reading: The Cron Job That Overlapped Itself and Doubled the Work explores a connected angle.

Related reading: The Word Doing All the Work in the AWS Data Loss Notice explores a connected angle.

Related reading: The Agent Traffic That Took Down a Query Service explores a connected angle.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud