> ## Content Index > Fetch the complete content index at: https://debugly.dev/llms.txt > Use this file to discover other available public pages before exploring further. # The Embedding Swap That Silently Broke Semantic Search - URL: https://debugly.dev/embedding-swap-broke-semantic-search/ - Published: 2026-10-09T14:34:00.000Z - Updated: 2026-10-17T05:56:16.000Z - Description: A newer embedding model went live for queries while the index still held the old vectors, and the two spaces were unrelated, so similarity became noise and… - Author: Rohit Bhadani - Tags: Bug Hunt, AI, Search It ended at the incident. It started at the plausible. Every hand-off in between did exactly what it was told, and this is the one that lied. The embedding space mismatch has a single weak property, and here it is: the dimensions matched, so nothing raised, and the similarity returned a number, and the number was valid, and the valid number was the meaningless comparison. I have since met the same mechanics in plenty of vector index whose query model and index model can diverge. ## The hypothesis that failed first The first hypothesis was the data, because the search was worse and the data is the usual suspect, and the data had not changed, and the unchanged data ruled out the data, and the ruled out was the second hypothesis, which was the ranking weights, and the weights had not changed either, and the unchanged weights ruled out the weights, and the third hypothesis was the retrieval, and the retrieval was returning candidates, and the candidates were in the index, and the index was fine, and the fine was the dead end. The dead end was the clue, because everything in the pipeline was fine and the output was wrong, and the everything fine with the output wrong is the signature of a silent mismatch, and the mismatch is not a failure, it is a comparison that returns a number, and the number is the problem, and the problem is invisible because the number is valid. The three days were the cost of the plausible results, because the crash would have been caught in minutes, and the minutes are the cheap detection, and the cheap detection is what the silent mismatch does not provide, and the not providing is the reason the mismatch is worse than the crash. ## The breakthrough The breakthrough was the dimensions, and the dimensions were the same, and the same dimensions were the reason nothing raised, and the not raising was the trap, because the vector library checks the shape and the shape matched, and the shape is the length, and the length was identical in both models, so the library had no reason to complain, and the not complaining was the silence. The actual check was the provenance, and the provenance was the question nobody had asked: which model produced this vector. The index vectors were six months old, produced by the previous model, and the query vectors were fresh, produced by the new one, and the new model was better, and the better was the reason for the swap, and the swap was the deployment, and the deployment updated the query path, and the query path was the only path updated, and the index path was the reindex, and the reindex was scheduled for later, and the later was the gap, and the gap was the three days. The two models had the same output dimension and completely different learned spaces, so a vector from one has no meaningful relationship to a vector from the other, and the cosine of two unrelated vectors is a number between minus one and one, and the number is valid, and the valid is the meaningless, and the meaningless is the ranking. ## What I changed **Reindexed before switching the query model.** The order was reversed, so the index was rebuilt with the new model first, and the rebuilt index was the prerequisite, and the prerequisite was the fix, because the query model and the index model must match, and the match is the atomic change, and the atomic is the deployment, and the deployment is the fix. **Stored the model version with each vector.** Each vector now carries the identifier of the model that produced it, and the identifier is the provenance, and the provenance is the check, and the check is the refusal, because the query refuses to compare against a vector from a different model, and the refusal is the loud failure, and the loud failure is the fix, and the fix is the one that turns the silent mismatch into an error. **Added a canary query set.** A small set of queries with known good results runs after every index or model change, and the known good is the baseline, and the baseline is the comparison, and the comparison is the regression, and the regression is the alert, and the alert is the detection, and the detection is the minutes rather than the days, and the minutes are the fix. **Treated the embedding model as a schema change.** The model swap is now a migration, with the same discipline as a database schema change, because the vectors are the schema, and the schema change requires the data migration, and the migration is the reindex, and the reindex is the prerequisite, and the prerequisite is the discipline, and the discipline is the prevention. ## If it happens again An embedding index and the query model must produce vectors from the same model, because similarity between different spaces is meaningless, and identical dimensions hide the mismatch. Reindex before switching the query model, store the model version with each vector, refuse cross-model comparisons, and run a canary query set on every change. The search returned confident nonsense for three days, because the query vectors and the index vectors came from different models with the same dimension and unrelated spaces, and the cosine of two unrelated vectors is still a number. I now check for that property first, because it is where the failure actually lives. --- Published on [debugly.dev](https://debugly.dev/) by [Rohit Bhadani](https://debugly.dev/about/) | Founder of [Krova Cloud](https://krova.cloud/?ref=debugly.dev)