An AI that trusted the wrong manual
August 31, 2026
A generative AI assistant built to help field technicians did exactly what it was designed to do, and still gave wrong answers with total confidence. The reason has nothing to do with the AI's intelligence.
What happened
- A company built a GenAI field-service assistant for technicians to consult during repairs
- It read the manufacturer's documentation correctly and gave clear, confident instructions
- The instructions were wrong: the manuals it was trained on described the official, 30-year-old procedure, not what technicians actually did in practice
- Nobody had told the AI, or the documentation system, that the real-world procedure had quietly changed
- BCG's conclusion: this is a data-governance failure, not a model-capability failure
The instructions were correct. They were also thirty years out of date.
BCG documents a case worth sitting with in full. A company built a generative AI assistant to help field technicians work through repair procedures. It worked exactly as designed. It read the manufacturer's documentation, understood the steps, and gave clear instructions.
The instructions were wrong.
Not because the AI hallucinated, and not because it misunderstood the documents. The documents themselves were the problem: the manuals it had been trained on described the official, technically correct procedure from thirty years earlier. Nobody in the field had actually followed that exact procedure in years, it had been quietly replaced by better practice that never made it back into the official record. The people doing the work knew the update. The documentation, and therefore the AI, didn't.
The AI had no way to know that. It read what it was given, and what it was given was outdated. It did its job. Its job was just built on the wrong foundation.
This is the failure mode most AI conversations skip
Most conversations about AI risk focus on the model itself: is it accurate, does it hallucinate, is it biased. That's a legitimate conversation, and plenty of research covers it. But this case demonstrates a different, quieter failure that gets far less attention and is arguably more common in practice: the model can be working exactly as intended, reading documents that are exactly what they claim to be, and still produce a confidently wrong answer, because the source material itself went stale and nobody flagged it.
An AI system is only ever as current as what it's given to read. If the real, working version of a procedure lives in a technician's head, in a message thread from eighteen months ago, or in a revision that never made it back into the official document set, the AI will never see it. It will keep citing the old version with full confidence, because as far as it can tell, that is the current version. There's no internal signal that says "this might be stale." Confidence and correctness are two entirely separate properties, and an AI reading bad inputs produces high confidence in a wrong answer just as readily as it produces high confidence in a right one.
Why this connects to two other findings worth knowing
MIT Sloan's research on AI in finance functions describes a related version of the same gap from the other direction: pilots that look promising get built, and then quietly stop being used the moment real pressure hits, because the organizational behavior around the underlying process never actually changed to match the new tool. A dashboard gets built and refreshed. Nobody changes how they close the books. The tool and the reality it's supposed to reflect drift apart, silently.
Gartner's own customer research adds the trust dimension: even as GenAI-assisted interactions get objectively easier for customers, 87% still say it's essential to have the option to reach a human when something matters. That instinct isn't irrational technophobia. It's a reasonable response to exactly the failure mode BCG documented: people have learned, correctly, that an AI's confidence is not the same thing as an AI's correctness, and they want a way to check when the stakes are high enough to matter.
Why this matters specifically for property and real estate
For anything involving physical property, buildings, appliances, warranties, and the documents that accompany them, this failure mode isn't a hypothetical edge case. It's close to the default condition of most property documentation.
Warranty terms change or get superseded by a manufacturer update. Appliance manuals get replaced by newer models with different specifications. A maintenance record that was accurate at the point of handover can be meaningfully stale within a year if nothing actively keeps it current. An AI system reading an unmaintained property archive will make exactly the same mistake as that field technician's assistant: read what it's given, trust it completely, and give a perfectly confident answer that happens to be wrong, about a warranty that's expired, a procedure that's been superseded, or a fact that changed the moment ownership transferred and nobody updated the record.
The fix isn't a smarter model
BCG's own conclusion points squarely at governance, not model capability. This is a data conversation, not a modeling conversation. The question that actually determines whether an AI system will be reliable isn't "how good is the model." It's "how do we know this specific piece of information is still current, and who is responsible for keeping it that way." A better model reading the same stale manual produces the same confidently wrong answer, just phrased more articulately.
Frequently asked questions
Couldn't a more advanced AI model have caught this mistake on its own?
No. Model capability and source-data freshness are independent variables. A more advanced model reading the same outdated manual will describe the wrong procedure just as fluently and confidently as a simpler one. The fix has to happen at the data layer, not the model layer.
How would anyone even know their documentation had gone stale like this?
In the BCG case, the gap surfaced when a technician flagged that the AI's instructions didn't match what was actually done in the field, after the fact, not before. The more reliable approach is a system that tracks when a record was last verified as current, so staleness is visible before it causes a wrong answer, not after.
Does this only apply to AI systems, or is it a broader documentation problem?
It's a broader problem that AI happens to make more visible, faster. A human reading the same stale manual might notice something felt off and ask around. An AI reads it and states the outdated procedure with total confidence, with no built-in instinct to double-check, which is exactly why the underlying documentation discipline matters more, not less, once AI is involved.
What's the practical first step to avoid this in a property context?
Know, for every piece of documentation that matters (warranties, manuals, maintenance records), when it was last verified as current and by whom. A record with no verification history is a record you can't actually trust an AI, or a person, to rely on without checking first.
CroVIN keeps property documentation current and verifiable, not just stored, so an AI, or a person, reading a warranty, manual, or maintenance record can trust that what it's reading is still true.