5 Spooky Things About Trusting AI with Reliability
The scariest engineering story I read this year wasn’t fiction. In August, Reuters reported on internal Meta posts showing what happened as the company pushed its engineers toward AI tooling. Code changes to internal platforms rose 220% year over year. Changes that actually reached users rose 36%. Major technical and security incidents rose 40%, and the time engineers spent firefighting them rose 70%. (Meta declined to comment on those figures.)
More code. Somewhat more product. A lot more fire!
That isn’t a reason to stop using AI. We are using it at Gremlin, both to build and as part of our newest product launch. The problem is trusting it to write, review, and fix code without anything checking its work against how real systems fail. Here are five reasons that should keep engineering leaders up at night this Halloween, and what to do about each.
1. It writes code faster than anyone can review it
AI didn’t just speed up coding. It multiplied it. The number of people who can review that code, test it, and understand how it behaves in production didn’t multiply with it.
I’ve said that 10x the code means 10x the opportunity for failure. The checks haven’t kept pace. New Relic’s 2026 State of AI Coding report found that 62% of technology leaders say their teams often ship AI-generated code to production without line-by-line verification. 95% of organizations allow vibe coding for production services, and none ban it outright.
When volume outruns the checks, every other risk on this list gets worse.
2. It looks right even when it isn’t
The guardrail most teams still lean on is code review. AI-generated code is very good at passing it.
In the same New Relic report, 94% of respondents rate AI-generated code as higher quality than human-written code at review. Yet 78% report more incidents once it ships, and 82% have hit at least one production failure tied to AI-generated code in the past six months.
That isn’t a contradiction. Review checks what code says. Reliability depends on what code does when a dependency times out under peak load. AI writes clean, confident code whether it’s right or wrong, so the code that reads best isn’t necessarily the code that holds up.
3. It knows how systems should work, not how they fail
Good engineers carry scar tissue. When I was a call leader at Amazon, every minute the retail site was down cost tens of thousands of dollars in lost revenue. You don’t forget the lessons you learn under that pressure: the retry storm, the cache that fell over, the failover that didn’t.
General-purpose AI learns from documentation, best practices, and public code. That’s a record of how systems are supposed to work. It isn’t a record of what actually happens when they break, and none of those painful lessons come along for the ride.
The gap matters most for the novel failures behind the worst outages, which is exactly where best practices run out.
4. It can’t see your production
Even if AI knew everything about how systems fail in general, it wouldn’t know how yours behaves today. An AI tool sees your repo and whatever telemetry you give it. Telemetry shows what has already happened. It can’t show how your system will respond to a failure it hasn’t had yet.
Production is more than code and dashboards: security groups, load balancer settings, a config someone tweaked last sprint, a dependency nobody documented, customers who do things you never expected. I’ve long argued that if you only test in dev or staging, you miss a whole class of failures that can only happen in production. The same is true of AI reasoning from code.
And systems drift. What held last month may not hold today, and nothing in the repo will tell you.
5. It can’t prove its own fix
Ask an AI tool to fix a reliability problem and you’ll get a fix. Ask whether it worked and you’ll get a yes. That’s grading its own homework.
Any fix, from an AI or an engineer, is a hypothesis until it’s tested. The only way to know a failover works is to fail something over, safely and with a contained blast radius, and watch what happens. Once at Netflix, during a live incident, we reproduced the failure on our own accounts, pinpointed the cause, hotfixed it, and confirmed it worked on ourselves before rolling it out to everyone.
Teams are already giving AI a lot of trust and doing very little verification. But trust is earned slowly and lost in a single outage, and AI hasn’t earned it yet. Until it does, every fix it suggests needs proof.
What it takes to trust AI with reliability
None of this is an argument to slow down. The teams that come out ahead will use AI aggressively and keep their engineering discipline. Leverage AI, but don’t let it run in production without understanding the risks and validating that it’s reliable and resilient.
What teams need are reliability guardrails: automated checks that let AI move fast without putting production at risk. Here are five, one for each risk above:
- Put reliability guardrails in the pipeline. Every change gets tested automatically before it ships, so checking scales with the code, not with headcount.
- Make failure behavior the release bar. Passing a review isn’t enough. A change ships when it holds up while a dependency fails or latency spikes.
- Check AI’s recommendations against real failure patterns. Before anyone acts on a suggestion, test it against how systems actually break, not just against best practices.
- Safely test your live systems on a schedule. Drift and undocumented dependencies should show up in a test, not an outage.
- Don’t close a fix until it’s proven. Re-run the failure that exposed the risk and confirm the system holds.
That’s the thinking behind Foresight AI. Its supervised agents are grounded in the Failure Atlas, our record of how real systems fail, built from more than ten years of resilience testing. They test with safe, controlled fault injection, recommend specific fixes, and re-run the test to prove each fix held.
Halloween is the one night a year people go looking for a scare. Reliability works the same way. Find the failures before they find your customers.
The post 5 Spooky Things About Trusting AI with Reliability appeared first on SD Times.
Tech Developers
No comments