The Hidden Risk in Self-Healing Test Automation: A Governance Blueprint for Digital Banking
A test failed. No one even realizes it, but thanks to an AI-powered self-healing tool the failed locator is replaced and the pipeline becomes green again. So far, all was expected from such tool for many development teams. However, in some cases, such approach can cause real defects to become a part of the application in disguise of passing test.
This is what the article is all about – how AI-powered self-healing tools can mask real bugs in the product. The question is how to avoid it.
The Problem Nobody’s Pricing In
The ability to repair tests on the fly and restore the productivity of engineers is already implemented in AI-powered test automation. If a UI element is moved, renamed, or even removed from the page, it is detected by such tools as Healenium and the necessary adjustments are applied automatically without any help from the developer. From that point of view, this technology offers quite evident advantage – less time spent for finding broken locators and more time for testing, fast releases, and finally no calls at 3 a.m. when a test fails because someone changed the name of the button.
The drawback is that the self-healing technology that is focused on keeping the pipeline green does not have a way to differentiate between two quite different situations: the cosmetic change of the UI that does not influence the test and real product defect that coincidentally looks like such change. Without additional measures, both problems are fixed in the same way. We can call it silent coverage erosion.
But this gap is not aesthetic in the case of banking. A non-validated locator update during a core system migration or a compliance-flow release may be an actual regulatory violation, broken transaction flow or poor customer experience that hides quietly behind a green checkmark in the CI/CD dashboard.
A Governance Layer, Not a Kill Switch
It may sound as the solution in the regulated environment to disable the self-healing process altogether to avoid these risks. However, this approach will lead to wasting of the benefits related to reduced maintenance costs. Instead, one should consider each of the proposed by AI solutions as the production changes, which should pass through the governance process in order to gain acceptance.
We developed a governance layer that works between the AI healing engine and the test repository. Each of the proposed locators’ updates should pass through three validations in order to be considered for an automatic acceptance: proximity of the healed element to the original one, possibility of its location via the same route for the assistive technologies and match between its context and the expected match of the test. The low-risk changes with high similarity score, like ID rename, are getting accepted automatically. Everything else goes to a human reviewer with an auto-created ticket with the original locator, proposed fix and similarity metrics included in it. No decision is made automatically, and any decision, whether automated or human made is recorded.
What is the process of passing through the governed self-healing repair? Structural changes are routed to a human; low-risk and high confidence changes are not.
What We Found
We compared our solution in the form of the governed self-healing with the static CI/CD pipeline with no self-healing and the CI/CD pipeline with AI-based ungoverned self-healing, running them through a 12 month-long simulation with three different windows in banking environment.
The governed self-healing system decreased both the manual maintenance efforts and coverage-erosion incidents at once.
- The static baseline required 180 maintenance hours per month and failed to detect 12 defects.
- The ungoverned AI self-healing system cut this number to 99 hours a month (a huge step in the right direction), but it increased coverage-erosion incidents by four times, from 7 to 28. This problem was mostly caused by locator updates which no one ever verified.
- As compared to both, the governed framework was far superior: 58 hours of maintenance work per month (which is 68% fewer than for the static baseline), a false-positive interception rate of 94%, and 2 escaped defects and 1 coverage-erosion incident during the entire year.
The Part That Surprised Us
Governance didn’t cost speed. It bought more of it.
Initially, it was expected that more validation gates would cost some time. In fact, the governed pipeline made the process more time-efficient: it decreased the maintenance effort even more (68% as compared to 45% of the ungoverned pipeline) and decreased coverage erosion tenfold. Possible reason: some of the “costs” of the ungoverned pipeline weren’t costs at all. Some of the 45% savings in maintenance hours are actually undetected defects. When masked defects are accounted for, governed self-healing no longer seems to be an additional burden over automation, it’s just the more efficient solution.
Where the Value Actually Comes From
We also conducted an ablation study where we eliminated one governance element after another to understand how much value each one provided. There were three main insights from this part of our research.
- Eliminating the human-in-the-loop review process for critical changes proved to be an absolute game changer, as there was a 60% increase in the number of detected critical bugs during the highest volatility migration period. No other element had such a significant impact.
- The automatic gates still played an important role, but only gradually. They detected 15% to 35% more false positive bugs than others did depending on which of the gates had been removed.
- The difference in performance between the governed and ungoverned approaches was insignificant when the user interface was stable and became absolutely huge when the volatility spiked up – that is exactly the moment when a purely automatic validation gate is likely to allow a wrong fix through.
To summarize the key insight for software engineers: if you can only implement one governance component, it should be a human checkpoint for critical changes.
What This Means for Your Team
None of this is a case for not automating tests with AI. It’s a case for ensuring that auditability comes as one of your primary design considerations. Some ideas to begin with:
- Treat each proposed repair by your AI as a suggestion, not a final decision. Nothing needs to end up in your test repository unless it gets validated first and its validation documented.
- Use human intervention only where necessary. Send anything structural, and anything that doesn’t meet your similarity threshold, for human review, but allow automatic processing of everything else.
- Keep a track of things regardless of the result. Record both approved and rejected repairs not just because you have to for compliance, but for identifying patterns in your AI’s mistakes.
- Monitor masked defects, not just maintenance effort. Your AI tool may seem efficient in your dashboard view, yet it could be slowly destroying the quality of your test suite.
Beyond Test Automation
Such a problem is not unique to test automation. It does not matter what industry the product is in and whether it relies on AI or not. An agent that is able to alter the system behavior without being noticed carries the risk of introducing an invisible root cause to the system. Thus, any company that deals with products whose defects go beyond inefficiency – healthcare, insurance, critical infrastructure – faces the same challenge: it needs to be efficient but at the same time maintain visibility. The lesson to be learned from this study is counterintuitive, but true: governance is what enables, not hinders autonomous automation.
Bottom Line
AI self-healing test automation cannot be stopped; the reasons why are obvious. Yet, “the pipeline is green” and “the product works” are two different things and at the moment only the former one is proven by self-healing tools. This gap does not have to be filled at the expense of losing automation – all it takes is a careful choice of what kind of problems the AI can solve autonomously and what should be checked by engineers.
The post The Hidden Risk in Self-Healing Test Automation: A Governance Blueprint for Digital Banking appeared first on SD Times.
Tech Developers
No comments