I love card tricks.
I can watch videos for hours. And then someone comes and spoils the magic by explaining exactly how it’s done.
I’ll be that guy today.
So let’s talk about the magic of self-healing tests. How do they do it, and why the magic involved can have destructive effects.
Self healing tests are usually talked about in regards to UI interfaces. Automated tests use frameworks (Playwright, Selenium and others) to find elements on the screen and press them, click them, drag them around.
But the trick is they have to find them first. That’s a problem (20 year old one). The frameworks don’t see the screens, they read the HTML DOM, and based on it, they search, in different ways for the elements.
If you have a “Next” button on your page, the framework will search for a HTML Button element with the text “Next” on it.
Found it? Good. The test can continue.
Now the button can misbehave. Let’s say it moves a couple of pixels. Or change its color. What will happen?
That depends how we’re looking for the element. If it’s only by name, location and color don’t matter.
But if you have 3 buttons, all with “Next” on them, in 3 different colors, now it’s trickier. We need to tell the framework which button exactly we’re looking for.
No magic required here.
Ok, next step. Our button looks a little bland. So our developer sat with the designer, and created an image that serves as a button. Very stylish.
Only it’s not a button anymore. Now it’s an IMG element. What happens when we’re looking for a button? That’s right, we won’t find it.
Here’s another case. Our original button gets translated to multiple languages. So sometimes it says “Next” and sometimes it says “Suivant” when there are French people around.
Looking for the English text of the button will fail us.
Now what do these failures mean? Our tests check some behavior, but to get to the checking part, it relies on finding the elements to operate and read them.
If the test fails to do so, it will fail. But, here’s the catch. We built the test to check the behavior, and we didn’t even get to that point.
The test failure doesn’t mean what we want. It only means that the setup steps didn’t work.
Or did they?
We need to debug and see what’s going on. And if it’s really a “find the element” problem, that means the developers changed something, and the test needs rewriting.
That’s extra work we don’t really want to do. Oh, if only there was a magician around that will make the test “heal thyself”…
Self-healing tests are not really self-healing. That’s part of the trick. Tests don’t have that power.
Here’s what really happens.
An agent runs the test. It looks at the test result. If it passes – everything is ok. But if it fails, it analyzes the result – reads logs, maybe re-runs, debugs the DOM. It then can come to a conclusion that the test tried to find our button but failed.
It then tries to guess what we’re really after, and if there’s a replacement for that. If we’re looking for a button with “Next” on it, maybe that image with “Next” on it will work. Or maybe it waits a bit more until the button appears.
It rewrites the test and runs it. Has it passed? Great problem solved. The test was healed. Just need to make sure the edits are saved.
Or maybe it didn’t work? It will try different heuristics to find the missing elements. If it finds a suitable replacement it’s the solution. It tries until it comes to a decision there’s none.
And reports a “real” failure. Now it’s a real bug, or at least something that requires debugging.
Ok, you might have noticed a problem with my scenario. If the agent just debugs the HTML, it wouldn’t really know if the image has “Next” on it. Could be an image of a mermaid. Maybe even clickable.
So the level of guessing really depends on how “smart” the agent is. Which really comes down to the data it was trained on.
But the cool thing is – if clicking the mermaid passes the test, the test is healed!
Wait, what?
Yes. There are a few problems here.
The guesses may be wrong, and the tests still fail. That’s not really a problem. We just spent a few tokens and waited a few minutes more for the result.
The guess is right. Yay! But we got the mermaid effect. You read the test, and it doesn’t make sense. You debug it, and you don’t really understand how it actually works. Or why.
That’s a maintenance and knowledge problem. Tests are the real live specs of the app. If we don’t understand them it’s hard to understand what they check, how they come to conclusions that things work or not, and how to fix them.
- The guess is right. Yay! Wait. The test passes now, because the updated test waits a bit more for the button to appear.
But what happened to make the button appear late? An actual back-end bug.
But there’s no bug, according to our test, that now passes.
And so, our self-healing test is now covering up a real bug.
That’s ok, I’m sure a customer will report it.
Self-healing tests are just a hijacked term, that hides power and risk. They promise less time on maintenance – the genie will fix it for ya! – but since they only live in the front-end world, they don’t recognize, and sometimes even hide the real information we need.
That’s the whole trick. See, it doesn’t look as cool when you know how it works.
Originally published at testingil.com.
I'm Gil Zilberfeld. I teach API testing and test automation, and I write about what AI-generated code does to quality.
