If software testers were chefs, classic tests would be like following a recipe: add salt, stir, cook for 20 minutes. However, when testing AI-based systems, we increasingly feel more like the judges on MasterChef than chefs. The dish is ready, but we don’t know the recipe, so our task is to assess whether it tastes as it should. This is where an enigmatic, unofficial but increasingly popular term comes into play: Vibe testing.
This relatively new approach to testing and quality control is gaining importance as artificial intelligence plays an increasingly significant role in the software development process. It attempts to answer the increasingly frequent question of how to evaluate an application that was largely created by AI.
The concept grew out of Vibe coding, which assumes that the programmer defines their intentions, and the code then creates a language model based on these intentions. Vibe testing goes beyond checking whether an application works correctly from a technical standpoint, i.e. whether it meets the formal requirements specified in the documentation. It also assesses whether the application behaves in accordance with the creator’s general intention — whether it fits the context and communicates in a manner consistent with the intended tone, style, and user experience. It’s similar to the difference between someone who has followed our instructions literally and someone who has truly understood us.
Where does this ‘vibe’ come from?
The word ‘vibe’ itself has an elusive quality. In the slang of younger generations, it can mean ‘atmosphere’, ‘feeling’ or ‘energy’. In the world of AI, ‘vibe’ is simply a system response that sounds odd. There is no logical bug. Everything seems correct, yet something is amiss.
An example? Consider a recommendation system in an English language learning app. After finishing a lesson about pets, the AI suggests an article on ‘The philosophical aspect of pig consciousness in posthumanist literature’. Is this a topic related to animals? Technically, yes. But does it make sense as a suggestion after a beginner’s lesson? Not necessarily.
That ‘not necessarily’ sums up the feeling. It’s something that’s difficult to capture with a classic functional test.
Vibe testing is the art of catching AI out when it does something strange, not when it does something obviously wrong. Too bold. Or too cautious. Or, on the contrary, disturbingly detached.
So, how does this process work in practice?
The first step is to put yourself in the user’s shoes. It’s not just about the accuracy of the data; it’s also about the impression left by the interaction. It’s like after a conversation with someone: sometimes they’ve said everything correctly, but we don’t feel like asking any more questions.
Vibe testing is the moment when we ask ourselves: ‘As a user, do I feel taken seriously, understood and encouraged to continue interacting?’.
After conducting a dozen or so tests, it becomes easy to spot recurring errors in terms of style, tone, and content selection. Does the model apologise for everything? Vibe: too formal. Does the model always agree with the user? Vibe: too docile. Does the model always cite sources, even when it is not necessary? Vibe: artificial.
Vibe testing versus classic testing
In traditional testing, we create test cases comprising an input condition, an expected result and an assertion. Simple, clear and measurable.
In Vibe testing, however, we don’t find hard metrics unless we create our own (and some people do). We don’t compare the output to a reference result file. Our focus is not on bugs, but on identifying stylistic deviations that disrupt the user experience. You could say that Vibe testing is closely related to UX or exploratory testing, but with an added psychological and narrative analysis element.
An anecdote from the battlefield
I recently spoke with a tester who was involved in testing AI for generating automatic property descriptions. The system worked well: the descriptions were grammatically correct and neat, and matched the data. But then the AI started using words like ‘a magical place full of nostalgia and the scent of memories’. This did not fit with the corporate sales platform. The model had not made a mistake, but it had crossed an invisible line of good taste. The tester reported this as a ‘style drift’ — after all, that’s what Vibe testing is all about.
Can the vibe be tested automatically?
This question keeps coming back like a boomerang. The answer is… it’s complicated.
Some companies try to use AI models to evaluate other models, a process known as ‘model-on-model evaluation’. Example: GPT-4 evaluates the response of GPT-3.5. Others build tone classifiers that detect ‘emotional inadequacy’ or ‘stiffness of speech’. Some even train a meta-model that learns from manually labelled examples. ‘This response sounds natural’, ‘This one sounds like it was written by a 90s robot’.
For now, however, Vibe testing remains largely manual. And maybe that’s a good thing. A tester with empathy and an ear for language will pick up on nuances that are transparent to a machine.
Vibe testing is a skill for the future
It may sound like a passing fad today, but Vibe testing will eventually become as necessary a skill as security or performance testing. Why?
Because users don’t abandon AI just because something went wrong. They leave because something was disturbing. Because the chatbot was ‘weird’. Because the recommendation system used a voice that didn’t fit the brand. Because the AI didn’t recognise irony. Or because it sounded like it was from another planet.
If you’re not sure whether an AI response is appropriate, say it out loud to someone. It could be a friend, a colleague, or even your dog (if they’re patient enough). If you feel slightly embarrassed reading this statement, you have just identified a ‘Vibe issue’.
