AI in Software Testing, and How to Test AI Systems

What is AI-augmented quality engineering?

AI can now help at almost every step of testing: reviewing requirements, drafting test cases, spotting duplicate defects, summarising release risk. It is fast, fluent and occasionally, confidently wrong. This guide has two halves that meet in the middle. First, AI for testing: how to use these assistants well, and how to avoid the quiet danger of trusting them too much. Second, the testing of AI systems: the products you test are starting to contain AI themselves (a chat assistant, a claims classifier, a recommendation), and an AI system fails in ways that ordinary software does not. The aim is to use AI as a strong assistant while staying the one who decides. It draws on the ISTQB Certified Tester AI Testing (CT-AI) syllabus, the OWASP Top 10 for Large Language Model Applications and the NIST AI Risk Management Framework.

What you will be able to do after this guide

• Classify the ways AI helps testing, and say where it should not be used.
• Recognise and counter automation bias, and prompt AI tools safely and effectively.
• Evaluate an AI tool's claims with measurements (precision, recall) on your own data.
• Run the full QE cycle on a feature with AI at each step, while recording your own judgment.
• Explain how machine-learning systems and large language models fail, and how to test them (data, metamorphic tests, evaluation sets, safety).
• Name the main security risks of LLM applications, and the responsible-AI concerns of bias, privacy and transparency.

PART 1: AI FOR TESTING

1. Where AI actually helps a tester: a precise taxonomy

ISTQB CT-AI groups AI-for-testing into three categories. Mapped onto the capabilities of an AI-assisted test management tool such as SQValex:

Other places AI is used in testing: summarising long defect threads and logs, drafting test data and boundary cases, explaining a failure from a stack trace, suggesting locators or repairing a broken one (with care, because repairing a locator can hide a real defect), prioritising tests by predicted risk, detecting visual differences, and triaging incoming defects.

2. Where AI is deliberately not used, and why that is not a gap to fill

Per CT-AI's own scope, today's narrow AI is not well suited to specifying test oracles from scratch, resolving genuine ambiguity by asking the right clarifying question, proposing improvements grounded in real user empathy, or challenging a stakeholder's assumption with informed pushback. Those need general reasoning and product and business context.

This is not a temporary gap waiting for a better prompt. It is structural: narrow AI performs a bounded task well; it does not understand your business the way you do after months on the product. Treat every AI output as a fast first draft from a capable but context-blind assistant, never as a decision. Three things AI cannot give you, and which remain yours: accountability (you sign the report), judgment about risk appetite, and knowledge of what the business really meant.

3. Automation bias: the risk this week is testing for

Research cited in the CT-AI syllabus found that over-reliance on an automated recommendation can measurably reduce decision quality, even when the human believes they are still exercising judgment. Fluent, well-formatted answers feel authoritative, which makes this worse. Habits that counter it:

• Form your own view first, then compare with the AI's. If you read the AI answer first, you anchor on it.
• Spot-check on purpose. Verify a sample of AI outputs against the source, even when they look right, and record how often it was wrong.
• Ask for evidence. Prefer outputs that cite the requirement, line or defect they come from, and check the citation is real.
• Keep a record of overrides: what you changed or rejected, and why. It is your evidence that you stayed the decision-maker, and it tells you where the tool is weak.

4. Prompting well, and protecting data

AI output is only as good as the input. A reliable structure for a testing prompt:

1. Role and goal: "You are a senior QA engineer reviewing requirements for an insurance quote system."
2. Context: the requirement text, the rule source, related requirements, the audience.
3. Task: specific and bounded ("list ambiguous phrases, then propose a testable rewrite for each").
4. Constraints: what to avoid, the technique to apply ("use three-value boundary analysis on ages 16 to 99"), and "state your assumptions".
5. Output format: a table, with columns you name, so the result is easy to review.
6. Examples of a good answer, when the format matters.

Then iterate: ask it to find gaps in its own answer, to explain its reasoning, or to challenge your assumption. Practise the structure on your own requirements until it is a habit.

Protect data. Anything you paste into an AI tool may be stored or used by a third party, depending on the service. Never paste personal data, customer records, credentials, API keys or confidential designs. Use synthetic or anonymised examples. Check your organisation's policy and the tool's data terms before use. Remember also that AI services have cost and rate limits, and that outputs can be non-deterministic: the same prompt can return different answers, so do not treat one run as proof.

5. Evaluating an AI tool, with numbers

A vendor will say their AI "finds duplicates with 95% accuracy". Do not accept that: measure it on your data. Take 100 pairs of defects where you know the truth, 20 of which are genuine duplicates. The tool flags 25 pairs, and 16 of those are real duplicates.

• 100pairs of defects checked
• 20are genuine duplicates
• 25pairs the tool flagged
• 16flagged and genuinely duplicates

• Precision = 16 ÷ (16 + 9) = 64%: of the pairs it flagged, how many were right. Low precision wastes your time on false alarms.
• Recall = 16 ÷ (16 + 4) = 80%: of the real duplicates, how many it found. Low recall means duplicates slip through.
• F1 score = the balance of the two = 2 × 0.64 × 0.80 ÷ (0.64 + 0.80) ≈ 71%.
• Accuracy = (16 + 71) ÷ 100 = 87%, which sounds excellent and hides the 36% false-alarm rate. Accuracy misleads when one class is rare (here only 20% are duplicates), which is why precision and recall matter.

• Accuracy87%
• Recall80%
• F1 score71%
• Precision64%

Which to favour depends on the cost of each mistake. For duplicate detection a false alarm is cheap (you look and dismiss), so favour recall. For an AI that closes defects automatically, a false positive is costly, so favour precision. Also ask: does it explain its answer, how does it behave on unusual inputs, does it drift over time, and where does your data go?

6. Running the full cycle on one new feature

The cycle runs in five steps, and your judgment sits between every pair of them:

1. An AI-reviewed requirement.
2. AI-generated draft test cases.
3. Execution.
4. Defects raised, with an AI duplicate check.
5. An AI executive summary of release readiness.

1. 1AI-reviewed requirementAmbiguities and gaps flagged before any test is written
2. 2AI-drafted test casesA first draft for you to keep, change or reject
3. 3ExecutionRun the tests; the evidence is real
4. 4Defects raisedWith an AI duplicate check you can verify
5. 5AI executive summaryThe first draft of the release story, checked against the data

For a feature such as "multi-policy discount bundling auto and home", you touch every stage of the quality engineering cycle. The difference is that you now watch your own process: at each arrow, are you reviewing the AI output critically, or rubber-stamping it because the week is long and the AI is fast? Write one sentence at each step on what you kept, changed or rejected.

AI executive summaries are a synthesis tool, not a release decision. A summary can digest dozens of defects and results into something a release owner reads in two minutes. It cannot decide whether to ship: that weighs business priorities, risk appetite and context it does not have. Treat it as the first draft of the executive summary, and check it against the underlying data before anyone relies on it. Look for what it smooths over: a single blocking defect described as "minor issues remain".

PART 2: TESTING AI SYSTEMS

7. Why AI systems need different testing

Ordinary software follows rules a programmer wrote, so the same input gives the same output and the specification says what is right. A machine-learning (ML) system learns its behaviour from data. Consequences for the tester:

• The data is part of the specification. Defects can live in the training data, not the code.
• Outputs are statistical. A model is right most of the time; "correct" becomes a measured rate, not a yes or no.
• The oracle problem is harder. There may be no single known-correct answer to compare against.
• Behaviour changes when the data or model is retrained, even if no code changes. Regression testing never ends.
• Failures can be subtle and unfair: a model can work well overall and badly for one group.

Machine-learning basics a tester needs. Supervised learning learns from labelled examples (claims marked fraud or not fraud); unsupervised learning finds patterns without labels (grouping similar customers); reinforcement learning learns by trial and reward. Data is split into training data (to learn), validation data (to tune) and test data (held back, to measure honestly on unseen examples). A model that does well on training data and badly on new data has overfitted: it memorised instead of learning. A common test-design defect is data leakage, where information from the test set sneaks into training and inflates the results.

8. What to test in an ML system

Metamorphic testing: a way round the oracle problem

When you cannot say what the right answer is, you can still say how the answer should change when the input changes in a known way. These metamorphic relations become tests:

• A premium-estimating model: making a driver one year older must not lower the risk score for a very young driver; adding a recorded claim must not reduce the premium.
• A text classifier: rewording a complaint without changing its meaning must not change its category.
• Fairness: swapping a customer's name or gender, with everything else identical, must not change an eligibility decision.
• A summariser: the summary of a document plus an irrelevant sentence should not materially change.

If a relation is violated, you have found a defect without ever needing the "correct" answer.

9. Testing large language models, RAG and agents

Large language models (LLMs) generate text by predicting likely continuations. They are fluent and general, and they fail in characteristic ways:

Retrieval-augmented generation (RAG) gives the model relevant documents to answer from. Test the two halves separately: retrieval (did it fetch the right passages? measure precision and recall of retrieved chunks) and generation (is the answer faithful to those passages, or does it add claims not in them?). A great answer built on the wrong document is still a defect.

Agents let a model take actions with tools (look up a policy, send an email, change a record). They add a new risk: excessive agency, where the agent can do more than it should. Test with the least privilege idea: what is the worst the agent can do if it is tricked? Check confirmation steps for risky actions, limits on loops and cost, and that every action is logged.

Evaluation sets and regression for prompts

Because outputs vary, you test an LLM feature with an evaluation set (a "golden set"): a curated list of inputs with expected properties, run each time you change the prompt, model or data, scored automatically where possible. An LLM-as-judge (another model scoring the output) is useful for scale but can be biased and wrong, so calibrate it against human ratings. Track the pass rate over time. Treat a prompt like code: version it, review changes, and run the evaluation set before release (the same idea as a CI pipeline, applied to AI).

10. Security and responsible AI

The OWASP Top 10 for Large Language Model Applications lists the main risks, including prompt injection, sensitive information disclosure, improper output handling, excessive agency, system prompt leakage, misinformation and unbounded consumption (resource and cost abuse). Several are the AI-era cousins of classic web security weaknesses: untrusted input reaching an interpreter, over-privileged components, and missing limits.

Responsible AI adds concerns that testers should help raise:

• Bias and fairness: unequal outcomes for different groups, which can be unlawful as well as unethical.
• Privacy and data protection: what personal data trains or is sent to the model, consent, retention, and laws such as India's Digital Personal Data Protection Act and the GDPR.
• Transparency and explainability: people affected by a decision may be entitled to an explanation.
• Human oversight: a person who can review and override important decisions.
• Regulation: rules are arriving, such as the EU AI Act, which sorts AI systems by risk. For an insurer, systems that price or approve cover affect people materially and deserve extra care.

A framework such as the NIST AI Risk Management Framework (govern, map, measure, manage) gives a structure for talking about these risks in an organisation.

11. A broader picture: where AI for testing is heading

Research and tools are moving towards test generation from requirements and code, self-healing locators (useful, but they can hide real defects), prediction of risky areas from process and organisational signals and not only code metrics, and autonomous agents that explore an application. None of these is part of SQValex's current toolset, but knowing they exist helps you reason about what is coming, and to keep evaluating each new feature with the same lens: what does it help with, what does it still need a human for, and how would I measure whether it works?

12. Common mistakes

1. Accepting an AI answer because it is fluent and well formatted.
2. Reading the AI's answer before forming your own.
3. Pasting personal data, secrets or confidential designs into an AI tool.
4. Judging an AI tool by "accuracy" alone, on someone else's data.
5. Testing an ML model only with the data it was trained on.
6. Treating a single LLM run as proof, when the output varies.
7. Trusting an LLM-as-judge without checking it against human ratings.
8. Giving an agent more access than its job needs.

13. Check yourself

1. What is automation bias? Over-trusting an automated recommendation and letting it replace your own judgment.
2. A duplicate detector flags 25 pairs, 16 correctly, and misses 4 real duplicates. What is its precision and recall? Precision 64% (16 of 25); recall 80% (16 of 20).
3. Why can accuracy mislead? When one class is rare, a model can score high accuracy while being poor at the class that matters.
4. What is overfitting? A model that fits its training data too closely and performs badly on new, unseen data.
5. How does metamorphic testing help when there is no known correct answer? It checks how the output should change when the input changes in a known way, so violations are defects.
6. What is prompt injection? Input text that overrides the model's instructions, including text hidden in documents it reads.
7. Why test RAG retrieval and generation separately? A good answer from the wrong document, or a wrong answer from the right one, are different defects.
8. What is excessive agency? An agent being able to take actions beyond what its job needs, so a trick or error can do real harm.

14. Interview questions, with short answers

1. How would you use AI in testing responsibly? As a fast first draft I review, with my own view formed first, spot-checks, a record of overrides, and no sensitive data in prompts.
2. How do you test a machine-learning model? Data quality, performance metrics on held-out data, bias across groups, robustness, explainability and drift, plus metamorphic tests where there is no clear oracle.
3. How do you test an LLM-based feature? An evaluation set run on every change, checks for hallucination, inconsistency and injection, safety red-teaming, and treating the output as untrusted input.
4. What is the difference between precision and recall? Precision is the share of flagged items that are right; recall is the share of real items that were found.
5. What are the main risks of an AI agent? Excessive agency, prompt injection through the content it reads, runaway cost or loops, and actions that are hard to undo.
6. How do you check an AI-written summary? Compare it with the underlying data and look for what it smooths over or omits.

Practise it

• Take a feature such as "multi-policy discount bundling auto and home" from a blank requirement through the full cycle: AI-reviewed requirement, AI-generated draft test cases, execution, defects raised and triaged with an AI duplicate check, and an AI executive summary.
• For at least three AI outputs along the way, write one sentence on what you changed or rejected, and why. This is your evidence that you stayed the decision-maker.
• Compare an AI executive summary against the underlying defect and test data: does it represent release readiness accurately, or smooth over something that matters?
• Calculate precision and recall for one AI suggestion type from a sample you checked by hand, and say which you would favour and why.

In our AI-Led Quality Engineering course this is Week 15, with hands-on labs for metamorphic testing and prompt-injection cases. To go deeper on the basics of test design first, read test design techniques.