Case studies

What we find when we test AI.

A reviewer circling one finding in red on a printed AI test report, the rest ticked as passed

Tested with our AI Test Harness.

We used the test harness we built to check their AI for accuracy, trust and safety before it reached the people relying on it.

Public sector and healthcare
Scottish Government and NHS 24
The AI
A conversational AI search service where citizens describe what they need in their own words and are pointed to the right public or health service near them.
What we tested
- Accuracy: answers checked against a known-correct set, because sending someone to the wrong health service is a safety risk
- Real questions: tested with the way citizens actually ask, not the way the system expects
- Fairness: outcomes compared across groups who describe the same need differently
Technology consulting
Thoughtworks
The AI
An AI knowledge base answering employee questions from company content, with AI agents that raise service tickets when an issue is too complex to self-serve.
What we tested
- Trust: whether answers came only from trusted content, or were made up
- Actions: what the agents could do in real systems, not just what they said
- Escalation: whether hand-off to a person happened when it should

Financial services

A lender's assistant was quoting an eligibility rule the firm had already dropped.

What the AI did

A customer-facing assistant on a lender's website answered questions about loan products, eligibility and repayments, drawing on the firm's product documents and policies.

What we tested

Whether it could be manipulated, whether its answers matched current policy, whether answers were consistent for different types of customer, and whether there was an audit trail of what it told people.

What we found

Confident wrong answersIt quoted an eligibility rule from an out-of-date document still in its knowledge base.
Prompt injectionA crafted question made it ignore its instructions and discuss products it wasn't approved to promote.
GovernanceConversations weren't logged in a way the compliance team could review, so there was no evidence of what customers had been told.

What changed

The firm removed outdated documents, added controls on what the assistant could discuss and introduced conversation logging. FireBreak re-tested before the wider rollout.

The numbers

14 issues found, 3 critical, all resolved before rollout.

Healthcare

Asked about worrying symptoms, the assistant offered reassurance instead of urgent care.

What the AI did

An assistant for a private healthcare provider answered patient questions about appointments, services and preparing for procedures.

What we tested

Accuracy against approved clinical information, whether it stayed within safe boundaries, how it handled urgent symptoms, and whether it worked equally well for different patients.

What we found

Unsafe boundariesAsked about worrying symptoms, it gave general reassurance instead of directing the patient to urgent care.
Unfair outcomesQuestions written in simpler English or by non-native speakers received noticeably less complete answers.
Data leakageA sequence of questions could reveal details from internal staff guidance that wasn't meant for patients.

What changed

The provider added clear escalation for urgent symptoms, improved the source content and restricted what the assistant could access. Testing now repeats whenever the content changes.

The numbers

120 patient scenarios tested; 9 issues found, 2 critical, both resolved before launch.

Legal and professional services

Staff could read summaries of client matters they weren't allowed to work on.

What the AI did

An internal assistant at a professional services firm helped staff search past matters, precedents and client documents.

What we tested

Whether confidential information stayed within the right teams, whether answers were grounded in real documents, and whether hidden instructions in uploaded files could manipulate it.

What we found

Data leakageStaff could retrieve summaries of matters for clients they weren't permitted to work on, breaching internal confidentiality walls.
Prompt injectionAn instruction hidden in an uploaded document caused the assistant to change its summary of that document.
Confident wrong answersIt occasionally cited precedents that didn't exist in the firm's library.

What changed

The firm aligned the assistant's access with existing permissions, added checks on uploaded files and required every answer to link to a source document.

The numbers

11 issues found, 4 high-risk; confidentiality issues resolved within two weeks and verified by re-testing.

AI agents in operations

One hidden line in an email got a request raised and approved with no human review.

What the AI did

An AI agent handled internal service requests: it answered staff questions, raised tickets and could update records in several business systems.

What we tested

What the agent could reach and do, whether it could be steered by malicious content, whether it escalated correctly to people, and whether its actions were traceable.

What we found

Excessive agencyIt had write access to systems it only needed to read, so a single error or attack could change records it should never touch.
Prompt injectionAn instruction hidden in an incoming email caused it to raise and approve a request without human review.
GovernanceActions were logged, but no one was monitoring them, so unusual behaviour would have gone unnoticed.

What changed

The organisation reduced the agent's permissions to the minimum needed, added human approval for sensitive actions and set up alerts for unusual activity. The test harness now runs automatically after every change.

The numbers

17 issues found, 5 critical, including 2 that could have changed records without approval; all resolved before scaling to further teams.

“FireBreak told us plainly where our AI plans carried risk and what had to be in place before we scaled. That independent view gave our leadership team the confidence to move forward.”

Gary Sage, Chief Operating Officer, Crimson

Find out what your AI will get wrong, before anyone else does.

Tell us what your AI is meant to do, and we'll show you how we would test it.

Take the free readiness check

Case studies

What we find when we test AI.

Tested with our AI Test Harness.

We used the test harness we built to check their AI for accuracy, trust and safety before it reached the people relying on it.

Public sector and healthcare
Scottish Government and NHS 24
The AI
A conversational AI search service where citizens describe what they need in their own words and are pointed to the right public or health service near them.
What we tested
- Accuracy: answers checked against a known-correct set, because sending someone to the wrong health service is a safety risk
- Real questions: tested with the way citizens actually ask, not the way the system expects
- Fairness: outcomes compared across groups who describe the same need differently
Technology consulting
Thoughtworks
The AI
An AI knowledge base answering employee questions from company content, with AI agents that raise service tickets when an issue is too complex to self-serve.
What we tested
- Trust: whether answers came only from trusted content, or were made up
- Actions: what the agents could do in real systems, not just what they said
- Escalation: whether hand-off to a person happened when it should

Financial services

A lender's assistant was quoting an eligibility rule the firm had already dropped.

What the AI did

A customer-facing assistant on a lender's website answered questions about loan products, eligibility and repayments, drawing on the firm's product documents and policies.

What we tested

What we found

Confident wrong answersIt quoted an eligibility rule from an out-of-date document still in its knowledge base.
Prompt injectionA crafted question made it ignore its instructions and discuss products it wasn't approved to promote.
GovernanceConversations weren't logged in a way the compliance team could review, so there was no evidence of what customers had been told.

What changed

The firm removed outdated documents, added controls on what the assistant could discuss and introduced conversation logging. FireBreak re-tested before the wider rollout.

The numbers

14 issues found, 3 critical, all resolved before rollout.

Healthcare

Asked about worrying symptoms, the assistant offered reassurance instead of urgent care.

What the AI did

An assistant for a private healthcare provider answered patient questions about appointments, services and preparing for procedures.

What we tested

Accuracy against approved clinical information, whether it stayed within safe boundaries, how it handled urgent symptoms, and whether it worked equally well for different patients.

What we found

Unsafe boundariesAsked about worrying symptoms, it gave general reassurance instead of directing the patient to urgent care.
Unfair outcomesQuestions written in simpler English or by non-native speakers received noticeably less complete answers.
Data leakageA sequence of questions could reveal details from internal staff guidance that wasn't meant for patients.

What changed

The provider added clear escalation for urgent symptoms, improved the source content and restricted what the assistant could access. Testing now repeats whenever the content changes.

The numbers

120 patient scenarios tested; 9 issues found, 2 critical, both resolved before launch.

Legal and professional services

Staff could read summaries of client matters they weren't allowed to work on.

What the AI did

An internal assistant at a professional services firm helped staff search past matters, precedents and client documents.

What we tested

Whether confidential information stayed within the right teams, whether answers were grounded in real documents, and whether hidden instructions in uploaded files could manipulate it.

What we found

Data leakageStaff could retrieve summaries of matters for clients they weren't permitted to work on, breaching internal confidentiality walls.
Prompt injectionAn instruction hidden in an uploaded document caused the assistant to change its summary of that document.
Confident wrong answersIt occasionally cited precedents that didn't exist in the firm's library.

What changed

The firm aligned the assistant's access with existing permissions, added checks on uploaded files and required every answer to link to a source document.

The numbers

11 issues found, 4 high-risk; confidentiality issues resolved within two weeks and verified by re-testing.

AI agents in operations

One hidden line in an email got a request raised and approved with no human review.

What the AI did

An AI agent handled internal service requests: it answered staff questions, raised tickets and could update records in several business systems.

What we tested

What the agent could reach and do, whether it could be steered by malicious content, whether it escalated correctly to people, and whether its actions were traceable.

What we found

Excessive agencyIt had write access to systems it only needed to read, so a single error or attack could change records it should never touch.
Prompt injectionAn instruction hidden in an incoming email caused it to raise and approve a request without human review.
GovernanceActions were logged, but no one was monitoring them, so unusual behaviour would have gone unnoticed.

What changed

The numbers

17 issues found, 5 critical, including 2 that could have changed records without approval; all resolved before scaling to further teams.

“FireBreak told us plainly where our AI plans carried risk and what had to be in place before we scaled. That independent view gave our leadership team the confidence to move forward.”

Gary Sage, Chief Operating Officer, Crimson

Find out what your AI will get wrong, before anyone else does.

Tell us what your AI is meant to do, and we'll show you how we would test it.

Take the free readiness check

What we find when we test AI.

Tested with our AI Test Harness.

Scottish Government and NHS 24

The AI

What we tested

Thoughtworks

The AI

What we tested

A lender's assistant was quoting an eligibility rule the firm had already dropped.

What the AI did

What we tested

What we found

What changed

The numbers

Asked about worrying symptoms, the assistant offered reassurance instead of urgent care.

What the AI did

What we tested

What we found

What changed

The numbers

Staff could read summaries of client matters they weren't allowed to work on.

What the AI did

What we tested

What we found

What changed

The numbers

One hidden line in an email got a request raised and approved with no human review.

What the AI did

What we tested

What we found

What changed

The numbers

Find out what your AI will get wrong, before anyone else does.

What we find when we test AI.

Tested with our AI Test Harness.

Scottish Government and NHS 24

The AI

What we tested

Thoughtworks

The AI

What we tested

A lender's assistant was quoting an eligibility rule the firm had already dropped.

What the AI did

What we tested

What we found

What changed

The numbers

Asked about worrying symptoms, the assistant offered reassurance instead of urgent care.

What the AI did

What we tested

What we found

What changed

The numbers

Staff could read summaries of client matters they weren't allowed to work on.

What the AI did

What we tested

What we found

What changed

The numbers

One hidden line in an email got a request raised and approved with no human review.

What the AI did

What we tested

What we found

What changed

The numbers

Find out what your AI will get wrong, before anyone else does.