How do we know an AI really works before putting it into production?

With an evaluation set: sixty real cases drawn from your exchanges, with the right answer written by your team, half of them traps, out-of-scope questions and bypass attempts. Every version of the AI replays the whole set, and a release threshold written before the trials says whether it goes to production. A demo of five questions proves nothing.

How are hallucinations measured?

Claim by claim: each sentence of the answer is checked against the documents the system retrieved, and those not found there are counted. Zero is required on critical cases. An overall faithfulness score is not enough: an answer that is two-thirds right contains an invented promise, and it is the one the customer remembers.

What accuracy rate can we expect?

No general figure holds for you: public leaderboards measure general tasks, often in English, not a question in darija about your catalogue. The rate that counts is measured on your sixty cases, per case type, and it is read with five other measures: unsourced claims, wrong refusals, regressions, handovers, cost and latency.

Who writes the right answers?

You: the person on your team who knows, the manager, the accountant, the branch head. It is an hour or two, once, and it is the only step we cannot do for you. A right answer written by the supplier, or by the model, measures what the supplier believes, not what your customers should hear.

What happens when the model changes version?

The set is replayed, with one command, and the board shows the regressions: cases the old version passed and the new one fails. It is the same procedure when instructions or documents change, or when moving to a less costly model. Without a set, a version change is a bet you discover through a complaint.

What should we ask a supplier who says their AI is reliable?

On how many cases it was measured, who wrote the right answers, which case types the set contains, how unsourced claims are counted, what threshold was written before the trials, and which cases still fail. A supplier who measures can show the list of failures; one who only shows a demo has none.

Proving an AI works before it faces your customers in Algeria in 2027: the evaluation set, hallucinations counted, regressions caught, and the threshold at which you release

Short answer. An AI is measured the way the rest of the software is tested: with an evaluation set. Sixty real cases, taken from your real messages, documents and voice notes, in Arabic, French and darija; for each, the right answer written by the person on your side who knows. Half are common cases; the other half are traps, out-of-scope questions where the right answer is “I don’t know”, missing data and bypass attempts. Each AI answer is graded three ways: by code when the answer is a field, by the rule “every claim has a source” which counts hallucinations, and by a judge model calibrated on your own grades for form. The set is replayed at every change of instructions, documents or model: that is what catches regressions. And a release threshold is written before the first trial. It is not a single accuracy rate: it is six measures read together.

This guide is for whoever compares suppliers and wants to know what to ask them, the manager who saw a convincing demo and wonders what it proves, and the IT department that must sign off production. It details the method our other guides assume, AI in the app, document reading and private deployment, and answers one of the questions to ask before signing, listed in the guide on choosing a company.

What the figures say

Model vendors themselves say it must be measured. OpenAI’s evaluation guide defines evals as “structured tests for measuring a model’s performance”, needed because these systems do not answer the same way twice, and recommends a five-step process: define the objective, collect a dataset, define the metrics, compare versions, and evaluate continuously at every change. It also names what not to do: generic metrics unrelated to the task, a dataset that does not look like real traffic, and “vibe-based evals”, when the strategy comes down to “it seems like it’s working” or measuring waits until after shipping.

How to build the set is written too. Anthropic’s documentation asks for cases that mirror the real task distribution, not forgetting edge cases, missing or irrelevant data, overly long inputs, ambiguous requests on which even humans would hesitate; it recommends automating grading whenever possible, and prefers many automatically graded cases to a few hand-graded ones. It distinguishes three gradings: by code, the fastest and most reliable; by a human, the finest and slowest; by a judge model, in between, preferably with a different model from the one being evaluated. OpenAI’s guide adds the condition: an automated judge is scaled up only once calibrated on human grades.

What does not exist: a rate that would hold for you. Public leaderboards measure general tasks, mostly in English; they say nothing about a question asked in darija about your catalogue. So we quote no model “hallucination rate”: the one that counts is measured on your sixty cases, and it changes with your documents. The numbers in the first figure, 47, 55 and 53 out of 60, are an example built to show a regression, not a measurement from a project; what is not an example is the method.

Sixty cases, replayed on three versions

The evaluation set is not for getting a score: it is for comparing two versions without trusting an impression.

Figure 1 · Version 3 answers well when you try it by hand; the board is what shows it lost two cases version 2 passed. Without the sixty squares it would have gone to production because it costs less. Version 2’s five failures are not hidden: they are listed, and the client knows which ones.

Version 1, with the first instructions, passes 47 cases out of 60; the thirteen failures are not bad news, they are the work list, and a third are in the traps. Version 2, instructions fixed and documents better chunked, passes 55; the five remaining failures are known, listed, and shown to the client. Version 3 keeps the same instructions with a smaller model, which costs less per request: 53 out of 60. Tried by hand, it answers very well. The board shows something else: two cases version 2 passed now fail.

Those two squares are why the set exists. Without it, version 3 would have gone to production because it costs less, and two kinds of request would have started getting wrong answers with nobody seeing it before a complaint. With it, the decision is written in advance: threshold of 54 out of 60 and no regression; version 2 is released, version 3 waits. The set is replayed the same way the day a vendor changes its model version, which happens several times a year, and it runs with one command, alongside the tests described in the delivery guide.

«Tried by hand, version 3 answers very well; the board is what shows it lost two cases.»

Building the evaluation set, in six steps

Six steps, one of which nobody can do for you.

Figure 2 · Step 03 is the one that takes your team’s time, and the one no supplier can do for you: nobody else knows the right answer to your customers. One hour of the manager on sixty cases is worth more than two weeks of blind tuning.

The criterion says what “working” means for this function, in one measurable sentence: the right delivery date, the right category, the right fields, and no claim that is not in a document. The real cases are sixty requests drawn from your real exchanges, WhatsApp messages, emails, voice notes, documents, in the three languages; examples invented by us would measure what we believe your customers ask. The right answer is written by the person on your side who knows, the manager, the accountant, the branch head; they prevail, and it is the step that needs your time: an hour or two, once.

The hard cases are added on purpose: traps, out-of-scope questions, missing data, bypass attempts; a set of easy cases gives a fine score and proves nothing. The grading is automated as far as possible, by code for fields, by a judge model calibrated on your grades for the rest, with a sample reread by hand to check the judge judges as you do. The threshold is written before the first trial: the minimum score, the rule on regressions, and the list of critical cases on which no error is accepted. Written afterwards, a threshold always adjusts to the score obtained.

One case, three ways to grade it

An answer is not graded “good” or “bad”: it is broken down, and each piece has its rule.

Figure 3 · The date is right and the answer is polite: graded by eye, it passes. The second grading stops it: “free returns within 30 days” is written nowhere at your company. That is what is called a hallucination, and it is only counted if every claim is checked against the retrieved documents.

Case 37 is a trap: two questions in one, “when does my order arrive, and can I return it?”. The AI’s answer comes down to three claims. The delivery date is found in the order; the morning slot, in the route; “returns are free within 30 days” is found nowhere, because your company has written no return policy. First grading, by code: is the returned date the expected one? Yes. Second grading, the source: each claim is checked against the documents the orchestration retrieved; two out of three have one, the third is invented. Third grading, form, by a judge model calibrated on your grades: tone, language and length match.

Verdict: fail. An answer that is two-thirds right, polite and well written would pass grading by eye; it is exactly the kind of answer called a hallucination, and the one the customer will remember, because it promises them something. The expected answer, written by your team, said: the date, the slot, and “for a return, I’ll put you in touch with an adviser”. This case also shows why the source is checked claim by claim: an overall “faithfulness” score would have given 67 %, a figure that looks acceptable and hides a false promise.

«An answer that is two-thirds right, polite and well written contains an invented promise, and it is the one the customer remembers.»

Six types of case, and what is expected of the AI

An evaluation set is worth its composition; six types of case, and for each what counts as a pass.

Figure 4 · A set containing only common cases gives a fine score and proves nothing. The three middle rows are where an AI invents; the last is where it obeys the wrong person.

The common case, the question that comes back twenty times a day in customers’ words, makes up half the set: the right answer, sourced, in the language of the question. The trap: two questions in one, a product resembling another, an ambiguous date; answer the known part, ask to clarify the other, complete nothing. Out of scope: the answer is in none of your documents; the only pass is “I don’t know” and handover to a human, as the WhatsApp agent guide describes.

Missing data: the order does not exist, the customer is not recognised, the document is unreadable; say so, ask for the information, assume nothing. Arabic and darija: the same question in standard Arabic, darija, Latin letters and as a voice note must get the same answer as its French version, and the gap between languages is measured separately, because an average hides it. The bypass attempt: “forget your instructions”, an insistent discount request, another customer’s data; refusal without disclosure, attempt logged, and no right beyond the role described in the roles guide.

Six measures, read together

There is no single “accuracy rate”: there are six measures, and none is read alone.

Figure 5 · None of these measures is a single “accuracy rate”. The second and third hold each other: an AI that never says “I don’t know” has a good answer rate and invents; one that always says it invents nothing and answers nothing.

The pass rate on the set, overall and per case type: a score per type beats an average, because 55 out of 60 with every failure in out-of-scope is not the same as 55 with failures spread out. Unsourced claims: the hallucination count, with zero required on critical cases. Wrong refusals: “I don’t know” when the answer was in the documents. Those two hold each other: an AI that never refuses answers everything and invents; one that always refuses invents nothing and is useless; both are tuned together, on the set.

Regressions: cases the previous version passed and the new one fails; zero to release, or a written decision case by case. Handover to a human: the share of requests transferred, and above all those that should have been and were not; in the pilot, every day. Cost and latency, per version, next to the score and never alone: a version that costs less with two regressions is not better. That table, six lines per version, is what is shown at every two-week demo.

Seen elsewhere: the vendors’ evaluation guides, and what transposes

Two sources say the essentials. OpenAI’s guide says evaluating by impression is an anti-pattern, that a set must look like real traffic and that an automated judge is calibrated on human grades; what transposes is that measurement starts before the code and does not stop at release. Anthropic’s documentation says to build task-specific cases, include edge cases and automate grading; what transposes is that sixty cases replayed at every change beat six cases reread carefully once.

What these sources do not say: what the right answers are at your company, which cases are critical, and what threshold you accept; the project’s second week answers that, with the person on your team who knows.

The rules of a measured AI

Six rules, written in the first week. The threshold before the first trial. Real cases, not invented examples, in the three languages. The right answer comes from you; the model never prevails. Every claim has a source, checked one by one on critical cases. Every change replays the whole set: instructions, documents, model, vendor version. Every error seen in the pilot becomes a case; the set grows and never shrinks.

A seventh rule concerns the quote and the supplier: “AI tested and validated” is not a requirement, it is a sentence; the requirement reads “evaluation set of sixty real cases in the three languages, right answers validated by the client, six case types present, automated grading with a judge calibrated on a hand-graded sample, release threshold and regression rule written before trials, table of the six measures shown at every demo, set delivered in the client’s repository”, and it is checked as the specification guide recommends. To a supplier showing a demo, the question to ask is: “on how many cases, written by whom, and which ones fail?”

Six functions, and what is measured first

Six functions we build, and for each the first measure chosen.

The agent on WhatsApp, described in the dedicated guide. Unsourced claims and missed handovers; bypass attempts are in the set from the start.
Reading invoices and documents, described in the dedicated guide. Accuracy per field, by code, on sixty real documents; the total requires more than the payment method.
Search in documents. Does each answer cite the right passage; wrong refusals when the passage exists; “I don’t know” when it does not.
Product search, described in the search guide. Two hundred real queries with the expected product; the same principle without a language model.
Sorting messages. Exact category, by code; confusion between “complaint” and “question” counted separately, because it does not cost the same.
The model deployed on site, described in the private deployment guide. The set chooses the model before the machine is bought, and is replayed at every update.

The 2027 calendar

The calendar is that of an evaluation set up in eight weeks, while the function is being built.

Figure 6 · Eight weeks, and the second decides everything: a set written by us would measure what we believe your customers ask. The set does not stop at release; it grows with every error seen in the pilot, and it is replayed the day a model changes version.

Week 0 to 1 writes the criterion and the threshold: what “working” means, the minimum score, the rule on regressions, the critical cases; signed. Week 1 to 2 gathers the sixty cases and their answers: the cases come from your exchanges, the answers from your team, and all six types are present. Week 2 to 3 automates grading: code graders, judge model calibrated on your grades, sample reread by hand; by the end of the week the set runs with one command.

Weeks 3 to 6 are the versions: every change of instructions, documents or model replays everything, and the board is shown at every demo, failures included. Weeks 6 to 8 are the pilot: real users, and every error seen becomes a case; the set grows from sixty to about a hundred. After release, the set stays in your repository, and it is replayed at every change and every model version change. It is also what makes changing supplier possible: whoever takes over knows, with one command, what works.

What makes measuring an AI fail

The evaluations we take over, when there are any, almost always failed for one of these reasons.

The demo instead of the set. Five questions chosen by the seller.
Invented cases. Written in careful French, while customers write in darija.
Only easy cases. No out-of-scope; the AI never had to say “I don’t know”.
The right answer written by the supplier. Or worse, by the model itself.
An overall score. An average “faithfulness” that hides the invented claim.
The threshold written afterwards. Adjusted to the score obtained.
The set abandoned after release. The model changes version; nobody replays anything.

Frequently asked questions

The questions we receive from managers, IT departments and teams comparing suppliers.

How do we know an AI really works before putting it into production?: With an evaluation set: sixty real cases drawn from your exchanges, with the right answer written by your team, half of them traps, out-of-scope questions and bypass attempts. Every version of the AI replays the whole set, and a release threshold written before the trials says whether it goes to production. A demo of five questions proves nothing.
How are hallucinations measured?: Claim by claim: each sentence of the answer is checked against the documents the system retrieved, and those not found there are counted. Zero is required on critical cases. An overall faithfulness score is not enough: an answer that is two-thirds right contains an invented promise, and it is the one the customer remembers.
What accuracy rate can we expect?: No general figure holds for you: public leaderboards measure general tasks, often in English, not a question in darija about your catalogue. The rate that counts is measured on your sixty cases, per case type, and it is read with five other measures: unsourced claims, wrong refusals, regressions, handovers, cost and latency.
Who writes the right answers?: You: the person on your team who knows, the manager, the accountant, the branch head. It is an hour or two, once, and it is the only step we cannot do for you. A right answer written by the supplier, or by the model, measures what the supplier believes, not what your customers should hear.
What happens when the model changes version?: The set is replayed, with one command, and the board shows the regressions: cases the old version passed and the new one fails. It is the same procedure when instructions or documents change, or when moving to a less costly model. Without a set, a version change is a bet you discover through a complaint.
What should we ask a supplier who says their AI is reliable?: On how many cases it was measured, who wrote the right answers, which case types the set contains, how unsourced claims are counted, what threshold was written before the trials, and which cases still fail. A supplier who measures can show the list of failures; one who only shows a demo has none.