The demonstration is magnificent. A sentence becomes a spreadsheet. A spreadsheet becomes a presentation. The presentation becomes a confident explanation of why your company should apparently purchase more presentations.

Nobody interrupts. The sample data is tidy, the internet connection behaves, and every customer has the good manners to want exactly what the demonstration knows how to provide.

Then Monday arrives.

A customer uses the wrong product name. Someone uploads last year’s policy. Two departments disagree about which spreadsheet is authoritative. The system produces an answer so polished that the newest employee assumes it must be correct. There is no dramatic explosion. There is a normal-looking email containing an expensive mistake.

This gap between a demonstration and ordinary work is where an AI project becomes interesting. The demo establishes that something can happen. Deployment asks whether it should happen, under which conditions, and who will notice when it happens badly.

A clean example can hide a dirty problem

Imagine a team testing an assistant that drafts replies to customer questions. On ten carefully selected questions, the drafts look excellent. Everyone celebrates the time saved.

But the selection has already done part of the assistant’s job. The difficult questions, incomplete messages, contradictory requests, and unusual situations have been quietly left outside the room. The team has measured performance on the problems it was comfortable showing.

A better evaluation includes ordinary cases, inconvenient cases, and cases in which the correct response is to ask a question or decline to guess. The test set should resemble the work, including the work people dislike.

Keep a few examples separate from the material used to adjust the system. Otherwise, you can spend a week improving your score on the same small collection and discover that you have trained a very sophisticated student to memorize yesterday’s quiz.

The point is not to make the tool fail. It is to locate the boundaries before a customer finds them for you.

Decide what “good” means before looking at the answer

Fluent writing is immediately visible. Correctness, appropriate escalation, and careful handling of private information are harder to judge. This makes style an unusually persuasive distraction.

Before running the test, write down the requirements. Does the response use the current policy? Does it distinguish a fact from an assumption? Does it avoid making commitments the organization cannot honor? Can the recipient tell what to do next?

Some errors are annoying; others are unacceptable. A slightly stiff greeting and an invented refund entitlement should not receive the same penalty merely because both occupy one row in a review sheet.

A useful evaluation separates these dimensions. It also compares the assistant with the existing process. If humans already make a particular mistake, that does not excuse the system. It helps you understand whether the new arrangement reduces the problem, moves it, or makes it happen faster.

NIST’s AI Risk Management Framework treats evaluation and risk management as continuing activities across a system’s use. That is a more useful posture than treating launch day as graduation from responsibility.

The review step needs an actual job

“Someone will check it” sounds reassuring until you meet Someone at 4:47 on Friday.

That person is tired, has twelve other tasks, and may not know the subject well enough to challenge a convincing answer. Adding a human to a diagram does not automatically add meaningful oversight.

Specify what the reviewer checks and what information is available. If a draft quotes a rule, show the rule and its date. If the source is uncertain, display the uncertainty. If a request falls outside the approved scope, send it to a person with authority rather than offering a decorative warning nobody reads.

Give reviewers enough time to do the work. If an assistant generates fifty items in the time previously available for five, review may become a new bottleneck. The time saving has changed departments rather than disappeared.

For a small initial rollout, keep consequential actions separate from draft generation. An assistant can suggest a message without being allowed to send it. It can identify a possible discrepancy without changing the customer’s account.

Keep a route back to ordinary work

A sensible pilot has an owner, an end date, a limited task, and a way to stop. It also has a description of what happens when the service is unavailable.

Suppose the assistant helps prepare internal summaries. The team might test it on a defined group of documents for two weeks, record corrections, and compare the total time spent with the previous method. The review should count setup and correction time, not only the delightful seconds during which text appears.

At the end, the team can continue, narrow the task, change the workflow, or abandon the experiment. None of those outcomes needs to become an identity crisis. You are evaluating a tool, not choosing a religion.

The best result may be a modest one: excellent performance on a boring, bounded task, with clear limits everyone understands. That is less cinematic than a machine that transforms the entire business before lunch. It is also considerably easier to use on Monday.

Sources & further reading