An AI assistant is not finished at launch: measuring quality from real conversations

A demo can look flawless. The questions are prepared, the knowledge base is fresh and the answers sound convincing. After launch, real people arrive: they write incomplete sentences, use internal names, change the subject halfway through and ask about situations nobody considered during the pilot. That is when serious quality work begins.

A good AI assistant is therefore not a one-off project. It is a service that must be monitored, tested and improved using evidence. The goal is not to answer at any cost. It is to help reliably, acknowledge uncertainty and hand the conversation to a person at the right moment.

AI assistant quality is not defined by its best demo. It is defined by how it handles ordinary, borderline and uncomfortable questions in real operation.

Quality is not a single score

Conversation volume or a thumbs-up rate is not enough. High satisfaction can hide a factual mistake, while a short answer can be more useful than a long explanation. Quality should combine several signals and interpret them in the context of the assistant's actual job.

1. Useful resolution

The most important question is whether the visitor reached the right next step. In support, that may be an exact procedure; in e-commerce, a suitable product; for a municipality, the correct form or contact. Do not measure only whether the assistant replied. Verify whether the answer moved the user toward the intended outcome.

2. Grounding in trusted sources

An answer should come from current, approved material. Check whether a citation opens the right page, whether the stated fact exists in the source and whether the assistant confused similar products, services or rules. High-impact topics also require human review.

3. Knowledge gaps

An unanswered question is not merely a failure. It is a precise signal that something in the knowledge base is missing, unclear or outdated. Group similar questions and prioritise those that recur or carry significant impact. One careful source improvement can improve dozens of future conversations.

4. Safe behaviour and human handoff

The assistant must know its boundaries. Watch for unnecessary requests for personal data, promises it cannot guarantee and sensitive or ambiguous requests without a safe next step. A correct handoff to an operator is a successful outcome, not a defeat.

5. Stability after change

A new document, instruction or model can fix one answer and break another. Turn both successful and problematic conversations into regression scenarios. Run them again before release and verify that the target case improved without degrading the rest.

6. Speed and availability

Even an accurate answer loses value if it arrives too late or the widget fails to load. Measure response time, technical errors and the share of completed requests. Interpret these figures together with content quality rather than in isolation.

7. Cost per useful outcome

The cheapest answer is not a win if it does not solve the problem. Relate cost to usefully resolved questions, time saved for the team and conversions. This makes it possible to compare shorter context, more precise retrieval or another model without cutting quality simply to reduce spend.

From conversation to safe improvement

1. Capture the signal. Mark negative feedback, a human handoff, a missing source or a suspicious answer.
2. Find the cause. Separate missing content from failed retrieval, an unclear instruction or the assistant exceeding its role.
3. Fix the smallest correct thing. Adding one authoritative page or clarifying one rule is often enough. The whole system rarely needs rewriting.
4. Add a test. Save the original question and expected behaviour as a regression scenario.
5. Verify before publishing. Compare the change with the current version, have risky answers reviewed and keep a rollback path.

A small realistic test set beats hundreds of artificial questions

Start with dozens of scenarios that represent real operation. Include common requests, incomplete phrases, typos, synonyms, follow-up questions and cases where acknowledging uncertainty is the correct answer. Add edge and adversarial inputs that try to bypass rules or extract non-public information.

Every scenario needs a clear purpose and an observable expectation. An automated link or fact check may be sufficient in some cases. Others need human judgment of usefulness, tone and safety. Set thresholds according to risk: opening hours and a complaint or healthcare service should not be held to identical standards.

When not to publish a change

Stop a release if the average improved but a critical group of questions became worse, unsupported claims increased or human handoff stopped working. Be equally cautious when changing sources without a known owner and update date. With AI, rollback is a practical safety mechanism, not a sign of failure.

How Informio helps

Informio brings knowledge sources, conversation history, feedback, unanswered questions and usage auditing together. In Quality Lab, real situations can become scenarios, behaviour can be compared and a change can be checked for the expected improvement before deployment. Sensitive decisions remain under human control.

The best starting point is not a huge dashboard. Choose ten of the most frequent or highest-risk questions, define the correct behaviour and track them after every change. Once this becomes an operating habit, the AI assistant grows from an impressive demo into a dependable service.

Explore Informio Quality Lab and build your first set of realistic scenarios.

Sources

• NIST AI Risk Management Framework – Core
• NIST Generative AI Profile
• OpenAI – Evaluation best practices