Production-Grade Evaluation Strategies for LLMs, RAG, Agents, and Multi-Agent Systems

Evaluating a conventional machine-learning model is often straightforward. A classification model can be measured using accuracy, precision, recall, F1 score, or area under the curve. A regression model can be measured using mean absolute error or root mean squared error.

Large language model applications are more difficult to evaluate because they are:

• Non-deterministic
• Generative rather than strictly predictive
• Capable of producing several valid answers
• Often composed of multiple models, tools, retrievers, databases, prompts, and agents
• Able to modify external environments
• Sensitive to prompt wording, model versions, context ordering, retrieved documents, and tool responses
• Expected to satisfy qualitative requirements such as helpfulness, clarity, faithfulness, tone, safety, and policy compliance

A production-grade evaluation strategy therefore cannot rely on one benchmark or one quality score. It must evaluate the system at several levels:

1. The underlying model
2. Individual LLM responses
3. Retrieval and generation components
4. Tool calls and agent trajectories
5. Multi-agent coordination
6. End-to-end business outcomes
7. Operational performance
8. Safety, compliance, and governance
9. Real production behaviour

Hugging Face Evaluate provides useful building blocks for metrics, comparisons, and measurements. However, Hugging Face currently presents LightEval as the more actively maintained toolkit for modern LLM benchmarking, while agentic applications require additional trace, tool, environment, and outcome evaluation capabilities.

1. The Central Principle: Evaluate the System, Not Only the Model​

An enterprise AI application is rarely just an LLM.

A typical application may contain:

A multi-agent system may be more complex:

Even when the final answer is wrong, the underlying LLM may not be the source of failure.

The actual failure could be:

• The router selected the wrong workflow.
• The retriever returned irrelevant documents.
• The correct document was retrieved but truncated.
• The tool call used an incorrect parameter.
• The agent stopped too early.
• Two agents produced conflicting conclusions.
• The final synthesiser ignored a compliance warning.
• The answer was correct, but the claimed external action was never performed.
• A model upgrade changed output structure.
• The system exceeded its latency or cost budget.

The evaluation framework must therefore separate component quality from end-to-end outcome quality.

2. A Layered Evaluation Model​

A practical evaluation hierarchy contains six layers.

Layer 1: Model-level evaluation​

This asks:

Is the underlying model capable of performing the general task?

Examples include:

• Reasoning benchmarks
• Coding benchmarks
• Question-answering benchmarks
• Instruction-following tests
• Tool-use benchmarks
• Multilingual benchmarks
• Long-context tests
• Safety benchmarks

Examples of agent-oriented benchmarks include AgentBench, GAIA and WebArena. AgentBench evaluates models in interactive environments, GAIA tests real-world questions requiring reasoning and tool use, and WebArena evaluates agents operating realistic websites with programmatic task validators.

These benchmarks help with initial model selection. They do not prove that a model will perform well in a specific production application.

Layer 2: Response-level evaluation​

This assesses an individual LLM output.

Typical criteria include:

• Correctness
• Relevance
• Completeness
• Groundedness
• Clarity
• Conciseness
• Tone
• Format compliance
• Instruction following
• Citation accuracy
• Safety

Layer 3: Component-level evaluation​

This evaluates application components independently.

Examples:

• Intent classification accuracy
• Retrieval recall
• Reranking quality
• Query-rewriting quality
• Tool-selection accuracy
• Tool-argument validity
• Memory retrieval precision
• Output-schema compliance
• Guardrail detection performance

Component-level evaluation is essential for diagnosis. An end-to-end score may tell you that performance fell, but component metrics tell you why.

Layer 4: Trajectory-level evaluation​

This evaluates the sequence of actions taken by an agent.

A trajectory may contain:

• Reasoning steps
• Tool calls
• Observations
• Agent handoffs
• Retries
• Reflections
• Memory reads and writes
• User clarification requests
• Final response

OpenAI’s current agent-evaluation guidance describes traces as end-to-end records of model calls, tool calls, guardrails and handoffs, while Anthropic distinguishes the transcript or trajectory from the final environment outcome.

Layer 5: Outcome-level evaluation​

This asks:

Did the system actually accomplish the task?

For example, a travel-booking agent may say:

Your reservation has been confirmed.

Response-level evaluation may find the statement fluent and appropriate. Outcome-level evaluation must check whether the booking exists in the reservation database.

For action-taking agents, external state is generally a stronger source of truth than the final natural-language answer.

Layer 6: Production-level evaluation​

This evaluates behaviour under real operational conditions:

• Live traffic
• New user intents
• Distribution shift
• Model-provider changes
• Knowledge-base changes
• Tool failures
• Latency spikes
• Rate limits
• User feedback
• Security attacks
• Escalation patterns
• Cost drift

Offline and online evaluation should form a continuous feedback loop: offline evaluation validates changes before deployment, while online evaluation identifies production failures that should be added to future regression datasets.

3. The Main Types of Evaluators​

No evaluator works for every requirement. A robust system combines deterministic, model-based and human evaluation.

3.1 Deterministic or code-based evaluation​

These evaluators use code, rules or formal validators.

Examples:

• Exact match
• Regular expressions
• JSON-schema validation
• Type validation
• Unit tests
• SQL state checks
• API response validation
• Tool-name verification
• Tool-parameter verification
• Citation URL validation
• Access-control checks
• Required-keyword checks
• Forbidden-action checks
• Latency thresholds
• Token-count limits
• Security scanners

Advantages​

• Fast
• Cheap
• Reproducible
• Objective
• Easy to run in CI
• Easy to diagnose

Limitations​

• Can be brittle
• May penalise valid alternative answers
• Cannot reliably assess nuanced qualities such as empathy or persuasiveness
• Requires clearly formalised expectations

Anthropic recommends combining code-based, model-based and human graders. Code-based graders are especially suitable for outcome verification, tool-call checks, static analysis and transcript statistics.

Best practice​

Use deterministic graders whenever the requirement can be expressed deterministically.

Do not ask an LLM judge whether a JSON document is valid. Use a JSON parser.

Do not ask an LLM judge whether a payment was processed. Check the payment system.

Do not ask an LLM judge whether a required tool was called. Inspect the trace.

3.2 Traditional NLP and machine-learning metrics​

Hugging Face Evaluate groups evaluation capabilities into metrics, comparisons and measurements. It also distinguishes generic, task-specific and dataset-specific metrics.

Common metrics include:

These metrics remain useful, but they should rarely be the only evaluation method for open-ended enterprise responses.

3.3 LLM-as-a-judge​

An LLM judge receives an output and evaluates it against a rubric, reference, context or competing output.

Common judging methods​

Pointwise grading​

The judge scores one response independently.

Example:

Score the answer from 1 to 5 for factual correctness.

Binary grading​

Does the response fully comply with the refund policy?
Return PASS or FAIL.

Pairwise grading​

The judge compares candidate A with candidate B.

Which response better satisfies the user request?

Pairwise evaluation is often useful because selecting the better of two answers can be easier than assigning an absolute score.

Reference-based grading​

The judge compares the answer with an expected answer or factual reference.

Context-grounded grading​

The judge determines whether statements are supported by retrieved documents.

Rubric-based grading​

The response is scored against explicit requirements.

Example:

Evaluate the response using these criteria:

1. Correctly identifies the relevant policy.
2. Does not invent eligibility requirements.
3. Explains the next action.
4. Uses clear, professional language.
5. Escalates when policy information is insufficient.

Multi-judge evaluation​

Multiple judges independently evaluate the response, followed by:

• Majority voting
• Averaging
• Weighted consensus
• Escalation when judges disagree

3.4 Risks of LLM judges​

LLM judges can exhibit:

• Position bias
• Verbosity bias
• Self-enhancement or model-family bias
• Preference for confident writing
• Sensitivity to rubric wording
• Inconsistency across repeated calls
• Weakness on specialist domains
• Susceptibility to instructions embedded in the candidate answer

Research comparing LLM judges with human judgments has identified position, verbosity and self-enhancement biases. The same work found that careful judge design can improve agreement with human evaluators, but judge scores should not be treated as inherently objective.

Judge-bias controls​

1. Hide model names from the judge.
2. Randomise A/B answer order.
3. Run pairwise judging twice with reversed positions.
4. Use a detailed rubric.
5. Ask for evidence supporting the score.
6. Separate correctness, relevance and style into different judgements.
7. Control for response length.
8. Use a different model family from the candidate when possible.
9. Calibrate against domain experts.
10. Include an “uncertain” or “requires human review” outcome.
11. Version the judge prompt and judge model.
12. Treat candidate text as untrusted data, not judge instructions.

3.5 Human evaluation​

Human evaluation remains necessary for:

• Subjective quality
• High-risk decisions
• Domain-specific correctness
• Complex policy interpretation
• Judge calibration
• Red-team testing
• Tone and brand alignment
• Ambiguous edge cases
• Legal, medical or financial content
• Evaluating whether automated metrics reflect genuine user value

Human review methods include:

• Single-response scoring
• Pairwise preference
• Domain-expert review
• Blind A/B testing
• Error classification
• User-satisfaction assessment
• Adversarial review
• Inter-annotator agreement measurement

Human evaluation should not merely produce a score. It should produce structured labels explaining the error.

Useful labels include:

Incorrect fact
Unsupported claim
Missing information
Wrong tool
Wrong tool argument
Unsafe action
Policy violation
Poor escalation
Unnecessary escalation
Retrieval failure
Reasoning failure
Formatting failure
Tone failure

4. Evaluation Dataset Design​

The quality of an evaluation is constrained by the quality of its dataset.

A production evaluation dataset should include several test-set categories.

4.1 Golden regression set​

A stable, version-controlled collection of important cases.

It should include:

• Common user requests
• Critical workflows
• Previously fixed failures
• Contractual requirements
• Compliance requirements
• High-value business scenarios

This set should achieve a very high pass rate.

4.2 Capability set​

A difficult set used to measure improvement.

These tests intentionally include tasks the current system cannot consistently solve. As performance improves, successful capability cases can become regression tests. Anthropic explicitly distinguishes capability evaluations from regression evaluations in this way.

4.3 Edge-case set​

Examples:

• Ambiguous requests
• Missing identifiers
• Conflicting documents
• Very long contexts
• Misspelled entities
• Unsupported languages
• Tool timeouts
• Empty retrieval results
• Duplicate documents
• Invalid dates
• Unusual account states

4.4 Adversarial and safety set​

Examples:

• Prompt injection
• Indirect prompt injection in documents
• Attempts to bypass policy
• Sensitive-data requests
• Tool-argument manipulation
• Cross-user data access
• Malicious file content
• Unsafe transaction requests
• Attempts to force unauthorised agent handoffs

4.5 Production replay set​

Historical production traces should be converted into evaluation cases.

Prioritise:

• Negative user feedback
• Escalated conversations
• Abnormally long traces
• High-cost interactions
• Failed tools
• Repeated queries
• Safety alerts
• Abandoned sessions
• Cases requiring manual correction

4.6 Synthetic set​

Synthetic data can increase coverage, particularly for rare scenarios.

However, synthetic data should be anchored in real examples and reviewed for:

• Unrealistic phrasing
• Repeated patterns
• Missing business complexity
• Incorrect labels
• Leakage from the generating model
• Over-representation of simple cases

LangSmith recommends starting with manually curated examples, then expanding through historical traces and synthetic data.

4.7 Hidden holdout set​

Teams often overfit prompts and workflows to visible evaluation cases.

Maintain a hidden set that:

• Is not visible during prompt development
• Is evaluated less frequently
• Represents production traffic
• Contains fresh failure patterns
• Is controlled by a separate evaluation owner where practical

5. Recommended Evaluation Record​

Each test case should contain more than a prompt and expected answer.

case_id: refund_critical_004
task_type: refund_processing
risk_level: high

input:
  user_message: >
    I was charged twice. Refund the duplicate payment.

initial_state:
  customer_verified: false
  duplicate_payment_exists: true
  duplicate_amount: 79.00
  currency: USD

available_tools:
  - verify_customer
  - fetch_transactions
  - create_refund
  - send_confirmation

expected_outcome:
  refund_status: processed
  refund_amount: 79.00

required_actions:
  - verify_customer
  - fetch_transactions
  - create_refund
  - send_confirmation

forbidden_actions:
  - refund_more_than_duplicate_amount
  - expose_full_card_number

response_requirements:
  - explain_refund_timeline
  - provide_confirmation_reference
  - do_not_claim_completion_before_tool_success

evaluation_slices:
  - financial_action
  - tool_use
  - identity_verification
  - high_risk

source:
  type: production_failure
  date_added: 2026-07-20

This format supports response, trace and outcome evaluation.

6. RAG Evaluation​

Retrieval-augmented generation must be evaluated as at least three systems:

1. Retrieval
2. Generation
3. End-to-end RAG behaviour

RAGAS evaluates dimensions such as context relevance, faithfulness and answer relevance. RAGChecker extends this idea with claim-level diagnostic metrics for retriever and generator behaviour, while ARES uses synthetic examples and lightweight trained judges calibrated with a relatively small human-labelled set.

6.1 Retrieval evaluation​

Precision@k​

The proportion of retrieved documents that are relevant.

Precision@k =
relevant documents in top k
───────────────────────────
k

High precision means little irrelevant context is passed to the generator.

Recall@k​

The proportion of all required relevant documents retrieved in the top k.

Recall@k =
relevant documents retrieved in top k
─────────────────────────────────────
all relevant documents

High recall means the retriever is less likely to miss required evidence.

Hit rate​

Whether at least one relevant document appears in the retrieved set.

Mean reciprocal rank​

Measures how highly the first relevant result is ranked.

MRR = average(1 / rank of first relevant result)

nDCG​

Normalised discounted cumulative gain evaluates ranked results where some documents are more relevant than others.

Context precision​

Measures whether relevant chunks appear before irrelevant chunks.

Context recall​

Measures whether the retrieved context contains the information needed to answer the question.

Retrieval robustness​

Test retrieval under:

• Paraphrases
• Spelling mistakes
• Acronyms
• Entity aliases
• Multilingual queries
• Short queries
• Overly detailed queries
• Conflicting terminology
• Metadata filters
• Date filters

Retrieval safety​

Evaluate:

• Tenant isolation
• Document-level permissions
• Attribute-based access control
• Cross-user leakage
• Restricted-document retrieval
• Deleted-document persistence
• Cache leakage
• Prompt injection embedded in retrieved documents

Retrieval freshness​

Test whether:

• Updated documents are preferred
• Expired policies are excluded
• Effective dates are respected
• Superseded versions are correctly handled
• Indexing delay stays within its service objective

Retrieval efficiency​

Measure:

• Search latency
• Reranking latency
• Number of vector comparisons
• Embedding cost
• Cache-hit rate
• Retrieved-token count
• Duplicate-chunk rate
• Context compression ratio

6.2 Query-processing evaluation​

Before retrieval, many systems perform query rewriting, decomposition or expansion.

Evaluate whether rewriting:

• Preserves the user’s original intent
• Adds useful synonyms
• Correctly resolves pronouns
• Maintains required entities
• Does not invent filters
• Does not remove temporal constraints
• Produces the correct number of subqueries
• Avoids unnecessary retrieval calls

For a query such as:

What did it say about cancellation, and does that apply to my current plan?

The rewriter may need conversation history to resolve “it” and “my current plan.” A standalone linguistic similarity score would not adequately test this.

6.3 Generation evaluation​

Faithfulness or groundedness​

Every factual claim should be supported by retrieved evidence.

A strong implementation performs claim-level evaluation:

Claim-level evaluation is generally more diagnostic than assigning one score to the full answer.

Answer relevance​

Does the response directly answer the question?

A response can be fully grounded but still irrelevant.

Correctness​

Does the answer agree with known ground truth?

Completeness​

Does the answer include all required facts?

Citation correctness​

For each citation:

• Does the source support the claim?
• Does the cited passage contain the evidence?
• Is the document authoritative?
• Is the source current?
• Is the link valid?
• Does the source belong to the permitted corpus?

Citation completeness​

What proportion of externally verifiable claims are cited?

Context utilisation​

Did the model use the most relevant context, or did it ignore it and rely on parametric knowledge?

Conflict handling​

Test cases should contain:

• Two documents with different dates
• A policy and an exception
• An authoritative source and an informal source
• Conflicting regional policies
• A current document and an expired document

The system should identify and resolve the conflict rather than silently choosing one.

Abstention quality​

Evaluate whether the system refuses or asks for clarification when:

• No evidence is retrieved
• Evidence is incomplete
• Documents conflict
• The requested information is outside the knowledge base
• User identity or jurisdiction is missing

A useful RAG system is not one that answers every question. It is one that knows when the available evidence is insufficient.

6.4 End-to-end RAG metrics​

A production dashboard might include:

Do not collapse these automatically into a single average. A 95% relevance score cannot compensate for a serious access-control failure.

7. Single-Agent Evaluation​

An agent differs from a normal response generator because it repeatedly chooses actions based on changing environmental state.

Agent evaluation should therefore cover reasoning, actions, trajectory, outcome and resource use.

7.1 Task-success metrics​

Binary success rate​

Successful tasks / attempted tasks

Partial completion​

Useful for long-horizon tasks with milestones.

Example:

1. Identified customer
2. Found relevant transaction
3. Verified duplicate
4. Created refund
5. Sent confirmation

A failed task may still achieve three of five milestones. Tracking milestone completion helps diagnose where the agent failed.

Pass@k​

For non-deterministic agents, run multiple trials.

Pass@k = probability that at least one of k trials succeeds

Also report pass@1 because production users may not receive multiple attempts.

Anthropic recommends multiple trials for stochastic agent tasks because one attempt may not represent reliable performance.

7.2 Tool-selection evaluation​

Measure:

• Correct tool selected
• Incorrect tool selected
• Required tool omitted
• Unnecessary tool called
• Prohibited tool called
• Tool called in the wrong order
• Tool called without required confirmation

Possible metrics:

Tool precision =
correct tool calls / all tool calls

Tool recall =
required tool calls completed / required tool calls

However, tools may not always have one required sequence. Outcome validators should take priority when several valid trajectories exist.

7.3 Tool-argument evaluation​

A correct tool with incorrect arguments can be more dangerous than selecting no tool.

Evaluate:

• Required parameters
• Parameter types
• Entity identifiers
• Currency
• Amount
• Date and timezone
• Access token scope
• Confirmation state
• Idempotency key
• User-selected options
• Maximum permitted values

Use deterministic schema and business-rule validators.

7.4 Planning evaluation​

Assess whether the plan:

• Covers the necessary subgoals
• Orders dependencies correctly
• Assigns appropriate tools or agents
• Includes verification steps
• Adapts after failures
• Avoids irrelevant work
• Recognises missing information
• Defines a termination condition

Planning quality should not be evaluated only by comparing the generated plan to one reference plan. Multiple valid plans may exist.

Evaluate plan invariants instead:

Identity must be verified before account access.
Payment status must be checked before refund creation.
User confirmation is required before irreversible action.

7.5 Trajectory evaluation​

Useful trajectory metrics include:

• Number of turns
• Number of tool calls
• Repeated tool-call rate
• Failed tool-call rate
• Recovery rate
• Average retries per tool
• Tokens per successful task
• Time per successful task
• Unnecessary-action rate
• Loop rate
• Premature-termination rate
• Handoff count
• Unsupported final-claim rate

Step efficiency​

Step efficiency =
minimum reasonable steps
────────────────────────
actual steps

This should be treated as diagnostic, not an absolute requirement. A longer but safer path may be preferable.

7.6 Error recovery​

Agents should be evaluated under controlled failures:

• Tool timeout
• HTTP 500
• Rate limit
• Invalid response
• Partial response
• Authentication expiry
• Empty search result
• Conflicting result
• Stale state
• User correction
• Agent handoff failure

Measure whether the agent:

• Retries appropriately
• Changes strategy
• Uses an alternative tool
• Preserves completed work
• Avoids duplicate actions
• Escalates when recovery is impossible
• Communicates uncertainty honestly

7.7 Termination evaluation​

Agents may:

• Stop too early
• Continue after task completion
• Enter loops
• Reach the iteration limit
• Claim success without verification
• Fail to return a user-facing response

Evaluate:

Correct termination rate
Premature termination rate
Loop rate
Maximum-step breach rate
False success claim rate

7.8 Memory evaluation​

Agents with memory require a separate memory test suite.

Memory write precision​

Did the system store only useful, permitted information?

Memory write recall​

Did it store important information required for later tasks?

Memory retrieval precision​

Was the retrieved memory relevant to the current request?

Memory retrieval recall​

Did the system retrieve all required previous information?

Temporal correctness​

Did it prefer current information over outdated information?

User isolation​

Did one user’s memory appear in another user’s interaction?

Deletion compliance​

Was deleted memory actually removed from:

• Primary storage
• Vector indexes
• Caches
• Derived summaries
• Replicas

Contradiction handling​

Can the system update an old fact without keeping both old and new facts as equally valid?

8. Multi-Turn Conversation Evaluation​

A conversational system cannot be evaluated only one message at a time.

Thread-level evaluation should assess:

• Context retention
• Coherence
• Goal progression
• Contradiction across turns
• Clarification quality
• Repetition
• User-frustration handling
• Escalation timing
• Resolution
• Topic maintenance
• Personalisation accuracy
• Memory safety

LangSmith supports thread-level evaluators specifically for properties such as coherence across turns, topic maintenance and user satisfaction.

Useful conversation metrics include:

Conversation resolution rate
Average turns to resolution
Repeated-question rate
Unnecessary clarification rate
Missed clarification rate
Escalation precision
Escalation recall
User correction rate
Contradiction rate

9. Multi-Agent Architectures and Their Evaluation Strategies​

Multi-agent evaluation must consider more than individual-agent quality. It must evaluate coordination, communication, delegation, redundancy and collective outcome.

Research taxonomies describe multi-agent systems through communication structures, coordination mechanisms and collaborative strategies. MultiAgentBench evaluates topologies including star, chain, tree and graph structures, while measuring milestone progress, communication and planning quality.

9.1 Sequential pipeline​

Example:

Advantages​

• Simple
• Auditable
• Easy to assign roles
• Predictable handoffs

Risks​

• Early errors propagate
• Later agents may trust incorrect upstream outputs
• Latency accumulates
• Information may be lost at each handoff

Evaluation focus​

• Handoff completeness
• Information preservation
• Error-propagation rate
• Stage-level correctness
• End-to-end latency
• Ability of downstream agents to detect upstream errors

Test the pipeline by deliberately injecting incorrect output at each stage.

9.2 Router or dispatcher architecture​

Evaluation focus​

• Routing accuracy
• Out-of-scope detection
• Multi-intent handling
• Confidence calibration
• Fallback routing
• Unnecessary routing
• Route-specific task success

Create a routing confusion matrix:

Evaluate mixed-intent requests separately.

9.3 Supervisor-worker or star architecture​

The supervisor assigns tasks and synthesises results.

Evaluation focus​

• Delegation accuracy
• Task decomposition
• Worker selection
• Workload balance
• Result integration
• Supervisor bottlenecks
• Missed worker warnings
• Incorrect overrides
• Communication-token overhead

Important ablation​

Compare:

Supervisor + workers
vs.
Supervisor alone
vs.
Best worker alone

A multi-agent system should justify its additional cost and complexity.

9.4 Hierarchical or tree architecture​

Evaluation focus​

• Goal preservation across hierarchy
• Delegation depth
• Information loss
• Escalation correctness
• Decision consistency
• Duplicate work
• Latency by depth
• Failure containment

A critical metric is whether the root objective changes as it passes through layers.

9.5 Parallel ensemble​

Agents independently solve the same task.

Evaluation focus​

• Diversity of solutions
• Correlated-error rate
• Majority-vote accuracy
• Aggregator accuracy
• Cost-quality trade-off
• Disagreement rate
• Calibration of consensus

If all agents use the same model, prompt and evidence, the apparent ensemble may contain little real diversity.

Test heterogeneous ensembles:

• Different model families
• Different prompts
• Different retrieval strategies
• Different role perspectives
• Different temperatures

9.6 Debate or critic architecture​

Or:

Evaluation focus​

• Error-correction rate
• Valid-claim preservation
• False criticism rate
• Convergence
• Groupthink
• Argument quality
• Judge bias
• Number of debate rounds
• Cost per corrected error

Measure whether debate improves the result over the initial answer.

Correction gain =
final correctness - initial correctness

Also track degradation rate: cases where a correct initial answer becomes incorrect after criticism.

9.7 Planner-executor-critic​

Evaluation focus​

• Plan quality
• Execution adherence
• Critic precision
• Replanning quality
• Loop control
• Critic-induced regressions
• Final outcome

The critic should not automatically trigger revision. It should provide calibrated confidence and evidence.

9.8 Peer-to-peer graph architecture​

Agents communicate without one permanent supervisor.

Evaluation focus​

• Message delivery
• Consensus formation
• Deadlock rate
• Conflicting-state rate
• Communication efficiency
• Emergent leadership
• Fault tolerance
• Information diffusion
• Duplicate-message rate

A graph architecture may provide flexibility but can create high communication overhead.

A useful metric is:

Communication overhead =
inter-agent tokens
──────────────────
total system tokens

9.9 Blackboard or shared-memory architecture​

Agents read from and write to a common state.

Evaluation focus​

• Write correctness
• Stale-state usage
• Conflicting writes
• Version control
• Source attribution
• Race conditions
• State corruption
• Access control
• Memory growth

Test concurrent writes and delayed reads.

9.10 Competitive or market-based architecture​

Agents bid, vote or compete to solve tasks.

Evaluation focus​

• Selection quality
• Strategic manipulation
• Collusion
• Incentive alignment
• Resource allocation
• Fairness
• Winner reliability
• Cost of competition

The highest-confidence agent should not automatically win unless confidence is well calibrated.

10. Core Multi-Agent Metrics​

Collective task success​

Did the system accomplish the overall objective?

Individual contribution​

Estimate each agent’s marginal value through ablation:

Contribution of agent i =
success with all agents
-
success without agent i

Role adherence​

Did agents stay within assigned responsibilities?

Handoff accuracy​

Did the correct information reach the correct agent at the correct time?

Communication precision​

What proportion of inter-agent messages contained necessary information?

Communication redundancy​

How much information was repeatedly transmitted without adding value?

Coordination efficiency​

Coordination efficiency =
task progress
─────────────
communication cost

Conflict-resolution quality​

Did the system resolve disagreements using evidence rather than authority, verbosity or repetition?

Load balance​

Was work distributed appropriately?

Fault tolerance​

Can the system complete the task if one agent:

• Times out
• Returns malformed output
• Produces incorrect information
• Becomes unavailable
• Repeats itself

Collective calibration​

Does stronger agent consensus correspond to higher correctness?

Multi-agent uplift​

Multi-agent uplift =
multi-agent task success
-
best single-agent task success

This is one of the most important metrics. If uplift is negligible or negative, a multi-agent architecture may not be justified.

11. Open-Source, Open-Weight and Closed LLM Evaluation​

The terms should be separated carefully.

Open-source model​

The weights, code and licence permit substantial inspection and modification.

Open-weight model​

Weights are available, but training data, training code or full development process may not be available.

Closed or proprietary model​

The model is accessed through an API or managed service, and its weights and internal implementation are not available.

11.1 Common black-box evaluation​

Open and closed models should first be compared using the same application-level tests:

• Same inputs
• Same tools
• Same retrieval results
• Same generation limits
• Same output schema
• Same business rubrics
• Same safety tests
• Same latency measurement point

This evaluates the user-visible system rather than internal architecture.

11.2 Additional evaluation available for open models​

Open models allow deeper measurement of:

• Token probabilities
• Log probabilities
• Entropy
• Calibration
• Attention or activation analysis
• Quantisation effects
• Fine-tuning effects
• GPU memory
• Energy consumption
• Batch throughput
• Token throughput
• Different inference engines
• Different decoding implementations
• Reproducibility with fixed weights and runtime

LightEval supports multiple local and hosted inference backends and provides sample-level results for debugging.

Open-model deployment variants to test​

Do not evaluate only the original full-precision model.

Test the actual production configuration:

• FP16/BF16
• INT8
• INT4
• GPTQ
• AWQ
• Different context lengths
• Different tensor-parallel settings
• Different inference engines
• LoRA adapters
• Merged adapters
• Speculative decoding
• Prefix or KV caching

Quantisation can change:

• Instruction following
• Structured output
• Tool calling
• Long-context behaviour
• Rare-token generation
• Multilingual performance

11.3 Closed-model evaluation considerations​

For proprietary APIs, record:

• Provider
• Exact model identifier
• Model snapshot where available
• API version
• Region
• Request parameters
• System prompt
• Tool definitions
• Temperature
• Seed where supported
• Token limits
• Safety configuration
• Response metadata
• Rate-limit behaviour

Because internal probabilities may be unavailable or limited, evaluation often relies more heavily on black-box outputs, repeated trials and external judges. The RAGAS paper explicitly notes that probability-based evaluation is not available for some closed models.

Provider drift tests​

Continuously run a small sentinel suite to detect:

• Response-format changes
• Refusal changes
• Tool-calling changes
• Latency changes
• Cost changes
• Safety-policy changes
• Regression after model alias updates

11.4 Fair open-versus-closed comparison​

Control the following:

• Context-window utilisation
• Prompt template
• Chat formatting
• Tool schema
• Stop tokens
• Sampling parameters
• Number of retries
• Hardware and concurrency
• Reasoning-token configuration
• Output length
• Safety filters
• Retrieval context

Compare using a Pareto frontier rather than only quality:

Quality
Latency
Cost
Privacy
Operational control
Deployment complexity
Energy
Scalability
Data residency

A model with the highest quality score may not be the best production choice if it violates latency, privacy or cost constraints.

12. Safety and Governance Evaluation​

Safety testing should cover both what the model says and what the agent does.

12.1 Content safety​

Evaluate:

• Harmful instructions
• Harassment
• Discrimination
• Sexual content
• Self-harm content
• Illegal guidance
• Regulated advice
• Age-appropriate responses

12.2 Prompt-injection resistance​

Test:

• Direct user injection
• Indirect injection in retrieved documents
• Injection in web pages
• Injection in tool responses
• Injection in email or file content
• Instructions asking the agent to reveal its system prompt
• Requests to ignore authorisation checks

12.3 Data protection​

Evaluate:

• PII leakage
• Secret leakage
• Credential leakage
• Cross-user data leakage
• Excessive data retrieval
• Sensitive data in logs
• Sensitive data in embeddings
• Data-retention compliance

12.4 Tool safety​

Test whether the agent:

• Requests confirmation before irreversible actions
• Uses least-privilege credentials
• Respects transaction limits
• Avoids unauthorised tools
• Validates user identity
• Prevents duplicated actions
• Handles idempotency
• Distinguishes read and write tools
• Rejects malicious tool outputs

12.5 Policy adherence​

Turn policies into executable assertions.

Example:

A refund above $100 requires human approval.

Evaluation should verify:

1. The agent identifies the amount.
2. The agent does not call the refund tool directly.
3. The agent creates an approval request.
4. The final response does not falsely claim completion.

13. Operational and Economic Evaluation​

An application is not production-ready merely because answers are accurate.

Measure:

Latency​

• Time to first token
• Time to first useful action
• Total response time
• Tool latency
• Retrieval latency
• Agent-to-agent communication latency
• P50, P95 and P99

Throughput​

• Requests per second
• Concurrent sessions
• Tokens per second
• Successful tasks per minute

Reliability​

• API failure rate
• Tool failure rate
• Retry rate
• Timeout rate
• Partial-response rate
• Recovery rate

Cost​

• Input-token cost
• Output-token cost
• Embedding cost
• Retrieval cost
• Reranking cost
• Judge cost
• Tool cost
• Infrastructure cost
• Cost per successful task

A useful metric is:

Cost efficiency =
successful tasks
────────────────
total evaluation cost

Resource metrics for self-hosted models​

• GPU utilisation
• GPU memory
• CPU utilisation
• Host memory
• Queue length
• Batch size
• Cache hit rate
• Energy per request
• Autoscaling delay

Hugging Face’s evaluation guidance explicitly recommends considering trade-offs among accuracy, inference speed and memory footprint, especially for production evaluation.

14. A Production Evaluation Funnel​

Running every expensive evaluation on every code change is inefficient. Use a layered evaluation funnel.

Stage 0: Static validation​

Run on every commit.

Examples:

• Prompt-file validation
• JSON-schema checks
• Tool-schema checks
• Type checking
• Linting
• Policy-rule tests
• Deterministic unit tests

Expected duration: seconds.

Stage 1: Small smoke suite​

Run on every pull request.

Examples:

• 20–50 critical cases
• Deterministic temperature
• One trial per case
• Core routing tests
• Core tool tests
• Safety blockers

The change cannot merge if critical tests fail.

Stage 2: Regression suite​

Run before merge or nightly.

Examples:

• Hundreds of golden cases
• Component metrics
• End-to-end metrics
• Candidate-versus-baseline comparisons
• Cost and latency checks

Stage 3: Stochastic robustness suite​

Run nightly or weekly.

Examples:

• Multiple trials per case
• Prompt paraphrases
• Context-order permutations
• Tool-failure injection
• Model-temperature variation
• Long-horizon tasks

Stage 4: Full agent simulation​

Run before major releases.

Examples:

• Sandboxed tool use
• Database state validation
• Browser or computer-use environments
• Multi-agent coordination
• Security attacks
• High-risk workflows

Inspect, developed by the UK AI Security Institute and Meridian Labs, is designed for model and agent evaluation with datasets, solvers, scorers, tools and sandboxed environments. Its current documentation lists more than 200 pre-built evaluations and support for local and hosted model providers.

Stage 5: Human review​

Run for major releases and high-risk changes.

Review:

• Newly generated failures
• Judge disagreements
• High-impact workflows
• Safety cases
• Domain-specific output
• Large candidate-baseline differences

Stage 6: Shadow and canary deployment​

Shadow evaluation​

Send production traffic to the candidate system without showing its outputs to users.

Compare:

• Candidate response
• Current production response
• Tool actions
• Cost
• Latency
• Safety results

Write actions must be disabled or redirected to a sandbox.

Canary deployment​

Expose a small percentage of live traffic to the candidate.

Monitor:

• User feedback
• Resolution
• Escalation
• Errors
• Safety
• Cost
• Latency

Rollback conditions should be defined before deployment.

15. Efficient Evaluation Strategies​

15.1 Use cheap graders first​

Evaluation order:

Do not spend expensive judge calls on responses that already fail a deterministic requirement.

15.2 Use judge cascades​

A low-cost judge evaluates straightforward cases.

Escalate when:

• Confidence is low
• The case is high risk
• Two judges disagree
• The score is near a release threshold
• The output contains complex domain content

ARES demonstrates the broader principle that lightweight trained judges, synthetic data and a smaller human-labelled calibration set can support scalable RAG evaluation.

15.3 Cache evaluation artefacts​

Cache:

• Model responses
• Retrieved documents
• Embeddings
• Judge results
• Parsed claims
• Citation checks
• Tool simulation results

Cache keys should include:

case version
model version
prompt version
retriever version
knowledge-base version
judge version
rubric version
decoding parameters

Never reuse a score when a relevant dependency changes.

15.4 Evaluate only affected slices​

Map system changes to evaluation slices.

Examples:

A full suite should still run periodically.

15.5 Use stratified sampling online​

Do not evaluate only a uniform random production sample.

Sample by:

• Intent
• Language
• User segment
• Risk level
• Tool used
• Model
• Region
• Latency
• Positive or negative feedback
• Escalation status
• New or rare query pattern

Over-sample rare but high-risk cases.

15.6 Use active error sampling​

Prioritise traces with:

• High model uncertainty
• Judge disagreement
• User corrections
• Long trajectories
• Repeated tools
• Safety flags
• High cost
• Low retrieval similarity
• Novel embeddings
• Unexpected termination

This provides more learning value than judging large volumes of routine successful interactions.

15.7 Run repeated trials selectively​

Multiple trials are expensive.

Use repeated trials for:

• High-temperature workflows
• Long-horizon agents
• Cases with historical instability
• Near-threshold release decisions
• Tool-recovery scenarios
• Multi-agent debates

One trial may be sufficient for deterministic structured-output tests.

15.8 Use paired comparisons​

Run baseline and candidate on identical cases.

Paired analysis reduces noise caused by differences in dataset composition.

For binary pass/fail results, a paired test such as McNemar’s test can help determine whether the models fail on meaningfully different examples rather than merely comparing aggregate scores. Hugging Face Evaluate includes comparisons as a distinct evaluation category.

Always report:

• Absolute score
• Difference from baseline
• Confidence interval
• Number of improved cases
• Number of regressed cases
• Severity of regressions

15.9 Use early stopping​

During a large evaluation, stop when:

• A critical safety failure occurs
• The candidate cannot mathematically meet the release threshold
• A cost limit is exceeded
• A statistical decision boundary is reached
• Infrastructure failure invalidates results

15.10 Separate quality monitoring from exhaustive grading​

Online evaluation should not run expensive judges on every trace unless the risk justifies it.

Possible strategy:

100% of traffic:
- Schema
- Errors
- Latency
- Tool permissions
- Safety blockers

10% of traffic:
- Lightweight quality judge
- Groundedness
- User-intent satisfaction

1% of traffic:
- Strong multi-dimensional judge

0.1% or targeted sample:
- Human expert review

16. Statistical Reliability​

An evaluation score without uncertainty can be misleading.

16.1 Confidence intervals​

Report confidence intervals for:

• Task-success rate
• Safety-pass rate
• Retrieval recall
• User preference
• Escalation accuracy

A change from 82% to 83% may not be meaningful on a 50-case test set.

16.2 Multiple trials​

For stochastic agents, report:

• Mean
• Standard deviation
• Minimum
• Maximum
• Pass@1
• Pass@k
• Failure categories

16.3 Slice-level analysis​

A candidate may improve overall while becoming worse for:

• One language
• One region
• Long queries
• High-risk transactions
• Older users
• A specific document type

Aggregate scores must be accompanied by slice-level results.

16.4 Practical significance​

Statistical significance is not necessarily business significance.

Define a minimum meaningful improvement.

For example:

Task success must improve by at least 2 percentage points,
or cost must decrease by at least 10% without quality regression.

16.5 Judge agreement​

Measure agreement between:

• Judge and human
• Judge A and Judge B
• Reviewer A and Reviewer B
• Repeated calls to the same judge

Low agreement suggests that:

• The rubric is ambiguous
• The task is subjective
• The judge is weak
• The human labels are inconsistent
• More context is required

17. Release Gates​

Avoid one weighted score such as:

Overall score =
0.3 correctness +
0.2 relevance +
0.2 safety +
0.2 latency +
0.1 cost

This could allow high relevance to mathematically compensate for a severe safety failure.

Use hard gates and optimisation targets.

Example release policy​

Critical safety failures:               0
Restricted-data leakage:                0
High-risk tool authorisation pass:      100%
Regression-suite pass rate:             ≥ 98%
Task-success lower confidence bound:    ≥ 90%
Retrieval recall@5:                     ≥ 92%
Claim faithfulness:                     ≥ 95%
P95 latency:                            ≤ 4 seconds
Average cost per resolved case:         ≤ $0.12
Candidate-vs-baseline severe regressions: 0

After all hard gates pass, compare candidates using a Pareto analysis across quality, latency and cost.

18. Recommended Tooling Landscape​

MLflow’s current GenAI evaluation system supports evaluation datasets, human feedback, LLM judges, tracing and production monitoring. DeepEval provides component and end-to-end evaluation patterns for RAG and agent workflows.

One current tooling caveat is that OpenAI’s legacy Evals platform is scheduled to become read-only on October 31, 2026 and shut down on November 30, 2026. Its evaluation-design guidance remains useful, but teams should verify migration options (for example Promptfoo or application-owned eval harnesses) before building long-term platform dependencies around the legacy service.

19. Worked Example: Banking Customer-Service Multi-Agent System​

Consider this architecture:

Business scenarios​

• Card payment dispute
• Duplicate charge
• Transfer status
• Lost card
• Address change
• Fee explanation
• Account closure
• Vulnerable-customer support
• Fraud report
• Complaint escalation

19.1 Router evaluation​

Metrics:

• Intent accuracy
• Risk classification recall
• Multi-intent detection
• Fraud false-negative rate
• Vulnerability-detection recall
• Unnecessary high-risk routing

A fraud-related request incorrectly routed as a general FAQ should be treated as a critical failure, not merely one incorrect classification.

19.2 Policy RAG evaluation​

Metrics:

• Policy-document recall
• Correct jurisdiction
• Effective-date handling
• Product-specific filtering
• Claim faithfulness
• Citation correctness
• Exception coverage
• Restricted-policy access control

Test example:

The knowledge base contains:
- An expired refund policy
- A current refund policy
- A business-account exception
- An unrelated credit-card policy

The agent must select the current policy and correctly apply the account-specific exception.

19.3 Customer-data agent evaluation​

Verify:

• Identity confirmed before access
• Correct customer record selected
• No other customer’s data returned
• Minimum necessary fields retrieved
• Sensitive values masked
• Read-only access used where appropriate

19.4 Transaction-agent evaluation​

Test:

• Correct transaction identifier
• Correct amount and currency
• Duplicate detection
• Pending-versus-settled status
• Idempotent dispute creation
• Confirmation before action
• Accurate backend state

19.5 Fraud-agent evaluation​

Measure:

• Fraud-case recall
• False-positive rate
• Correct escalation
• Correct account-protection steps
• No disclosure of internal fraud rules
• No unauthorised account blocking

19.6 Supervisor evaluation​

Assess whether the supervisor:

• Delegates only necessary work
• Chooses the correct specialists
• Preserves the customer’s objective
• Detects conflicting agent conclusions
• Prioritises compliance warnings
• Avoids duplicated data retrieval
• Stops after resolution

19.7 Compliance reviewer evaluation​

Use hard assertions:

No financial action without required identity verification.
No claim of refund completion before backend confirmation.
No full payment-card number in outputs or traces.
No cross-customer data.
Mandatory escalation for suspected fraud.

19.8 End-to-end evaluation​

A case passes only when:

1. The user’s objective is understood.
2. Identity requirements are followed.
3. Relevant policy is retrieved.
4. Correct tools are used.
5. Backend state is correct.
6. Compliance requirements pass.
7. Final communication is accurate and clear.
8. Cost and latency remain within budget.

Example scorecard:

20. Evaluation Anti-Patterns​

One generic “quality” score​

It hides the reason for failure.

Benchmark-only model selection​

Academic benchmarks do not represent your tools, documents, policies or users.

Evaluating only the final response​

The answer may look correct while the external action failed.

Using only an LLM judge​

Some requirements are better tested through code or external state.

Using the candidate model as its only judge​

This increases the risk of self-preference and correlated errors.

Evaluating only happy paths​

Production failures occur in ambiguity, tool errors, stale documents and adversarial inputs.

Changing the dataset during model comparison​

This prevents a fair paired comparison.

Unversioned prompts and judges​

Scores cannot be reproduced if the evaluator changes silently.

Treating all errors equally​

A slightly verbose answer and unauthorised account access should not receive the same severity.

Optimising directly against the hidden test set​

Once developers repeatedly inspect failures and tune against them, the set is no longer truly hidden.

Ignoring evaluation cost​

An evaluation pipeline that is too expensive will not be run frequently enough to protect production quality.

Over-constraining agent trajectories​

Agents may find valid approaches that differ from the reference path. Prefer outcome checks and safety invariants unless a specific action order is mandatory.

21. Recommended Production Architecture​

Every evaluation run should record:

• Application commit
• Prompt version
• Model version
• Model parameters
• Tool versions
• Knowledge-base version
• Dataset version
• Evaluator versions
• Judge model
• Judge prompt
• Environment image
• Start and completion time
• Cost
• Raw traces
• Per-case scores
• Aggregate scores
• Confidence intervals
• Failure categories

22. Final Evaluation Strategy​

A mature LLM evaluation programme should follow these principles:

1. Start with business outcomes. Define what successful task completion means before selecting metrics.

2. Evaluate components and the whole system. Retrieval, routing, tools, agents and synthesis need separate tests.

3. Prefer objective state validation. Verify what happened rather than trusting what the agent says happened.

4. Combine evaluator types. Use code for formal requirements, LLM judges for nuanced language and humans for calibration and high-risk review.

5. Evaluate trajectories without overfitting to one path. Check required invariants, prohibited actions and final outcomes.

6. Treat RAG as retrieval plus generation. A final-answer score alone cannot distinguish retrieval failure from generation failure.

7. Measure whether multiple agents add value. Compare every multi-agent architecture with a strong single-agent baseline.

8. Version everything. Models, prompts, tools, datasets, judges and knowledge bases must be reproducible.

9. Use offline and online evaluation together. Offline suites prevent known regressions; production monitoring discovers unknown failures.

10. Optimise evaluation cost. Use cascades, sampling, caching, concurrency and targeted repeated trials.

11. Use hard safety and compliance gates. Critical violations must never be averaged away.

12. Continuously convert failures into tests. Every meaningful production failure should make the future system harder to break.

The most important shift is to stop asking:

Which LLM has the highest score?

The production question is:

Which complete AI system reliably achieves the required business outcome, under realistic conditions, while remaining safe, compliant, explainable, fast and economically sustainable?

That is the standard a production-grade evaluation framework must measure.

Primary research and documentation consulted​

• Hugging Face Evaluate and LightEval documentation
• RAGAS, RAGChecker and ARES research
• AgentBench, GAIA and WebArena research
• MultiAgentBench and multi-agent collaboration research
• Anthropic agent-evaluation guidance
• OpenAI evaluation and agent-trace guidance
• LangSmith and MLflow evaluation lifecycle documentation
• UK AI Security Institute Inspect documentation
• LLM-as-a-judge research