Web Analytics Made Easy - Statcounter
Skip to content

Evals

Evals measure whether AI systems are doing the right thing. In Anosys, evals connect quality, safety, relevance, policy, cost, latency, and business outcomes to production traces.

What Evals Do

Evals can run on model outputs, agent runs, tool calls, conversation sessions, process units, and business events. They can run in CI, production, scheduled pipelines, or ad hoc investigations.

When To Use Evals

Use evals when:

  • A model or prompt changes.
  • Product quality is drifting.
  • Safety or policy compliance matters.
  • Cost optimization might hurt quality.
  • Business outcomes need to be measured, not assumed.

Prerequisites

  1. Capture the input, output, and context you want to evaluate.
  2. Define eval dimensions and thresholds.
  3. Decide where results should appear: dashboard, alert, release gate, or report.

Step-By-Step Setup

  1. Open Evals.
  2. Choose the source: dataset, trace sample, workflow, model, or process unit.
  3. Add eval dimensions such as quality, relevance, safety, policy, latency, or business outcome.
  4. Define pass/fail thresholds.
  5. Run against a historical sample.
  6. Attach evals to CI, production alerts, dashboards, or pipelines.

Example Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
name: agent-session-quality
source:
  workflow: support-agent
dimensions:
  - relevance
  - safety
  - resolution_quality
thresholds:
  relevance: 0.90
  safety: pass
  resolution_quality: 0.85

What Appears In The Console

You will see eval score trends, failing examples, related traces, affected workflows, model or prompt versions, and downstream alerts or reports.

Common Mistakes

  • Running evals only on offline datasets.
  • Evaluating output quality without customer or workflow context.
  • Optimizing cost without checking quality.
  • Not versioning rubrics or prompts.