Skip to content

Monitor agents in production

Goal: know when your agents get slower, more expensive or worse, before your users tell you. Time: about 15 minutes. You need: an agent already sending traces. If you don't have one yet, do Debug AI agents first.

%%{init: {"flowchart": {"curve": "linear"}}}%%
graph LR
    A["Live agent traces"] --> B["1. Dashboard<br/>health at a glance"]
    A --> C["2. Alert Builder<br/>Slack, email, webhook"]
    A --> D["3. Eval Runs<br/>quality on a schedule"]
    D --> C

Step 1. Build a health dashboard

  1. Go to Dashboards → Templates, pick a template that matches your source and deploy it. A template can set up dashboards, process units, triggers and pipelines in one go. Or go to Dashboards → Auto-Insights and let the wizard propose charts from your data.
  2. Make sure the dashboard has, at minimum:
    • runs and errors over time (Advanced Time Series),
    • latency per run (Advanced Time Series or Heatmap),
    • tokens and cost by model (Bar Chart or Table),
    • a Trace Timeline for drilling into a single run.

Result: one page that shows whether your agent is healthy right now.

Step 2. Get alerted

  1. Go to Data Processing → Alert Builder and click Create new trigger.
  2. Give the trigger a name and description.
  3. Configure Query: choose the endpoint and the data to watch (for example error count, latency, or cost per run).
  4. Configure Trigger Rule: set the condition (for example error rate above a threshold over the last 15 minutes).
  5. Configure Trigger Alerting Method: connect Slack, Email or Webhooks and choose recipients.
  6. Save.

Start quiet

Send new alerts to yourself or a test channel for a few days. Tune the threshold, then route them to the team.

Step 3. Check quality with scheduled evaluations

Errors and latency don't tell you whether answers are good. Evaluations score real outputs.

  1. Go to Evaluations → Eval Runs and start a new run. The wizard walks you through:
    1. Source & Spans: the endpoint and which spans to score (for example final agent answers).
    2. Pick Evals: choose from Evaluations → Templates (for example relevance, correctness or safety), or your own.
    3. Variable Mapping: map the eval's inputs (question, answer, reference) to your trace fields.
    4. Sample Validation: dry-run on a few real samples and check the scores make sense.
    5. Judge & Triggers: pick the judge and what should happen when scores drop.
    6. Pipeline & Review: set the schedule and send low scores to human review.
  2. Save. The run scores new traffic on schedule. You can also backfill past data.

Scores show up in Evaluations → Overview, and low-scoring items wait in Evaluations → Review & Calibration for a person to check.

See Evaluations for datasets and experiments.

Step 4. Investigate when something fires

  1. Open the alert's dashboard and narrow the time range.
  2. Open the failing runs in the Trace Timeline or Trace Tree.
  3. Ask the AI Assistant, for example "What changed in my agent's failed runs since yesterday?"
  4. For a written root-cause summary, add an RCA Report widget to the dashboard.

Next steps

  • Track AI spend: cost alerts and per-user cost
  • Pipelines: chain queries, evaluations and alerts into scheduled jobs