Skip to content

Observability for OpenAI Agents

Gain complete visibility into your OpenAI-powered agentic workflows. The Anosys Platform integrates with the OpenAI Python and JavaScript SDKs to automatically capture every API call, tool invocation, and model response — giving you the insights you need to optimize costs, debug failures, and ship faster.

Why Observability Matters for AI Agents

Agentic AI workflows are fundamentally different from traditional software — they are non-deterministic, multi-step, and often expensive to run. Without observability you are flying blind:

Challenge What Observability Gives You
Non-deterministic outputs Trace every reasoning step so you can reproduce and compare runs
Runaway token usage Real-time metrics on input/output tokens per session
Silent failures Structured logs with error classification and alerting
Slow iterations Latency breakdowns across tool calls and model invocations
Cost overruns Per-session and per-project cost attribution dashboards

Getting Started with OpenAI Agents

The Anosys SDK provides lightweight wrappers for both the OpenAI SDK and the OpenAI Agents SDK. Once initialized, every API call is automatically traced and exported — no manual instrumentation required.

To get started you need to:

  1. Create an ingestion channel in the Anosys Console. Go to the API tab and click Create New API Key to get your key.
  2. Install the Anosys SDK packages.
  3. Initialize the logger in your code before making OpenAI calls.

OpenAI SDK Integration

Use this when you're calling the OpenAI API directly (chat completions, embeddings, etc.).

Install

1
pip install anosys-sdk-openai

1
npm install anosys-sdk-openai

Initialize and Run

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os
from openai import OpenAI
from anosys_sdk_openai import AnosysOpenAILogger

# Set your API keys
os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
os.environ["ANOSYS_API_KEY"] = "YOUR_ANOSYS_API_KEY"

# Initialize the Anosys logger — this auto-instruments all OpenAI calls
AnosysOpenAILogger()

# Use the OpenAI client as usual
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"}
    ]
)

print(response.choices[0].message.content)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import { AnosysOpenAILogger } from 'anosys-sdk-openai';
import OpenAI from 'openai';

// Set your API keys as environment variables:
// OPENAI_API_KEY=your-openai-key
// ANOSYS_API_KEY=your-anosys-key

// Initialize once — instruments all subsequent OpenAI calls automatically
new AnosysOpenAILogger();

const client = new OpenAI();
const response = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Hello!' },
  ],
});

console.log(response.choices[0].message.content);

That's it. Every OpenAI API call made through the client is now automatically traced.

Environment variables

For production deployments, set OPENAI_API_KEY and ANOSYS_API_KEY as environment variables rather than hardcoding them. The SDK reads both from the environment automatically.

OpenAI Agents SDK Integration

Use this when you're building agents with the OpenAI Agents SDK (openai-agents). The Anosys integration plugs into the Agents SDK's tracing processor to capture agent runs, tool calls, handoffs, guardrails, and generation spans.

Install

1
pip install anosys-sdk-openai-agents

1
npm install anosys-sdk-openai-agents

Initialize and Run

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import os
from agents import Agent, Runner, set_tracing_processor
from anosys_sdk_openai_agents import AnosysOpenAIAgentsLogger

os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
os.environ["ANOSYS_API_KEY"] = "YOUR_ANOSYS_API_KEY"

# Set up the tracing processor
set_tracing_processor(AnosysOpenAIAgentsLogger())

# Create and run an agent
agent = Agent(
    name="Assistant",
    instructions="You are a helpful assistant."
)

result = Runner.run_sync(agent, "Hello!")
print(result.final_output)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import { AnosysOpenAIAgentsLogger } from 'anosys-sdk-openai-agents';
import { Agent, run, addTracingProcessor } from '@openai/agents';

// Register the tracing processor — captures all agent traces automatically
const logger = new AnosysOpenAIAgentsLogger();
addTracingProcessor(logger);

const agent = new Agent({ name: 'my-agent', instructions: 'You are helpful.' });
const result = await run(agent, 'Hello');
console.log(result.finalOutput);

User Context

Associate traces with user sessions for better segmentation and analysis:

1
2
3
4
5
6
7
8
def get_user_context():
    return {
        "session_id": "user-123",
        "token": "auth-token"
    }

processor = AnosysOpenAIAgentsLogger(get_user_context=get_user_context)
set_tracing_processor(processor)

1
2
3
new AnosysOpenAIAgentsLogger({
  getUserContext: () => ({ session_id: req.sessionId, token: req.token }),
});

Span Types Captured

The Agents SDK integration captures the following span types:

Span Type Description
agent Agent execution spans with handoffs and tool lists
function Tool/function call invocations
generation LLM generation calls with token usage
guardrail Guardrail evaluation checks
handoff Agent-to-agent handoffs
response Response spans
custom Custom user-defined spans
mcp_tools MCP tool invocations

Streaming Support

Streaming is automatically detected and logged with both Python and JavaScript SDKs:

1
2
3
4
5
6
7
8
9
stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Write a haiku"}],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

1
2
3
4
5
6
7
8
9
const stream = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Write a haiku' }],
  stream: true,
});

for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content || '');
}

Custom Function Logging

Log custom functions using decorators to capture inputs, outputs, and execution time:

1
2
3
4
5
from anosys_sdk_core import anosys_logger

@anosys_logger(source="my_app")
def process_response(response):
    return response.choices[0].message.content

1
2
3
4
5
6
import { anosysLogger } from 'anosys-sdk-openai';

const processResponse = anosysLogger('my_app.pipeline')(async (input) => {
  // your logic
  return result;
});

What the OpenAI SDK Captures

Because the Anosys logger wraps the OpenAI SDK directly, it captures rich data beyond standard OTLP telemetry:

Data Point Description
Model name & version The exact model used for each call (e.g. gpt-5, gpt-4.1-mini)
Prompt & completion tokens Precise input/output token counts per request
Request parameters Temperature, top-p, max tokens, stop sequences, and other model settings
Tool / function calls Names, arguments, and return values of any tool calls made by the agent
Streaming events Per-chunk latency for streaming responses
Response metadata Finish reason, system fingerprint, and response ID
Error details HTTP status codes, rate-limit headers, and retry counts

What You'll See in Anosys

Once the logger is active, the Anosys Platform automatically processes your OpenAI data and surfaces:

  • Request traces — End-to-end visibility into every OpenAI API call, including multi-turn conversations and chained agent executions.
  • Token usage metrics — Input and output token counts per request, per model, and over time.
  • Model comparison — Side-by-side performance and cost breakdowns across different models (e.g. gpt-5 vs gpt-4.1-mini).
  • Latency analysis — Identify slow model calls and bottlenecks, including time-to-first-token for streaming responses.
  • Error tracking — Structured error logs with automatic classification, including rate-limit events and API errors.
  • Cost insights — Per-request and per-project cost estimates based on actual token usage and model pricing.
  • Anomaly detection — ML-powered baselines that alert on token usage spikes, latency regressions, and model quality degradation without manual threshold configuration.
  • Root cause analysis — Causal graphs that connect failures to upstream triggers across multi-step agent executions, tool calls, and model invocations.
  • Alerts — Context-aware notifications via Slack, email, or webhooks when your agents hit errors, cost overruns, or performance regressions.
  • Custom dashboards — Build your own views or start with auto-generated dashboards for model health, agent reliability, and cost attribution.
  • Automated metric generation — Anosys automatically generates key metrics from your traces and logs so you get dashboards in minutes, not days.
  • Custom pipelines — Enrich, route, and transform your agent telemetry with automated remediation workflows.
  • Labeling — Tag and annotate agent sessions, models, or projects with custom labels for segmentation and drill-down analysis.
  • AI Copilot — Ask questions about your agent data in plain English and get answers backed by your telemetry.

Configuration Reference

Variable / Setting Description
OPENAI_API_KEY Your OpenAI API key
ANOSYS_API_KEY Your Anosys API key (from the Console — API tab)
ANOSYS_API_URL (Optional) Override the logging endpoint
ANOSYS_LOG_LEVEL (Optional, JS only) debug / info / warn / error / silent (default: warn)

Troubleshooting

I don't see any data in the Anosys Console
  • Verify that ANOSYS_API_KEY is set correctly and matches the key shown in your Anosys Console.
  • Make sure AnosysOpenAILogger() is called before you create the OpenAI() client.
  • For the Agents SDK, ensure set_tracing_processor() (Python) or addTracingProcessor() (JS) is called before running agents.
  • Confirm that your OpenAI calls are completing successfully (check for API errors).
  • Ensure outbound HTTPS traffic to api.anosys.ai is not blocked by a firewall or proxy.
Does this work with async and streaming calls?

Yes. The Anosys logger automatically instruments both synchronous and asynchronous OpenAI clients, including streaming responses via stream=True.

Can I use this alongside other OTLP collectors?

Yes. The Anosys logger can coexist with other OpenTelemetry instrumentation. If you need to fan out to multiple backends, place an OpenTelemetry Collector in between and configure multiple exporters there.

Which OpenAI SDK versions are supported?

The anosys-sdk-openai package supports OpenAI Python SDK v1.x and later. The anosys-sdk-openai-agents package supports the OpenAI Agents SDK.

Next Steps