Skip to content

Observability for Any LLM Provider

Anosys is vendor-agnostic. While we offer native integrations for OpenAI, Anthropic and OpenRouter, you can connect any LLM provider to the Anosys Platform using OpenTelemetry (OTLP) or our REST API. If your model can be called from code, it can be observed with Anosys.

Supported LLM Providers

Anosys works with every major LLM vendor and model hosting platform:

Provider Models Integration Method
Google Gemini Gemini 2.5 Pro, Gemini 2.5 Flash OTLP, REST API
Meta Llama Llama 4, Llama 3.3, Code Llama OTLP, REST API
Mistral AI Mistral Large, Mistral Medium, Codestral OTLP, REST API
Cohere Command R+, Embed, Rerank OTLP, REST API
AWS Bedrock Claude, Llama, Titan, Mistral via Bedrock OTLP, REST API
Azure OpenAI GPT-4o, GPT-4.1, GPT-5 via Azure OTLP, REST API
Google Vertex AI Gemini, PaLM, custom models OTLP, REST API
Hugging Face Inference API, Inference Endpoints OTLP, REST API
Fireworks AI Llama, Mixtral, custom fine-tunes OTLP, REST API
Together AI Open-source models at scale OTLP, REST API
Groq Llama, Mixtral on LPU hardware OTLP, REST API
Replicate Open-source models on demand OTLP, REST API
Ollama Local models (Llama, Mistral, Phi) OTLP, REST API
vLLM / TGI Self-hosted inference servers OTLP, REST API
OpenRouter Hundreds of models through one API OpenRouter Broadcast, no code. See OpenRouter
Custom / Private Fine-tuned models, proprietary endpoints OTLP, REST API

Don't see your provider listed? It doesn't matter — if you can call it from code, you can observe it with Anosys.

How to Integrate Any LLM

There are two approaches to adding observability for any model provider:

If your application is already instrumented with OpenTelemetry, or you're using a framework that supports it (LangChain, LlamaIndex, Haystack, CrewAI, AutoGen, etc.), point your OTLP exporter at your Anosys endpoint and data flows automatically.

Configure your environment:

1
2
3
4
5
6
export OTEL_SERVICE_NAME="my-llm-app"
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="YOUR_ANOSYS_OTLP_ENDPOINT"

Replace YOUR_ANOSYS_OTLP_ENDPOINT with the OTLP endpoint URL from your Anosys Console pixel of type Agentic AI.

This works with any OTEL-compatible library, including:

  • Python: opentelemetry-sdk, opentelemetry-instrumentation-*
  • JavaScript/TypeScript: @opentelemetry/sdk-node
  • Go: go.opentelemetry.io/otel
  • Java: io.opentelemetry

For a full OTLP/HTTP setup example in Python, see the OpenTelemetry guide.

Option 2 — REST API

Wrap your LLM calls with a simple HTTP POST to the Anosys ingestion endpoint. This works from any language without any SDK dependency.

Python example — instrumenting a Google Gemini call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import os
import time
import requests
import google.generativeai as genai

# Configure Gemini
genai.configure(api_key=os.environ["GEMINI_API_KEY"])
model = genai.GenerativeModel("gemini-2.5-flash")

# Anosys ingestion endpoint
ANOSYS_URL = "https://api.anosys.ai/ingestion/YOUR_WORKSPACE/YOUR_TYPE/YOUR_ENDPOINT"

# Make the LLM call and measure it
prompt = "Explain quantum computing in simple terms"
start_ts = time.time()

response = model.generate_content(prompt)

duration_ms = (time.time() - start_ts) * 1000

# Ship to Anosys
requests.post(ANOSYS_URL, json=[{
    "cvs1": prompt,                        # prompt text
    "cvs2": response.text,                 # response text
    "cvs3": "gemini-2.5-flash",            # model name
    "cvs4": "google",                      # provider
    "cvn1": duration_ms,                   # latency
    "cvn2": response.usage_metadata.prompt_token_count,     # input tokens
    "cvn3": response.usage_metadata.candidates_token_count  # output tokens
}], timeout=10)

cURL example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
curl -X POST "https://api.anosys.ai/ingestion/YOUR_WORKSPACE/YOUR_TYPE/YOUR_ENDPOINT" \
  -H "Content-Type: application/json" \
  -d '[{
    "cvs1": "Explain quantum computing",
    "cvs2": "Quantum computing uses qubits...",
    "cvs3": "llama-4-maverick",
    "cvs4": "meta",
    "cvn1": 342.5,
    "cvn2": 12,
    "cvn3": 156
  }]'

Option 3 — Python / JS Decorator

For the lightest instrumentation, use the Anosys decorator to auto-capture any function that wraps an LLM call:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from anosys_sdk_core import anosys_logger

@anosys_logger(source="mistral-chat")
def ask_mistral(prompt):
    # Your Mistral API call here
    response = mistral_client.chat.complete(
        model="mistral-large-latest",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

answer = ask_mistral("What are the benefits of RAG?")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import { anosysLogger } from 'anosys-sdk-openai';

const askMistral = anosysLogger('mistral-chat')(async (prompt) => {
  const response = await mistralClient.chat.complete({
    model: 'mistral-large-latest',
    messages: [{ role: 'user', content: prompt }],
  });
  return response.choices[0].message.content;
});

const answer = await askMistral('What are the benefits of RAG?');

The decorator automatically captures the function name, arguments, return value, and execution time.

What to Capture

Regardless of the integration method, we recommend sending these fields for maximum observability:

Field Type Description
cvs1 → prompt Text The user's input prompt
cvs2 → response Text The model's response text
cvs3 → model Text Model name and version
cvs4 → provider Text Vendor name (e.g. google, meta, mistral)
cvn1 → duration_ms Number End-to-end latency in milliseconds
cvn2 → input_tokens Number Input token count
cvn3 → output_tokens Number Output token count
cvn4 → cost_usd Number Estimated cost per request (optional)

Give these fields their names in Collecting → Field Mapping.

Agentic Framework Support

Many popular agentic frameworks already emit OpenTelemetry traces. Configure their OTLP exporter to point at Anosys and you get observability for free:

Framework Language OTEL Support
LangChain / LangGraph Python, JS Built-in via callbacks
LlamaIndex Python Built-in instrumentation
CrewAI Python OTEL-compatible
AutoGen Python OTEL-compatible
Haystack Python OTEL-compatible
Semantic Kernel C#, Python OTEL-compatible
Vercel AI SDK TypeScript OTEL-compatible
Mastra TypeScript OTEL-compatible

For frameworks without built-in OTEL support, use the REST API or Python decorator approach.

Next Steps