Skip to content

Observability for OpenAI ChatKit Apps

OpenAI's ChatKit SDK (@openai/chatkit-react) lets you embed a fully managed AI chat widget in any React application. The Anosys integration hooks into ChatKit's lifecycle callbacks to automatically capture every conversation turn, measure performance metrics, and correlate sessions — so you can understand how users interact with your AI assistant in production.

Why Monitor Your ChatKit App

A ChatKit-powered assistant is a production AI interface. Every conversation is a user experience that generates cost, latency, and potential failure points:

What Anosys Captures Why It Matters
Prompts & responses See exactly what users ask and what the model returns
Time-to-first-token Measure perceived responsiveness of the chat widget
Response duration End-to-end latency from user submit to full response
Thread & session IDs Correlate multi-turn conversations across page views
Agentic tasks & thoughts Capture intermediate reasoning steps and tool invocations
User actions Track message actions (copy, retry, feedback) within the widget

Architecture Overview

The integration uses a custom React hook (useChatObservability) that plugs into ChatKit's built-in callbacks:

%%{init: {"flowchart": {"curve": "linear"}}}%%
graph LR
    A[User Message] -->B["ChatKit Widget (@openai/chatkit-react)"]
    B -->C[OpenAI API]
    C -->B
    B -->D["useChatObservability Hook"]
    D -->E["Anosys Platform (POST)"]

    style B stroke:#1e88e5,color:#fff
    style D stroke:#c62828,color:#fff
    style E stroke:#388e3c,color:#fff

How it works:

  1. ChatKit fires onLog, onResponseStart, and onResponseEnd callbacks during each conversation turn.
  2. The observability hook timestamps each phase to compute time-to-first-token and response duration.
  3. On response end, the hook fetches the full message thread from the OpenAI API and extracts prompts, responses, and agentic task summaries.
  4. Everything is shipped to Anosys via a single POST request with structured custom fields.

Getting Started

Prerequisites

  • A React application using @openai/chatkit-react (v1.4+)
  • An Anosys pixel of type Agentic AI (create one at console.anosys.ai)

Step 1 — Install ChatKit

1
npm install @openai/chatkit-react

Step 2 — Add the Observability Hook

Create a custom hook that resolves your Anosys API key and provides the lifecycle callbacks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
// src/hooks/useChatObservability.js

import { useState, useRef, useEffect } from 'react';

export function useChatObservability() {
    const [correlationIds, setCorrelationIds] = useState({
        appSessionId: crypto.randomUUID(),
        threadId: null,
        afterItemId: null,
    });

    const metricsRef = useRef({
        composerSubmitTimestamp: null,
        responseStartTimestamp: null,
    });

    const ANOSYS_API_KEY = import.meta.env.VITE_ANOSYS_API_KEY;

    // Resolve your Anosys ingestion URL from the Console API
    const [ingestionUrl, setIngestionUrl] = useState(null);

    useEffect(() => {
        fetch(`https://console.anosys.ai/api/resolveapikeys?apikey=${ANOSYS_API_KEY}&type=chatkit`)
            .then(res => res.json())
            .then(data => setIngestionUrl(data.apiUrl))
            .catch(err => console.error('Anosys key resolution failed:', err));
    }, []);

    const sendToAnosys = async (payload) => {
        if (!ingestionUrl) return;
        await fetch(ingestionUrl, {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify(payload),
        });
    };

    // --- ChatKit Callbacks ---

    const onLog = (event) => {
        if (event.name === 'composer.submit') {
            metricsRef.current.composerSubmitTimestamp = Date.now();
            metricsRef.current.responseStartTimestamp = null;
        }
    };

    const onResponseStart = () => {
        if (!metricsRef.current.responseStartTimestamp) {
            metricsRef.current.responseStartTimestamp = Date.now();
        }
    };

    const onResponseEnd = async () => {
        const endTs = Date.now();
        const { composerSubmitTimestamp, responseStartTimestamp } = metricsRef.current;

        const metrics = (composerSubmitTimestamp && responseStartTimestamp) ? {
            timeToFirstToken: responseStartTimestamp - composerSubmitTimestamp,
            duration: endTs - responseStartTimestamp,
        } : {};

        // Fetch thread messages and ship to Anosys
        // (see full implementation in the demo repo)
        sendToAnosys({
            event_type: 'chatkit chat',
            timestamp_ms: Date.now(),
            chatkit: {
                session_id: correlationIds.appSessionId,
                thread_id: correlationIds.threadId,
            },
            ...metrics,
        });
    };

    return {
        correlationIds,
        setCorrelationIds,
        onLog,
        onResponseStart,
        onResponseEnd,
    };
}

Step 3 — Wire It Into Your ChatKit Widget

Pass the callbacks from the hook into useChatKit:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
// src/components/ChatWidget.jsx

import { ChatKit, useChatKit } from '@openai/chatkit-react';
import { useChatObservability } from '../hooks/useChatObservability';

const ChatWidget = () => {
    const {
        setCorrelationIds,
        onLog,
        onResponseStart,
        onResponseEnd,
    } = useChatObservability();

    const { control, ref } = useChatKit({
        api: {
            async getClientSecret() {
                // Create a ChatKit session via the OpenAI API
                const res = await fetch('https://api.openai.com/v1/chatkit/sessions', {
                    method: 'POST',
                    headers: {
                        'Authorization': `Bearer ${import.meta.env.VITE_OPENAI_API_SECRET_KEY}`,
                        'Content-Type': 'application/json',
                        'OpenAI-Beta': 'chatkit_beta=v1',
                    },
                    body: JSON.stringify({
                        workflow: { id: import.meta.env.VITE_OPENAI_WORKFLOW_ID },
                    }),
                });
                const data = await res.json();
                return data.client_secret;
            },
        },
        theme: { colorScheme: 'dark' },

        // Anosys observability callbacks
        onLog,
        onResponseStart,
        onResponseEnd,
    });

    return (
        <ChatKit ref={ref} control={control}
            style={{ width: '100%', height: '100%' }} />
    );
};

Step 4 — Set Environment Variables

Add the following to your .env file:

1
2
3
VITE_OPENAI_API_SECRET_KEY=sk-your-openai-key
VITE_OPENAI_WORKFLOW_ID=your-chatkit-workflow-id
VITE_ANOSYS_API_KEY=your-anosys-api-key

What Gets Sent to Anosys

Each conversation turn produces a structured payload with the following fields:

Field Type Description
event_type String Always "chatkit chat"
cvs1 String User's prompt text
cvs2 String Assistant's response text
cvs3 String App session ID (unique per page load)
cvs4 String OpenAI thread ID
cvn1 Number Response duration (ms)
cvn2 Number Time-to-first-token (ms)
messages[] Array Full conversation history with item IDs, timestamps, and message types (prompt, response, thought)

Agentic task groups (intermediate reasoning steps) are captured as thought messages, giving you visibility into the model's chain-of-thought process.

What You'll See in Anosys

Once the integration is live, the Anosys Platform surfaces:

  • Conversation traces — every prompt/response pair with timing metadata and thread correlation
  • Time-to-first-token trends — track perceived responsiveness over time and across users
  • Response duration analysis — identify slow responses, outliers, and degradation patterns
  • Session-level views — group all conversation turns within a user session for end-to-end context
  • Agentic reasoning visibility — inspect intermediate thoughts and task summaries when the model uses tools
  • User action tracking — monitor copy, retry, and feedback actions within the chat widget

Full Reference Implementation

The complete working implementation is available as an open-source demo:

Anosys-AI/AnoSys-Demo-Customer — a React + Vite application built with @openai/chatkit-react and full Anosys observability.

Key files:

  • src/components/ChatWidget.jsx — ChatKit widget with observability callbacks
  • src/hooks/useChatObservability.js — the observability hook (API key resolution, metrics, thread extraction)
  • src/utils/chatHelpers.js — helper functions for message extraction and Anosys payload mapping

Next Steps