Troubleshoot Olakai Agent Monitoring

This skill helps diagnose and fix common issues with Olakai AI agent monitoring, KPI calculations, and SDK integration.

For full documentation, see: https://app.olakai.ai/llms.txt

The Golden Rule: Test → Fetch → Validate

Always diagnose by generating a real event and inspecting it. Don’t guess – look at the actual data.

# 1. Trigger your agent/app to generate an event
# 2. Fetch the event
olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json

# 3. Inspect it completely
olakai activity get EVENT_ID --json | jq '{customData, kpiData}'

This reveals exactly what’s happening:

• No event? → SDK/API key problem
• customData missing fields? → SDK code problem
• kpiData shows strings? → Formula storage problem
• kpiData shows null? → CustomDataConfig or field name problem

Quick Diagnosis Commands

Run these first to understand the current state:

# Check CLI authentication
olakai whoami

# List recent events (are any coming through?)
olakai activity list --limit 10

# List agents (is your agent registered?)
olakai agents list

# List custom data configs (are they set up?)
olakai custom-data list

# List KPIs for an agent
olakai kpis list --agent-id YOUR_AGENT_ID

Issue: No Events Appearing

Symptom

Events from your agent aren’t showing up in olakai activity list or the dashboard.

Diagnostic Steps

1. Verify API Key

# Check environment variable is set
echo $OLAKAI_API_KEY

# Should start with "sk_" and be ~40+ characters
# If empty or wrong, set it:
export OLAKAI_API_KEY="sk_your_key_here"

2. Check SDK Initialization

TypeScript – ensure init() is awaited:

const olakai = new OlakaiSDK({ apiKey: process.env.OLAKAI_API_KEY! });
await olakai.init();  // <-- Must await this!

Python – ensure config is called before instrumentation:

olakai_config(os.getenv("OLAKAI_API_KEY"))  # <-- Must be first
instrument_openai()  # <-- Then instrument

3. Enable Debug Mode

TypeScript:

const olakai = new OlakaiSDK({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: true,  // <-- Add this
});

Python:

olakai_config(api_key, debug=True)

Look for error messages in console output.

4. Test Direct API Call

curl -X POST "https://app.olakai.ai/api/monitoring/prompt" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $OLAKAI_API_KEY" \
  -d '{
    "prompt": "Test prompt",
    "response": "Test response",
    "app": "test-agent"
  }'

Expected: 200 OK with event ID If 401: Invalid API key If 400: Check request format

5. Check Network/Firewall

Ensure your environment can reach https://app.olakai.ai

curl -I https://app.olakai.ai/api/health

Common Fixes

Issue: KPIs Showing String Values Instead of Numbers

Symptom

KPI values display as "VariableName" instead of numeric values like 42.

Example:

"kpiData": {
  "Tools Discovered": "ToolsDiscovered",  // Wrong: string
  "New Tools Found": "NewToolsFound"       // Wrong: string
}

Should be:

"kpiData": {
  "Tools Discovered": 20,  // Correct: number
  "New Tools Found": 3     // Correct: number
}

Root Cause

KPI formulas are stored as raw strings instead of parsed AST objects.

Diagnostic Steps

1. Check KPI Formula Storage

olakai kpis list --agent-id YOUR_AGENT_ID --json | jq '.[] | {name, calculatorParams}'

Wrong (raw string):

{
  "name": "Tools Discovered",
  "calculatorParams": {
    "formula": "ToolsDiscovered"  // <-- String, not object
  }
}

Correct (AST object):

{
  "name": "Tools Discovered",
  "calculatorParams": {
    "formula": {
      "type": "variable",
      "name": "ToolsDiscovered"
    }
  }
}

2. Check CustomDataConfig Exists

olakai custom-data list --json | jq '.[].name'

Ensure every variable referenced in KPI formulas has a corresponding CustomDataConfig.

Fix

Option A: Update via CLI (Recommended)

The CLI now validates and parses formulas automatically:

# This will parse "ToolsDiscovered" into proper AST
olakai kpis update KPI_ID --formula "ToolsDiscovered"

Option B: Validate First, Then Check

# Validate the formula
olakai kpis validate --formula "ToolsDiscovered" --agent-id YOUR_AGENT_ID

# Should return:
# {
#   "valid": true,
#   "type": "number",
#   "parsedFormula": { "type": "variable", "name": "ToolsDiscovered" }
# }

Option C: Recreate the KPI

# Delete the broken KPI
olakai kpis delete KPI_ID --force

# Create with proper formula (CLI now parses automatically)
olakai kpis create \
  --name "Tools Discovered" \
  --agent-id YOUR_AGENT_ID \
  --calculator-id formula \
  --formula "ToolsDiscovered" \
  --aggregation SUM

Post-Fix Verification

After fixing, trigger a new event and check:

olakai activity list --agent-id YOUR_AGENT_ID --limit 1 --json | jq '.prompts[0].kpiData'

Should now show numeric values.

Issue: CustomData Not Appearing in Events

Symptom

You’re sending customData but it’s not showing in event details.

Diagnostic Steps

1. Check Event Details

olakai activity get EVENT_ID --json | jq '.customData'

If null or missing fields, the SDK isn’t sending them.

2. Verify SDK Code

TypeScript – customData in call options:

const response = await openai.chat.completions.create(
  { model: "gpt-4o", messages },
  {
    customData: {
      myField: "value",      // <-- Check this is present
      myNumber: 42,
    },
  }
);

TypeScript – customData in manual event:

olakai.event({
  prompt,
  response,
  customData: {
    myField: "value",
    myNumber: 42,
  },
});

3. Check Field Names Match CustomDataConfig

Field names are case-sensitive. Ensure exact match:

# What you configured
olakai custom-data list --json | jq '.[].name'
# Output: "ToolsDiscovered", "NewToolsFound"

# What you're sending (must match exactly)
customData: {
  ToolsDiscovered: 20,    // Correct
  toolsDiscovered: 20,    // WRONG - case mismatch
  tools_discovered: 20,   // WRONG - different format
}

Common Fixes

Issue: customData Fields Not Available in KPIs

Symptom

You’re sending fields in customData but they can’t be used in KPI formulas – they don’t appear as available variables.

Root Cause

The SDK accepts any JSON in customData, but only fields with CustomDataConfigs become KPI variables.

SDK customData → CustomDataConfig (Schema) → Context Variable → KPI Formula
                        ↑
                  REQUIRED for field
                  to be usable in KPIs

Fields without CustomDataConfigs:

• ✅ Are stored in the event record
• ❌ Cannot be referenced in KPI formulas
• ❌ Don’t appear in formula variable suggestions
• ❌ Effectively wasted for analytics purposes

Diagnostic

1. List what CustomDataConfigs exist:

olakai custom-data list --json | jq '.[].name'

2. Compare to what you’re sending in SDK:

// If you're sending these fields:
customData: {
  ItemsProcessed: 10,   // Is there a CustomDataConfig for this?
  SuccessRate: 0.95,    // Is there a CustomDataConfig for this?
  RandomField: "xyz",   // Is there a CustomDataConfig for this?
}

3. Check for mismatches:

• Field sent but no config → Cannot use in KPIs
• Config exists but field not sent → KPI shows null

Fix

Create CustomDataConfigs for every field you want to use in KPIs:

# For each field you need in KPIs, create a config
olakai custom-data create --name "ItemsProcessed" --type NUMBER
olakai custom-data create --name "SuccessRate" --type NUMBER
olakai custom-data create --name "RandomField" --type STRING

# Verify
olakai custom-data list

Best Practice: Design your CustomDataConfigs FIRST, then write SDK code that sends only those fields.

Common Mistakes

Issue: Redundant customData Fields

Symptom

CustomDataConfigs exist for fields that are already tracked by the platform (sessionId, agentId, timestamps, etc.), cluttering the configuration.

Why This Happens

The SDK accepts any JSON in customData, so agents sometimes send “helpful” extra data that’s already tracked elsewhere.

What’s Already Tracked (Don’t Duplicate)

Fix

1. Identify redundant configs:

olakai custom-data list --json | jq '.[].name'

Look for names like: sessionId, agentId, timestamp, model, provider, tokenCount

2. Check if they’re used in KPIs:

olakai kpis list --agent-id YOUR_AGENT_ID --json | jq '.[].calculatorParams.formula'

3. If not used in KPIs, remove from SDK code: Don’t delete the configs (they may have historical data), but stop sending these fields.

4. Update SDK code to only send KPI-relevant fields:

customData: {
  // ✅ Keep: Used in KPI formulas
  ItemsProcessed: 10,
  SuccessRate: 1.0,

  // ❌ Remove: Already tracked by platform
  // sessionId: session.id,      // Already tracked
  // agentId: agentConfig.id,    // Already tracked
  // timestamp: Date.now(),      // Already tracked
}

Best Practice

Before creating a CustomDataConfig, ask:

• “Will I use this in a KPI formula?” → Yes = Create
• “Will I filter events by this?” → Yes = Create
• “Is this already tracked by the platform?” → Yes = Don’t create

Issue: KPIs Show null Values

Symptom

KPI values are null instead of numbers.

"kpiData": {
  "Success Rate": null,
  "Items Processed": null
}

Root Cause Options

1. CustomData field not sent – The event doesn’t include that field
2. CustomDataConfig doesn’t exist – Platform can’t resolve the variable
3. Type mismatch – Sending string when NUMBER expected

Diagnostic Steps

1. Check if CustomData is Present

olakai activity get EVENT_ID --json | jq '.customData'

If the field is missing from customData, the KPI can’t calculate.

2. Check CustomDataConfig Exists

olakai custom-data list --json | jq '.[].name'

Every field referenced in KPI formulas needs a config.

3. Check Formula Validity

olakai kpis validate --formula "YourVariable" --agent-id YOUR_AGENT_ID

If invalid, you’ll see the error.

Fix

1. Ensure CustomDataConfig Exists

olakai custom-data create --name "YourVariable" --type NUMBER

2. Ensure SDK Sends the Field

customData: {
  YourVariable: 42,  // Must be present and correct type
}

3. Trigger New Event

Old events won’t recalculate. Generate a new event to verify the fix.

Issue: CLI Authentication Failed

Symptom

Error: Authentication required
Error: Token expired
Error: Unauthorized

Fix

1. Re-authenticate

olakai logout
olakai login

2. Verify Authentication

olakai whoami

3. Check Credentials File

ls -la ~/.config/olakai/
cat ~/.config/olakai/credentials.json

Should contain accessToken and refreshToken.

4. Check Environment

# Default is production
olakai whoami

# For staging
OLAKAI_ENV=staging olakai whoami

Issue: Events Not Associated with Agent

Symptom

Events appear in general activity but not under your agent.

Root Cause

The app field in events doesn’t match the agent name.

Diagnostic

# Check agent name
olakai agents list --json | jq '.[] | {id, name}'

# Check event app field
olakai activity get EVENT_ID --json | jq '.app'

Fix

Option A: Match App Name to Agent

In your SDK code, ensure the app name matches:

olakai.event({
  prompt,
  response,
  app: "Your Agent Name",  // Must match agent name exactly
});

Or with wrapped client:

const openai = olakai.wrap(new OpenAI({ apiKey }), {
  provider: "openai",
  defaultContext: {
    app: "Your Agent Name",
  },
});

Option B: Update Agent Name

olakai agents update AGENT_ID --name "App Name From Events"

Issue: High Latency / Slow Events

Symptom

LLM calls take much longer when monitoring is enabled.

Diagnostic

Check if it’s the SDK or the LLM:

const startSDK = Date.now();
const response = await openai.chat.completions.create(...);
console.log(`Total time: ${Date.now() - startSDK}ms`);

Fixes

1. SDK Uses Fire-and-Forget

The SDK should NOT block your application. If it is:

// Ensure you're not awaiting event()
olakai.event(params);  // No await - fire and forget

2. Check Network Latency

time curl -I https://app.olakai.ai/api/health

If >500ms, consider:

• Running in a region closer to US-East
• Checking for proxy/VPN overhead

3. Disable Debug Mode in Production

const olakai = new OlakaiSDK({
  apiKey: process.env.OLAKAI_API_KEY!,
  debug: false,  // Disable in production
});

Issue: Duplicate Events

Symptom

Each LLM call creates multiple events.

Root Cause

Usually double-initialization or multiple wrapping.

Diagnostic

Search your code for:

• Multiple olakai.init() calls
• Multiple olakai.wrap() calls
• Multiple instrument_openai() calls

Fix

Use Singleton Pattern

// lib/olakai.ts
let instance: OlakaiSDK | null = null;

export async function getOlakai(): Promise<OlakaiSDK> {
  if (!instance) {
    instance = new OlakaiSDK({ apiKey: process.env.OLAKAI_API_KEY! });
    await instance.init();
  }
  return instance;
}

Diagnostic Commands Reference

# Authentication
olakai whoami                          # Check current user
olakai login                           # Re-authenticate

# Events/Activity
olakai activity list --limit N         # Recent events
olakai activity list --agent-id ID     # Events for specific agent
olakai activity get EVENT_ID --json    # Full event details

# Agents
olakai agents list                     # All agents
olakai agents list --json | jq '.[].name'  # Just names

# KPIs
olakai kpis list --agent-id ID         # KPIs for agent
olakai kpis list --agent-id ID --json | jq '.[] | {name, calculatorParams}'
olakai kpis validate --formula "X"     # Test formula

# Custom Data
olakai custom-data list                # All configs
olakai custom-data list --json | jq '.[].name'  # Just names

# Quick Health Check
olakai whoami && olakai activity list --limit 1 && echo "OK"

Troubleshooting Decision Tree

Events not appearing?
├── Check API key is set → export OLAKAI_API_KEY=sk_...
├── Check SDK initialized → await olakai.init()
├── Enable debug mode → debug: true
└── Test direct API → curl POST /api/monitoring/prompt

KPIs showing strings?
├── Check formula storage → olakai kpis list --json
├── Formula is raw string → olakai kpis update ID --formula "X"
└── Missing CustomDataConfig → olakai custom-data create

KPIs showing null?
├── Check customData sent → olakai activity get ID --json
├── Field missing → Add to customData in SDK
├── CustomDataConfig missing → olakai custom-data create
└── Type mismatch → NUMBER needs number, STRING needs string

customData field not usable in KPIs?
├── Check CustomDataConfig exists → olakai custom-data list
├── Config missing → olakai custom-data create --name "Field" --type NUMBER
└── Case mismatch → Ensure exact case match between SDK and config

Events not under agent?
├── Check app name matches → Compare event.app to agent.name
├── Mismatch → Update agent name or SDK app field

Key Insight: The customData → KPI Pipeline

Only fields registered as CustomDataConfigs become available in KPI formulas:

SDK customData → CustomDataConfig → Context Variable → KPI Formula → kpiData
       ↓                ↓                  ↓               ↓            ↓
  Any JSON         Schema definition   Available var   Expression   Computed value
                   (REQUIRED)

Common pitfall: Sending extra fields in customData without CustomDataConfigs – they’re stored but unusable for KPIs.