LLM API Benchmark

Automatically detects the current LLM API endpoint from environment variables and performs performance benchmarking.

Usage

Simply invoke this skill when you want to benchmark your current LLM API:

"Test API speed"
"Run benchmark"
"测试API速度"
"测试响应时间"

Or use the command directly:

python skills/llm-api-benchmark/scripts/benchmark.py

Supported Providers

The tool auto-detects these providers from environment variables:

Options

# List available presets
python benchmark.py --list-presets

# Use preset (recommended for consistent results)
python benchmark.py --preset throughput   # For TPS testing (~300-500 tokens)
python benchmark.py --preset quick        # Fast test (~10 tokens)
python benchmark.py --preset standard     # Medium (~20 tokens)

# Custom iterations
python benchmark.py --iterations 10

# Custom model
python benchmark.py --model gpt-4o

# Custom prompt
python benchmark.py --prompt "Your test prompt"

# Custom output directory
python benchmark.py --output-dir ./my-reports

# Quiet mode (less output)
python benchmark.py --preset throughput -q

Presets

Recommended: Use --preset throughput for accurate TPS measurement.

Output

Reports are saved to reports/llm-benchmark-{timestamp}/:

reports/llm-benchmark-20260227-103000/
├── benchmark-report.md    # Human-readable report
└── benchmark-data.json    # Raw data for programmatic access

Metrics

The benchmark measures:

• Response Time: End-to-end latency
• TTFT (Time To First Token): Time until first token received
• TPS (Tokens Per Second): Token generation throughput

Statistics include: average, min, max, P50, P95, P99

Example Report

See ../../plans/llm-api-benchmark-plan.md for expected report format.