Performance Profiling

Overview

Optimizing without measuring is guessing. Guessing wastes time and often makes things worse.

Core principle: ALWAYS measure before optimizing. Intuition about performance is wrong more often than right.

Violating the letter of this process is violating the spirit of performance engineering.

The Iron Law

MEASURE FIRST, OPTIMIZE SECOND - NO PREMATURE OPTIMIZATION WITHOUT PROFILING DATA

If you haven’t profiled it, you cannot optimize it. Hunches are not data.

When to Use

Use for ANY performance concern:

• Slow API responses
• High memory usage
• Slow page loads
• Database query timeouts
• Build/CI pipeline slowdowns
• Memory leaks
• Throughput degradation under load

Use this ESPECIALLY when:

• Someone says “this feels slow”
• Under pressure to “just make it faster”
• A “quick optimization” seems obvious
• You want to add caching
• You want to rewrite in a “faster” language
• You’re about to refactor for performance

Don’t skip when:

• The fix seems obvious (obvious fixes are often wrong)
• It’s “just” adding an index (measure the impact)
• You’re an expert (experts still measure)

The Four Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Establish Baseline

BEFORE changing ANY code:

1. Define What “Fast Enough” Means

   • Set concrete targets: “API responds in < 200ms at p95”
   • No vague goals like “make it faster”
   • Agree on metrics with stakeholders
   • If you can’t define the target, you can’t know when you’re done

2. Measure Current Performance

   # HTTP endpoints
   hey -n 1000 -c 50 https://api.example.com/endpoint
   
   # Application benchmarks
   hyperfine --warmup 3 'command-to-benchmark'
   
   # Database queries
   EXPLAIN ANALYZE SELECT ...;
   

3. Record the Baseline

   • Response time: p50, p95, p99
   • Throughput: requests/second
   • Resource usage: CPU%, memory, I/O
   • Error rate under load
   • Write these numbers down. You will compare against them.

4. Reproduce Under Realistic Conditions

   • Production-like data volume
   • Concurrent users matching real traffic
   • Warm caches if production has warm caches
   • Cold start if measuring cold start

No baseline? No optimization. Period.

Phase 2: Profile to Find Bottlenecks

Find where time is actually spent:

1. CPU Profiling

   # Node.js
   node --prof app.js
   node --prof-process isolate-*.log
   
   # Python
   python -m cProfile -o output.prof script.py
   
   # Go
   go tool pprof http://localhost:6060/debug/pprof/profile
   

2. Generate Flame Graphs

   • Flame graphs show exactly where CPU time goes
   • Wide bars = time spent there
   • Look for unexpected wide bars
   • Don’t guess – read the graph

3. Memory Profiling

   # Node.js
   node --inspect app.js
   # Chrome DevTools → Memory → Heap Snapshot
   
   # Python
   tracemalloc.start()
   # ... run code ...
   snapshot = tracemalloc.take_snapshot()
   

4. I/O and Network Profiling

   • Database query logging with timing
   • Network request tracing (OpenTelemetry)
   • File system I/O monitoring
   • External API call latency

5. Identify the Actual Bottleneck

   Bottleneck Type	Symptoms	Tools
   CPU-bound	High CPU%, slow computation	Profiler, flame graph
   Memory-bound	High memory, GC pauses, swapping	Heap profiler, memory tracker
   I/O-bound	Low CPU%, waiting on disk/network	Tracing, query logs
   Concurrency	Lock contention, thread starvation	Thread profiler, lock analysis

   Key insight: Most “slow” applications are I/O-bound, not CPU-bound. Don’t optimize CPU when you’re waiting on the database.

Phase 3: Algorithmic Complexity Analysis

Before micro-optimizing, check the algorithm:

1. Big-O Review

   • What is the time complexity of the hot path?
   • Is there an O(n^2) hiding in a loop?
   • Are you doing O(n) lookups where O(1) is possible (array vs. hash map)?
   • Nested database queries (N+1 problem)?

2. Common Hidden Complexity

3. Database Query Analysis

   -- Always EXPLAIN before optimizing
   EXPLAIN ANALYZE
   SELECT u.*, COUNT(o.id)
   FROM users u
   LEFT JOIN orders o ON o.user_id = u.id
   GROUP BY u.id;
   

   • Check for sequential scans on large tables
   • Verify indexes are being used
   • Look for unnecessary joins
   • Detect N+1 queries in ORM-generated SQL

4. N+1 Query Detection

Phase 4: Targeted Optimization with Benchmarks

Optimize only what profiling identified:

1. Write a Benchmark First

   // Before optimizing, capture current performance
   bench('current implementation', () => {
     currentFunction(testData);
   });
   
   bench('optimized implementation', () => {
     optimizedFunction(testData);
   });
   

2. Make ONE Change at a Time

   • Single optimization per iteration
   • Measure after each change
   • If no measurable improvement, revert
   • Don’t stack optimizations without measuring each

3. Verify Against Baseline

   • Does it meet the target defined in Phase 1?
   • Yes? Stop optimizing. You’re done.
   • No? Return to Phase 2, find next bottleneck.
   • Getting diminishing returns? Stop. Good enough is good enough.

4. Frontend-Specific Optimization

   • Core Web Vitals: LCP, FID/INP, CLS
   • Bundle size analysis (webpack-bundle-analyzer, source-map-explorer)
   • Lazy loading for below-fold content
   • Image optimization (WebP/AVIF, responsive images, lazy load)
   • Code splitting at route boundaries

5. Memory Leak Resolution

   • Take heap snapshots at intervals
   • Compare snapshots: what grows?
   • Common causes: event listeners not removed, closures holding references, growing caches without eviction
   • Fix the leak, verify with another snapshot series

Load Testing

Before any production deployment of performance-sensitive changes:

# k6 example
k6 run --vus 50 --duration 5m load-test.js

# Artillery example
artillery run load-test.yml

• Test at expected load AND 2-3x expected load
• Monitor error rates under load
• Check for degradation patterns (gradual vs. cliff)
• Verify resource consumption stays bounded

Red Flags – STOP and Follow Process

If you catch yourself thinking:

• “This is obviously the slow part”
• “Let me just add some caching”
• “Rewriting in Rust/Go will fix it”
• “Let me optimize this loop”
• “Add an index, that’ll fix it”
• “Just increase the timeout”
• “Premature optimization is bad, so I won’t measure”
• “The profiler is too hard to set up”
• “I’ll measure after I optimize”

ALL of these mean: STOP. Return to Phase 1.

Common Rationalizations

Anti-Patterns

Quick Reference

Verification Checklist

Before marking performance work complete:

• Performance target defined with concrete numbers
• Baseline measured and recorded before any changes
• Profiling data collected (not just intuition)
• Bottleneck identified from profiling data
• Algorithmic complexity reviewed
• Database queries analyzed with EXPLAIN
• Optimization benchmarked against baseline
• Performance target met (or documented why not)
• Load tested under realistic conditions
• No regressions in correctness (all tests pass)
• Memory stable under sustained load (no leaks)

Can’t check all boxes? You’re not done.

Integration with Other Skills

This skill requires using:

• systematic-debugging – REQUIRED when performance issue has an unclear cause (use Phase 1 root cause investigation)
• test-driven-development – REQUIRED for writing benchmarks and ensuring optimizations don’t break correctness

Complementary skills:

• documentation-generation – Document baseline metrics, optimization decisions, and load test results

Final Rule

No profiling data → no optimization
No baseline → no comparison
No target → no "done"

Measure. Profile. Optimize. Verify. In that order. Always.