Python Performance Optimization

Expert guidance for profiling, optimizing, and accelerating Python applications through systematic analysis, algorithmic improvements, efficient data structures, and acceleration techniques.

When to Use This Skill

• Code runs too slowly for production requirements
• High CPU usage or memory consumption issues
• Need to reduce API response times or batch processing duration
• Application fails to scale under load
• Optimizing data processing pipelines or scientific computing
• Reducing cloud infrastructure costs through efficiency gains
• Profile-guided optimization after measuring performance bottlenecks

Core Concepts

The Golden Rule: Never optimize without profiling first. 80% of execution time is spent in 20% of code.

Optimization Hierarchy (in priority order):

1. Algorithm complexity – O(n²) → O(n log n) provides exponential gains
2. Data structure choice – List → Set for lookups (10,000x faster)
3. Language features – Comprehensions, built-ins, generators
4. Caching – Memoization for repeated calculations
5. Compiled extensions – NumPy, Numba, Cython for hot paths
6. Parallelism – Multiprocessing for CPU-bound work

Key Principle: Algorithmic improvements beat micro-optimizations every time.

Quick Reference

Load detailed guides for specific optimization areas:

Optimization Workflow

Phase 1: Measure

1. Profile with cProfile – Identify slow functions
2. Line profile hot paths – Find exact slow lines
3. Memory profile – Check for memory bottlenecks
4. Benchmark baseline – Record current performance

Phase 2: Analyze

1. Check algorithm complexity – Is it O(n²) or worse?
2. Evaluate data structures – Are you using lists for lookups?
3. Identify repeated work – Can results be cached?
4. Find I/O bottlenecks – Database queries, file operations

Phase 3: Optimize

1. Improve algorithms first – Biggest impact
2. Use appropriate data structures – Set/dict for O(1) lookups
3. Apply caching – @lru_cache for expensive functions
4. Use generators – For large datasets
5. Leverage NumPy/Numba – For numerical code
6. Parallelize – Multiprocessing for CPU-bound tasks

Phase 4: Validate

1. Re-profile – Verify improvements
2. Benchmark – Measure speedup quantitatively
3. Test correctness – Ensure optimizations didn’t break functionality
4. Document – Explain why optimization was needed

Common Optimization Patterns

Pattern 1: Replace List with Set for Lookups

# Slow: O(n) lookup
if item in large_list:  # Bad

# Fast: O(1) lookup
if item in large_set:   # Good

Pattern 2: Use Comprehensions

# Slower
result = []
for i in range(n):
    result.append(i * 2)

# Faster (35% speedup)
result = [i * 2 for i in range(n)]

Pattern 3: Cache Expensive Calculations

from functools import lru_cache

@lru_cache(maxsize=None)
def expensive_function(n):
    # Result cached automatically
    return complex_calculation(n)

Pattern 4: Use Generators for Large Data

# Memory inefficient
def read_file(path):
    return [line for line in open(path)]  # Loads entire file

# Memory efficient
def read_file(path):
    for line in open(path):  # Streams line by line
        yield line.strip()

Pattern 5: Vectorize with NumPy

# Pure Python: ~500ms
result = sum(i**2 for i in range(1000000))

# NumPy: ~5ms (100x faster)
import numpy as np
result = np.sum(np.arange(1000000)**2)

Common Mistakes to Avoid

1. Optimizing before profiling – You’ll optimize the wrong code
2. Using lists for membership tests – Use sets/dicts instead
3. String concatenation in loops – Use "".join() or StringIO
4. Loading entire files into memory – Use generators
5. N+1 database queries – Use JOINs or batch queries
6. Ignoring built-in functions – They’re C-optimized and fast
7. Premature optimization – Focus on algorithmic improvements first
8. Not benchmarking – Always measure improvements quantitatively

Decision Tree

Start here: Profile with cProfile to find bottlenecks

Hot path is algorithm?

• Yes → Check complexity, improve algorithm, use better data structures
• No → Continue

Hot path is computation?

• Numerical loops → NumPy or Numba
• CPU-bound → Multiprocessing
• Already fast enough → Done

Hot path is memory?

• Large data → Generators, streaming
• Many objects → __slots__, object pooling
• Caching needed → @lru_cache or custom cache

Hot path is I/O?

• Database → Batch queries, indexes, connection pooling
• Files → Buffering, streaming
• Network → Async I/O, request batching

Best Practices

1. Profile before optimizing – Measure to find real bottlenecks
2. Optimize algorithms first – O(n²) → O(n) beats micro-optimizations
3. Use appropriate data structures – Set/dict for lookups, not lists
4. Leverage built-ins – C-implemented built-ins are faster than pure Python
5. Avoid premature optimization – Optimize hot paths identified by profiling
6. Use generators for large data – Reduce memory usage with lazy evaluation
7. Batch operations – Minimize overhead from syscalls and network requests
8. Cache expensive computations – Use @lru_cache or custom caching
9. Consider NumPy/Numba – Vectorization and JIT for numerical code
10. Parallelize CPU-bound work – Use multiprocessing to utilize all cores

Resources

• Python Performance: https://wiki.python.org/moin/PythonSpeed
• cProfile: https://docs.python.org/3/library/profile.html
• NumPy: https://numpy.org/doc/stable/user/absolute_beginners.html
• Numba: https://numba.pydata.org/
• Cython: https://cython.readthedocs.io/
• High Performance Python (Book by Gorelick & Ozsvald)