Python Community Python >=3.11 Best Practices

Comprehensive performance optimization guide for Python >=3.11 applications. Contains 45 rules across 9 categories, prioritized by impact to guide automated refactoring and code generation.

When to Apply

Reference these guidelines when:

• Writing new Python async/await code
• Processing large datasets or files
• Implementing caching and memoization
• Choosing data structures for performance
• Reviewing code for performance issues

Rule Categories by Priority

Quick Reference

1. I/O Patterns (CRITICAL)

• io-async-file-operations – Use async file I/O for non-blocking operations
• io-batch-database-operations – Batch database operations to reduce round trips
• io-connection-pooling – Use connection pooling for database and HTTP clients
• io-streaming-large-files – Stream large files instead of loading into memory
• io-use-buffered-io – Use buffered I/O for frequent small writes

2. Async Concurrency (CRITICAL)

• async-avoid-async-overhead – Avoid async overhead for CPU-bound work
• async-avoid-blocking-calls – Avoid blocking calls in async code
• async-create-task-fire-forget – Store references to fire-and-forget tasks
• async-gather-independent-operations – Use asyncio.gather() for independent operations
• async-semaphore-rate-limiting – Use semaphores for concurrency limiting
• async-taskgroup-structured-concurrency – Use TaskGroup for structured concurrency

3. Memory Management (HIGH)

• mem-avoid-intermediate-lists – Avoid intermediate lists in pipelines
• mem-generators-lazy-evaluation – Use generators for lazy evaluation
• mem-preallocate-lists – Preallocate lists when size is known
• mem-slots-dataclass – Use slots for memory-efficient classes
• mem-string-interning – Leverage string interning for repeated strings
• mem-weak-references – Use weak references for caches and observers

4. Data Structures (HIGH)

• ds-counter-for-counting – Use Counter for frequency counting
• ds-deque-for-queues – Use deque for O(1) queue operations
• ds-dict-get-default – Use dict.get() with default instead of KeyError handling
• ds-namedtuple-immutable-records – Use NamedTuple for immutable lightweight records
• ds-set-for-membership – Use Set for O(1) membership testing

5. Algorithm Efficiency (MEDIUM-HIGH)

• algo-avoid-repeated-computation – Cache expensive computations in loops
• algo-builtin-functions – Use built-in functions over manual implementation
• algo-itertools-recipes – Use itertools for efficient iteration patterns
• algo-list-comprehension – Use list comprehensions over manual loops
• algo-local-variable-lookup – Use local variables in hot loops
• algo-string-join – Use str.join() for string concatenation

6. Concurrency Model (MEDIUM)

• conc-asyncio-queues – Use asyncio.Queue for producer-consumer patterns
• conc-avoid-lock-contention – Minimize lock contention in threaded code
• conc-choose-right-model – Choose the right concurrency model
• conc-process-pool-chunking – Use chunking for ProcessPoolExecutor
• conc-thread-safe-globals – Use thread-safe data structures for shared state

7. Serialization (MEDIUM)

• serial-avoid-pickle-security – Avoid pickle for untrusted data
• serial-msgpack-binary – Use MessagePack for compact binary serialization
• serial-orjson-over-json – Use orjson for high-performance JSON
• serial-pydantic-validation – Use Pydantic for validated deserialization

8. Caching and Memoization (LOW-MEDIUM)

• cache-avoid-over-caching – Avoid over-caching low-value operations
• cache-cached-property – Use cached_property for expensive computed attributes
• cache-lru-cache-decorator – Use lru_cache for expensive pure functions
• cache-ttl-expiration – Implement TTL for time-sensitive caches

9. Runtime Tuning (LOW)

• runtime-avoid-global-lookups – Avoid repeated global and module lookups
• runtime-exception-handling-cost – Minimize exception handling in hot paths
• runtime-profile-before-optimizing – Profile before optimizing
• runtime-use-python311-plus – Upgrade to Python 3.11+ for free performance

How to Use

Read individual reference files for detailed explanations and code examples:

• Section definitions – Category structure and impact levels
• Rule template – Template for adding new rules

Each rule file contains:

• Brief explanation of why it matters
• Incorrect code example with explanation
• Correct code example with explanation
• Additional context and references

Reference Files