GrepAI Chunking Configuration

This skill covers how GrepAI splits code files into chunks for embedding, and how to optimize chunking for your codebase.

When to Use This Skill

• Optimizing search accuracy
• Adjusting for code style (verbose vs. concise)
• Troubleshooting search results
• Understanding how indexing works

What is Chunking?

Chunking is the process of splitting source files into smaller segments for embedding:

┌─────────────────────────────────────┐
│         Large Source File           │
│         (1000+ tokens)              │
└─────────────────────────────────────┘
                  ↓
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 │
│ ~512    │ │ ~512    │ │ ~512    │
│ tokens  │ │ tokens  │ │ tokens  │
└─────────┘ └─────────┘ └─────────┘
                  ↓
          Each chunk gets
          its own embedding

Why Chunking Matters

Embedding models have optimal input sizes:

• Too large chunks: Less precise search results
• Too small chunks: Lost context, fragmented results
• Just right: Good balance of precision and context

Configuration

Basic Settings

# .grepai/config.yaml
chunking:
  size: 512      # Tokens per chunk
  overlap: 50    # Overlap between chunks

Understanding Parameters

Chunk Size

The target number of tokens per chunk.

Overlap

Tokens shared between adjacent chunks. Preserves context at boundaries.

Visualization

With size=512 and overlap=50:

File: auth.go (1000 tokens)

Chunk 1: tokens 1-512
         ┌────────────────────────────────────┐
         │ func Login(user, pass)...          │
         └────────────────────────────────────┘
                                    ↘
                              50 token overlap
                                    ↙
Chunk 2: tokens 463-974
         ┌────────────────────────────────────┐
         │ ...validate credentials...         │
         └────────────────────────────────────┘
                                    ↘
                              50 token overlap
                                    ↙
Chunk 3: tokens 925-1000
         ┌──────────────┐
         │ ...return    │
         └──────────────┘

Recommended Settings by Language

Verbose Languages (Java, C#)

chunking:
  size: 768    # Larger to capture full methods
  overlap: 75

Concise Languages (Go, Python)

chunking:
  size: 512    # Standard size
  overlap: 50

Very Concise (Rust, Zig)

chunking:
  size: 384    # Smaller for precise results
  overlap: 40

Recommended Settings by Codebase

Small Functions (Microservices)

chunking:
  size: 384    # Capture individual functions
  overlap: 40

Large Classes (Monolith)

chunking:
  size: 768    # Capture more context
  overlap: 100

Mixed Codebase

chunking:
  size: 512    # Balanced default
  overlap: 50

How Tokens are Counted

GrepAI uses approximate token counting:

• ~4 characters = 1 token (for English text)
• Code varies based on identifiers and syntax

Example:

func calculateTotal(items []Item) float64 {
    total := 0.0
    for _, item := range items {
        total += item.Price * float64(item.Quantity)
    }
    return total
}

≈ 45 tokens

Impact on Index Size

Larger overlap = more chunks = larger index:

Impact on Search Quality

Too Small Chunks (size: 128)

Query: "authentication middleware"

Result: "...c.AbortWithStatus(401)..."
        (Fragment, missing context)

Just Right (size: 512)

Query: "authentication middleware"

Result: "func AuthMiddleware() gin.HandlerFunc {
            return func(c *gin.Context) {
                token := c.GetHeader("Authorization")
                if token == "" {
                    c.AbortWithStatus(401)
                    return
                }
                // validate token...
            }
        }"
        (Complete function with context)

Too Large Chunks (size: 2048)

Query: "authentication middleware"

Result: "// Multiple unrelated functions...
        func AuthMiddleware()... (your match)
        func LoggingMiddleware()...
        func CORSMiddleware()..."
        (Too much noise)

Experimentation

Testing Different Settings

1. Try smaller chunks for more precise results:

chunking:
  size: 384
  overlap: 40

2. Re-index:

rm .grepai/index.gob
grepai watch

3. Test with searches:

grepai search "your query"

4. Adjust and repeat until satisfied.

Comparing Results

Before changing settings, save a search result:

grepai search "authentication" > before.txt

After changing settings and re-indexing:

grepai search "authentication" > after.txt
diff before.txt after.txt

Chunk Boundaries

GrepAI tries to split at logical boundaries:

1. Empty lines (function/class boundaries)
2. Closing braces
3. Statement ends

This means actual chunk sizes may vary slightly from the target.

Best Practices

1. Start with defaults: 512/50 works well for most codebases
2. Adjust based on code style: Verbose = larger, concise = smaller
3. Test with real queries: See what your searches return
4. Re-index after changes: Must regenerate embeddings
5. Consider overlap: Don’t set to 0 unless index size is critical

Common Issues

❌ Problem: Search results are too fragmented ✅ Solution: Increase chunk size:

chunking:
  size: 768

❌ Problem: Search results have too much irrelevant context ✅ Solution: Decrease chunk size:

chunking:
  size: 384

❌ Problem: Results miss related code at function boundaries ✅ Solution: Increase overlap:

chunking:
  overlap: 100

❌ Problem: Index is too large ✅ Solutions:

• Decrease overlap
• Increase chunk size
• Add more ignore patterns

Output Format

Chunking status:

✅ Chunking Configuration

   Size: 512 tokens
   Overlap: 50 tokens

   Index Statistics:
   - Total files: 245
   - Total chunks: 1,234
   - Avg chunks/file: 5.0
   - Avg chunk size: 478 tokens

   Recommendations:
   - Current settings are balanced
   - Consider size: 384 for more precise results
   - Consider size: 768 for more context