doc-tspec-autopilot

Purpose

Automated Test Specifications (TSPEC) generation pipeline that processes SPEC documents to generate comprehensive test specifications for UTEST, ITEST, STEST, and FTEST with TASKS-Ready scoring.

Layer: 10

Upstream: SPEC (Layer 9)

Downstream: TASKS (Layer 11)

Skill Dependencies

Smart Document Detection

The autopilot automatically determines the action based on the input document type.

Input Type Recognition

Detection Algorithm

1. Parse input: Extract TYPE and NN from "{TYPE}-{NN}"
2. Determine action:
   - IF TYPE == "TSPEC": Review Mode
   - ELSE IF TYPE == "SPEC": Generate/Find Mode
   - ELSE: Error (invalid type for this autopilot)
3. For Generate/Find Mode:
   - Check: Does TSPEC-{NN} exist in docs/10_TSPEC/?
   - IF exists: Switch to Review Mode for TSPEC-{NN}
   - ELSE: Proceed with Generation from SPEC-{NN}

File Existence Check

# Check for nested folder structure (mandatory)
ls docs/10_TSPEC/TSPEC-{NN}_*/

Examples

# Review mode (same type - TSPEC input)
/doc-tspec-autopilot TSPEC-01        # Reviews existing TSPEC-01

# Generate/Find mode (upstream type - SPEC input)
/doc-tspec-autopilot SPEC-01         # Generates TSPEC-01 if missing, or reviews existing TSPEC-01

# Multiple inputs
/doc-tspec-autopilot SPEC-01,SPEC-02 # Generates/reviews TSPEC-01 and TSPEC-02
/doc-tspec-autopilot TSPEC-01,TSPEC-02 # Reviews TSPEC-01 and TSPEC-02

Action Determination Output

Input: SPEC-01
├── Detected Type: SPEC (upstream)
├── Expected TSPEC: TSPEC-01
├── TSPEC Exists: Yes → docs/10_TSPEC/TSPEC-01_f1_iam/
└── Action: REVIEW MODE - Running doc-tspec-reviewer on TSPEC-01

Input: SPEC-05
├── Detected Type: SPEC (upstream)
├── Expected TSPEC: TSPEC-05
├── TSPEC Exists: No
└── Action: GENERATE MODE - Creating TSPEC-05 from SPEC-05

Input: TSPEC-03
├── Detected Type: TSPEC (self)
└── Action: REVIEW MODE - Running doc-tspec-reviewer on TSPEC-03

Workflow Overview

flowchart TD
    subgraph Phase1["Phase 1: SPEC Analysis"]
        A[Start] --> B[Read SPEC Documents]
        B --> C[Extract Methods]
        C --> D[Extract Interfaces]
        D --> E[Extract Data Models]
        E --> F[Catalog Test Targets]
    end

    subgraph Phase2["Phase 2: Test Coverage Planning"]
        F --> G[Check SPEC TASKS-Ready Score]
        G --> H{Score >= 90%?}
        H -->|No| I[Flag SPEC Issues]
        I --> J{Auto-Fixable?}
        J -->|Yes| K[Fix SPEC Issues]
        K --> G
        J -->|No| L[Abort - Manual Fix Required]
        H -->|Yes| M[Plan Test Coverage]
        M --> N[Allocate Test Types]
    end

    subgraph Phase3["Phase 3: TSPEC Generation"]
        N --> O[Generate UTEST Specs]
        O --> P[Generate ITEST Specs]
        P --> Q[Generate STEST Specs]
        Q --> R[Generate FTEST Specs]
        R --> S[quality-advisor: Real-time Feedback]
        S --> T[Build Coverage Matrix]
        T --> U[Add Traceability Tags]
        U --> V[Write TSPEC Files]
    end

    subgraph Phase4["Phase 4: TSPEC Validation"]
        V --> W[Run doc-tspec-validator]
        W --> X{TASKS-Ready >= 90%?}
        X -->|No| Y[Auto-Fix TSPEC Issues]
        Y --> Z[Re-validate TSPEC]
        Z --> X
        X -->|Yes| AA[Mark TSPEC Validated]
    end

    subgraph Phase5["Phase 5: Review & Fix Cycle"]
        AA --> AB[Run doc-tspec-reviewer]
        AB --> AB2{Score >= 90?}
        AB2 -->|No| AB3[Run doc-tspec-fixer]
        AB3 --> AB4{Iteration < Max?}
        AB4 -->|Yes| AB
        AB4 -->|No| AB5[Flag Manual Review]
        AB2 -->|Yes| AC[Verify Quality Checks]
        AB5 --> AC
        AC --> AD[Update Traceability Matrix]
        AD --> AE[Generate Summary Report]
    end

    AE --> AF[Complete]
    L --> AG[Exit with Error]

Test Types

TSPEC Structure

All TSPEC use nested folders (TSPEC-NN_{slug}/). This keeps test specification files and companion files organized together.

docs/10_TSPEC/
├── TSPEC-01_authentication/
│   ├── TSPEC-01.0_index.md              # Suite index
│   ├── TSPEC-01.1_utest.md              # Unit tests
│   ├── TSPEC-01.2_itest.md              # Integration tests
│   ├── TSPEC-01.3_stest.md              # Smoke tests
│   ├── TSPEC-01.4_ftest.md              # Functional tests
│   ├── TSPEC-01.R_review_report_v001.md # Review report
│   ├── TSPEC-01.F_fix_report_v001.md    # Fix report
│   └── .drift_cache.json                 # Drift cache
└── TSPEC-01_authentication.md            # Redirect stub (optional)

Coverage Matrix Format

Element ID Format

Phase 5: Review & Fix Cycle (v2.3)

Iterative review and fix cycle to ensure TSPEC quality before completion.

flowchart TD
    A[Phase 5 Start] --> B[Run doc-tspec-reviewer]
    B --> C[Generate Review Report]
    C --> D{Review Score >= 90?}

    D -->|Yes| E[PASS - Proceed to Phase 6]
    D -->|No| F{Iteration < Max?}

    F -->|Yes| G[Run doc-tspec-fixer]
    G --> H[Apply Fixes]
    H --> I[Generate Fix Report]
    I --> J[Increment Iteration]
    J --> B

    F -->|No| K[Flag for Manual Review]
    K --> L[Generate Final Report with Remaining Issues]
    L --> E

5.1 Initial Review

Run doc-tspec-reviewer to identify issues.

/doc-tspec-reviewer TSPEC-NN

Output: TSPEC-NN.R_review_report_v001.md

5.2 Fix Cycle

If review score < 90%, invoke doc-tspec-fixer.

/doc-tspec-fixer TSPEC-NN --revalidate

Fix Categories:

Output: TSPEC-NN.F_fix_report_v001.md

5.3 Re-Review

After fixes, automatically re-run reviewer.

/doc-tspec-reviewer TSPEC-NN

Output: TSPEC-NN.R_review_report_v002.md

5.4 Iteration Control

Iteration Example:

Iteration 1:
  Review v001: Score 78 (2 errors, 6 warnings)
  Fix v001: Fixed 5 issues, created 1 test file

Iteration 2:
  Review v002: Score 91 (0 errors, 2 warnings)
  Status: PASS (score >= 90)

5.5 Quality Checks (Post-Fix)

After passing the fix cycle:

1. Test Type Completeness:

   • All 4 test types present (UTEST, ITEST, STEST, FTEST)
   • Each test type file has required sections
   • No placeholder text remaining

2. Coverage Matrix Accuracy:

   • All SPEC elements have test coverage
   • Coverage percentages calculated correctly
   • Target coverage met per test type

3. Element ID Compliance (per doc-naming skill):

   • All IDs use TSPEC.NN.TT.SS format
   • Element type codes valid for TSPEC (40, 41, 42, 43)
   • No legacy patterns (UT-XXX, IT-XXX, ST-XXX, FT-XXX)

4. TASKS-Ready Report:

   TASKS-Ready Score Breakdown
   ===========================
   Test Type Completeness:  25/25 (all 4 types present)
   Coverage Matrix:         18/20 (coverage targets met)
   SPEC Alignment:          20/20 (tests trace to SPEC)
   Element ID Format:       15/15 (valid format)
   Traceability Tags:       10/10 (8 required tags)
   Test Assertions:         8/10 (assertions present)
   ----------------------------
   Total TASKS-Ready Score: 96/100 (Target: >= 90)
   Status: READY FOR TASKS GENERATION
   

5. Traceability Matrix Update:

   # Update TSPEC traceability
   python ai_dev_flow/scripts/update_traceability_matrix.py \
     --tspec docs/10_TSPEC/TSPEC-NN_{slug}/ \
     --matrix docs/10_TSPEC/TSPEC-00_TRACEABILITY_MATRIX.md
   

Cumulative Tags (8 Required)

@brd: BRD.NN.TT.SS
@prd: PRD.NN.TT.SS
@ears: EARS.NN.TT.SS
@bdd: BDD.NN.TT.SS
@adr: ADR-NN
@sys: SYS.NN.TT.SS
@req: REQ.NN.TT.SS
@spec: SPEC.NN.TT.SS
@ctr: CTR.NN.TT.SS  # Optional

Configuration

Default Configuration

tspec_autopilot:
  version: "1.0"

  scoring:
    tasks_ready_min: 90
    strict_mode: false

  execution:
    max_parallel: 3        # HARD LIMIT - do not exceed
    chunk_size: 3          # Documents per chunk
    pause_between_chunks: true
    auto_fix: true
    continue_on_error: false
    timeout_per_spec: 180  # seconds

  output:
    structure: sectioned  # 4 test type files per TSPEC
    report_format: markdown

  validation:
    skip_validation: false
    fix_iterations_max: 3

  test_types:
    utest: true
    itest: true
    stest: true
    ftest: true

Execution Modes

Mode 1: Generate Mode (Default)

Standard TSPEC generation from SPEC documents (see Workflow Overview above).

Mode 2: Review Mode (v2.1)

Validate existing TSPEC documents without modification. Generates quality report with actionable recommendations.

Command:

# Review single TSPEC
/doc-tspec-autopilot TSPEC-01 --review

# Review all TSPEC in directory
/doc-tspec-autopilot docs/10_TSPEC/ --review --all

# Review with detailed report
/doc-tspec-autopilot TSPEC-01 --review --verbose

Review Process:

flowchart TD
    A[Input: TSPEC Document] --> B[Load TSPEC Content]
    B --> C[Validate Test Structure]
    C --> D[Check TASKS-Ready Score]
    D --> E[Verify All 4 Test Types]
    E --> F[Validate Coverage Matrix]
    F --> G[Check Element IDs]
    G --> H[Verify Cumulative Tags]
    H --> I[Generate Review Report]
    I --> J{Issues Found?}
    J -->|Yes| K[Categorize Issues]
    K --> L[Generate Fix Recommendations]
    L --> M[Output Review Report]
    J -->|No| M

Review Report Template:

# TSPEC Review Report: TSPEC-NN_{slug}

## Summary
- **TASKS-Ready Score**: NN% (✅/🟡/❌)
- **Total Issues**: N (E errors, W warnings)
- **Auto-Fixable**: N issues
- **Manual Review**: N issues

## Test Type Coverage
| Type | File | Tests | Coverage | Status |
|------|------|-------|----------|--------|
| UTEST | TSPEC-NN.1_utest.md | N | NN% | ✅/🟡/❌ |
| ITEST | TSPEC-NN.2_itest.md | N | NN% | ✅/🟡/❌ |
| STEST | TSPEC-NN.3_stest.md | N | NN% | ✅/🟡/❌ |
| FTEST | TSPEC-NN.4_ftest.md | N | NN% | ✅/🟡/❌ |

## Score Breakdown
| Category | Score | Max | Status |
|----------|-------|-----|--------|
| Test Type Completeness | NN | 25 | ✅/🟡/❌ |
| Coverage Matrix | NN | 20 | ✅/🟡/❌ |
| SPEC Alignment | NN | 20 | ✅/🟡/❌ |
| Element ID Format | NN | 15 | ✅/🟡/❌ |
| Traceability Tags | NN | 10 | ✅/🟡/❌ |
| Test Assertions | NN | 10 | ✅/🟡/❌ |

## Issues by Category

### Auto-Fixable Issues
| Issue | Location | Fix Action |
|-------|----------|------------|
| Legacy ID pattern | Line 45 | Convert UT-001 → TSPEC.01.40.01 |
| Missing cumulative tag | Traceability | Add @req: REQ.01.01.01 |

### Manual Review Required
| Issue | Location | Recommendation |
|-------|----------|----------------|
| Missing STEST file | TSPEC-01.3 | Create smoke test specifications |
| Low coverage | SPEC.01.28.05 | Add unit tests for method |

Score Indicators:

• ✅ Green (>=90%): TASKS-Ready
• 🟡 Yellow (70-89%): Needs improvement
• ❌ Red (<70%): Significant issues

Review Configuration:

review_mode:
  enabled: true
  checks:
    - test_structure      # All 4 test type files present
    - coverage_matrix     # Coverage >= target per SPEC element
    - spec_alignment      # Tests trace to SPEC methods
    - element_ids         # TSPEC.NN.40-43.SS format
    - cumulative_tags     # 8 required tags present
    - test_assertions     # Each test has clear assertions
  output:
    format: markdown      # markdown, json, html
    include_recommendations: true
    include_fix_commands: true
  coverage_targets:
    utest: 95            # Unit test coverage target %
    itest: 85            # Integration test coverage target %
    stest: 100           # Smoke test must cover critical path
    ftest: 75            # Functional test coverage target %

Mode 3: Fix Mode (v2.1)

Auto-repair existing TSPEC documents with backup and content preservation.

Command:

# Fix single TSPEC
/doc-tspec-autopilot TSPEC-01 --fix

# Fix with backup
/doc-tspec-autopilot TSPEC-01 --fix --backup

# Fix all TSPEC
/doc-tspec-autopilot docs/10_TSPEC/ --fix --all

# Fix specific categories only
/doc-tspec-autopilot TSPEC-01 --fix --only element_ids,tags

# Dry-run fix (preview changes)
/doc-tspec-autopilot TSPEC-01 --fix --dry-run

Fix Process:

flowchart TD
    A[Input: TSPEC Document] --> B[Create Backup]
    B --> C[Load TSPEC Content]
    C --> D[Run Review Analysis]
    D --> E{Auto-Fixable Issues?}
    E -->|No| F[Report: No Fixes Needed]
    E -->|Yes| G[Apply Fixes by Category]

    subgraph FixCategories["Fix Categories"]
        G --> H[Fix Element IDs]
        H --> I[Fix Test Type Structure]
        I --> J[Regenerate Coverage Matrix]
        J --> K[Add Missing Tags]
        K --> L[Fix Section Structure]
        L --> M[Add Missing Test Stubs]
    end

    M --> N[Validate Fixed Document]
    N --> O{Validation Passed?}
    O -->|No| P[Rollback to Backup]
    P --> Q[Report: Fix Failed]
    O -->|Yes| R[Save Fixed Document]
    R --> S[Generate Fix Report]

TSPEC-Specific Fix Categories:

Element ID Migration (Layer 10):

Content Preservation Rules:

Fix Configuration:

fix_mode:
  enabled: true
  backup:
    enabled: true
    location: "tmp/backups/"
    timestamp: true
  fix_categories:
    element_ids: true       # Convert legacy ID patterns
    test_types: true        # Create missing test type files
    coverage_matrix: true   # Regenerate coverage tracking
    cumulative_tags: true   # Add 8 required tags
    sections: true          # Add missing sections
    test_stubs: true        # Generate stubs for uncovered SPEC
    assertions: false       # Manual only (flag but don't auto-fix)
  validation:
    post_fix: true          # Validate after fixes
    rollback_on_fail: true  # Restore backup if validation fails
  preserve:
    test_descriptions: true
    test_assertions: true
    test_data: true
    spec_references: true

Fix Report Template:

# TSPEC Fix Report: TSPEC-NN_{slug}

## Summary
- **Backup Created**: tmp/backups/TSPEC-NN_{slug}_20260209_143022/
- **Issues Fixed**: N of M auto-fixable issues
- **Manual Review**: N issues flagged

## Fixes Applied

### Element ID Migration
| Original | Fixed | File | Location |
|----------|-------|------|----------|
| UT-001 | TSPEC.01.40.01 | TSPEC-01.1_utest.md | Line 45 |
| IT-001 | TSPEC.01.41.01 | TSPEC-01.2_itest.md | Line 23 |
| ST-001 | TSPEC.01.42.01 | TSPEC-01.3_stest.md | Line 12 |

### Test Type Files Created
| File | Tests Generated | Status |
|------|-----------------|--------|
| TSPEC-01.3_stest.md | 5 smoke tests | Created from template |

### Coverage Matrix Regenerated
| SPEC Element | Before | After |
|--------------|--------|-------|
| SPEC.01.28.01 | 50% | 75% |
| SPEC.01.28.02 | 25% | 75% |
| Overall | 45% | 78% |

### Cumulative Tags Added
- @ears: EARS.01.25.001 (added)
- @sys: SYS.01.01.01 (added)
- @req: REQ.01.01.01 (added)

## Test Stubs Generated
| SPEC Element | Test Type | Stub ID |
|--------------|-----------|---------|
| SPEC.01.28.05 | UTEST | TSPEC.01.40.12 |
| SPEC.01.28.05 | ITEST | TSPEC.01.41.08 |

## Manual Review Required

### Tests Without Assertions
| Test ID | File | Issue |
|---------|------|-------|
| TSPEC.01.40.03 | TSPEC-01.1_utest.md | No assert statements |
| TSPEC.01.41.02 | TSPEC-01.2_itest.md | Missing expected outcome |

### Low Coverage Areas
| SPEC Element | Current | Target | Action Needed |
|--------------|---------|--------|---------------|
| SPEC.01.28.07 | 25% | 75% | Add 3+ UTEST, 1+ ITEST |

## Validation Results
- **TASKS-Ready Score**: Before: 68% → After: 92%
- **Validation Errors**: Before: 12 → After: 0
- **Status**: ✅ All auto-fixes validated

Command Line Options (Review/Fix Modes):

Context Management

Chunked Parallel Execution (MANDATORY)

CRITICAL: To prevent conversation context overflow errors (“Prompt is too long”, “Conversation too long”), all autopilot operations MUST follow chunked execution rules:

Chunk Size Limit: Maximum 3 documents per chunk

Chunking Rules:

1. Chunk Formation: Group SPEC-derived TSPEC documents into chunks of maximum 3 at a time
2. Sequential Chunk Processing: Process one chunk at a time, completing all documents in a chunk before starting the next
3. Context Pause: After completing each chunk, provide a summary and pause for user acknowledgment
4. Progress Tracking: Display chunk progress (e.g., “Chunk 2/4: Processing TSPEC-04, TSPEC-05, TSPEC-06…”)

Why Chunking is Required:

• Prevents “Conversation too long” errors during batch processing
• Allows context compaction between chunks
• Enables recovery from failures without losing all progress
• Provides natural checkpoints for user review

Chunk Completion Template:

## Chunk N/M Complete

Generated:
- TSPEC-XX: TASKS-Ready Score 94% (UTEST + ITEST + STEST + FTEST)
- TSPEC-YY: TASKS-Ready Score 92% (UTEST + ITEST + STEST + FTEST)
- TSPEC-ZZ: TASKS-Ready Score 95% (UTEST + ITEST + STEST + FTEST)

Proceeding to next chunk...

Related Resources

• TSPEC Skill: .claude/skills/doc-tspec/SKILL.md
• TSPEC Validator: .claude/skills/doc-tspec-validator/SKILL.md
• Naming Standards: .claude/skills/doc-naming/SKILL.md
• Quality Advisor: .claude/skills/quality-advisor/SKILL.md
• TSPEC Template: ai_dev_flow/10_TSPEC/TSPEC-TEMPLATE.md

Review Document Standards (v2.2)

Review reports generated by this skill are formal project documents and MUST comply with shared standards.

Reference: See REVIEW_DOCUMENT_STANDARDS.md in the skills directory for complete requirements.

Key Requirements:

1. Storage Location: Same folder as the reviewed TSPEC document
2. File Naming: TSPEC-NN.R_review_report.md
3. YAML Frontmatter: Required with artifact_type: TSPEC-REVIEW, layer: 10
4. Score Field: impl_ready_score_claimed / impl_ready_score_validated
5. Parent Reference: Must link to parent TSPEC document

Example Location (ALWAYS use nested folders):

docs/10_TSPEC/TSPEC-03_f3_observability/
├── TSPEC-03_f3_observability.md         # ← Main document
├── TSPEC-03.R_review_report_v001.md     # ← Review report
├── TSPEC-03.F_fix_report_v001.md        # ← Fix report
└── .drift_cache.json                     # ← Drift cache

Nested Folder Rule: ALL TSPEC use nested folders (TSPEC-NN_{slug}/) regardless of size. This keeps companion files (review reports, fix reports, drift cache) organized with their parent document.

Version History