Scientific Validation Skill

Rigorous methodology for validating claims from any source – books, papers, theories, or intuition.

When This Skill Activates

• Testing claims from books, papers, or expert sources
• Validating rules, strategies, or hypotheses
• Running experiments or backtests
• Keywords: “validate”, “test hypothesis”, “experiment”, “backtest”, “prove”, “evidence”

Core Principle

Data is the arbiter. Sources can be wrong.

• Expert books can be wrong
• Only empirical validation decides what works
• Document negative results – they’re valuable

Phase Overview

See: workflow.md for detailed step-by-step instructions per phase.

Quick Reference

Claim Types

Sample Size Requirements (80% Power)

See: code-examples.md#power-analysis for calculation code.

Classification Criteria

Domain Effect Thresholds (Trading)

See: code-examples.md#effect-thresholds for other domains.

Bayes Factor Interpretation

Critical Rules

1. Pre-Registration

• Document hypothesis BEFORE seeing any results
• Define success criteria BEFORE testing
• No peeking at test data

2. Power Analysis (Phase 2.3)

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.5, power=0.80, alpha=0.05)

Rule: Underpowered studies cannot achieve VALIDATED status.

3. Walk-Forward for Time Series (Phase 3.5)

• Standard K-fold CV → INVALID (temporal leakage)
• Single train/test → CONDITIONAL at best
• Walk-forward → Can achieve VALIDATED

See: code-examples.md#walk-forward for implementation.

4. Multiple Comparison Correction

alpha_corrected = 0.05 / num_claims  # Bonferroni

For trading claims: require t-ratio > 3.0 (Harvey et al. standard).

5. Sensitivity Analysis (Phase 5.3)

Test ±20% parameter variation:

• All variations positive → Can achieve VALIDATED
• 1-2 sign flips → CONDITIONAL at best
• 3+ sign flips → REJECTED (fragile)

See: code-examples.md#sensitivity-analysis for implementation.

6. Adversarial Review (Phase 5.5)

Use Task tool:
  subagent_type: "experiment-critic"
  prompt: "Review experiment EXP-XXX"

MANDATORY before any classification.

Bias Prevention Checklist

Pre-Experiment Checklist

• Claim extracted with source citation
• ALL claims documented (not just tested ones)
• Hypothesis documented BEFORE results
• Power analysis: required n calculated
• Success criteria defined
• Walk-forward configured (time series)
• Costs/constraints specified

Post-Experiment Checklist

• Sample size adequate per power analysis
• p-value AND effect size reported
• Bayesian analysis if ambiguous
• Sensitivity analysis passed
• Adversarial review completed
• Negative results documented if REJECTED

Red Flags

• 100% success rate → Possible bias
• OOS better than training → Possible leakage
• Result flips with ±20% params → Fragile
• Only tested “interesting” claims → Selection bias

Key Principles

1. Hypothesis BEFORE data – No peeking
2. Power analysis BEFORE experiment – Know required n
3. Walk-forward for time series – Preserve temporal order
4. Sensitivity analysis – Results must survive ±20% changes
5. Adversarial self-critique – Challenge your methodology
6. Document negative results – Failures are valuable
7. Sources can be wrong – Even experts, even textbooks

Detailed Documentation

Hard Rules

FORBIDDEN:

• Reporting results without confidence intervals or statistical significance
• Cherry-picking favorable metrics while ignoring unfavorable ones
• Claiming causation from correlation without controlled experiments

REQUIRED:

• All experiments MUST have a documented hypothesis before execution
• All results MUST include sample size, variance, and statistical test used
• Negative results MUST be reported with the same rigor as positive results
• Baselines MUST be established and compared against for every metric