Library Research & Evaluation

Purpose

Systematic library evaluation framework that produces readable, actionable decision briefs for library adoption, migration, and upgrade decisions using a 6-dimension research methodology.

Quick Start

Create readable, actionable research that drives informed decisions.

This skill is about research methodology – knowing what to look for, where to find it, how to synthesize it, and how to present it. The goal is NOT to dump data, but to accelerate time to adoption through clear, confident recommendations.

Most common use case:

User: "Should we use Pydantic v2?"

→ Research 6 dimensions (docs, updates, alternatives, security, community, codebase)
→ Score each with evidence
→ Deliver decision brief with clear ADOPT/MIGRATE/STAY/AVOID recommendation

Result: 40 minutes (vs 2-4 hours ad-hoc research)

Table of Contents

1. When to Use This Skill
2. What This Skill Does
3. Instructions
4. Research Framework (6 Dimensions)
5. Output Format: Decision Brief
6. Usage Examples
7. Expected Outcomes
8. Integration Points
9. Expected Benefits
10. Success Metrics
11. Requirements
12. Troubleshooting
13. Red Flags to Avoid

When to Use This Skill

Use this skill when:

• Adoption decisions: “Should we use library X?”
• Migration decisions: “Should we switch from X to Y?”
• Upgrade decisions: “Should we upgrade to version Z?”
• Security audits: “Are our dependencies safe?”
• Performance optimization: “Is there a faster alternative?”
• Technology evaluation: “What’s the best tool for this job?”

What This Skill Does

This skill provides systematic library evaluation through:

1. Research 6 dimensions – Official docs, recent updates, alternatives, security, community, codebase usage
2. Score with evidence – Each dimension rated 1-10 with specific supporting data
3. Synthesize decision – Clear ADOPT/MIGRATE/UPGRADE/STAY/AVOID recommendation
4. Deliver decision brief – Scannable report with TL;DR, decision matrix, action items
5. Time-boxed process – Complete research in ~40 minutes (vs 2-4 hours ad-hoc)

See Instructions section below for detailed step-by-step workflow.

Instructions

Phase 1: Gather (15 minutes)

Goal: Collect evidence across 6 research dimensions

Steps:

1. Context: Read pyproject.toml for existing libraries and Context7 IDs
2. Docs: Get official docs (check pyproject.toml first, then resolve-library-id)
3. Updates: WebSearch for recent changes, changelogs
4. Alternatives: WebSearch for comparisons, discussions
5. Security: WebSearch for CVEs, advisories
6. Community: WebSearch on reddit, HN, GitHub, StackOverflow
7. Codebase: Search current usage patterns

Phase 2: Synthesize (10 minutes)

Goal: Transform evidence into clear decision

Steps:

1. Score each dimension (1-10) with specific evidence
2. Identify decision driver (what matters most?)
3. Determine recommendation (ADOPT/MIGRATE/UPGRADE/STAY/AVOID)
4. Assess confidence (High/Medium/Low)
5. List action items (what happens next?)

Phase 3: Write (15 minutes)

Goal: Create scannable decision brief

Steps:

1. TL;DR: Write last, summarize the decision
2. Decision Matrix: Fill in scores and evidence
3. Recommendation: State decision and rationale
4. Action Items: Specific, assignable tasks
5. Supporting Evidence: Details for skeptics
6. References: Links for deep dive

Total time: ~40 minutes (vs 1-2 hours of ad-hoc research)

Core Principles

1. Readability First

• ❌ Walls of text, data dumps, exhaustive lists
• ✅ Scannable sections, bullet points, clear structure
• ✅ Executive summary fits in one screen
• ✅ Each section answers ONE question

2. Decision-Focused

• Every fact must support a decision
• If it doesn’t change the recommendation, cut it
• Lead with “SO WHAT?” not “WHAT?”
• Clear action items, not observations

3. Confidence Over Completeness

• Better to be 80% confident with 6 sources than 60% with 20 sources
• Quality of sources > quantity
• Synthesize, don’t concatenate

4. Time to Value

• Reader should know the recommendation in 30 seconds
• Supporting evidence in 3 minutes
• Full context in 10 minutes
• Everything else is noise

Usage Examples

Example 1: Adoption Decision

Scenario: User asks: “Should we use Pydantic v2 for data validation?”

Process:

1. Research 6 dimensions: docs, updates, alternatives, security, community, codebase
2. Score each dimension with evidence
3. Synthesize into decision brief

Outcome:

📋 TL;DR
Recommendation: ✅ ADOPT
Confidence: High (8/10)
Key insight: Production-ready, 20x faster, easy migration

🎯 Decision Matrix: 8.2/10 overall
🚀 Action Items: POC this week, migrate next month

Time: ~40 minutes (vs 2-4 hours ad-hoc research)

Example 2: Migration Decision

Scenario: User asks: “Should we migrate from Neo4j to Memgraph?”

Outcome:

📋 TL;DR
Recommendation: ❌ STAY
Confidence: High (8/10)
Key insight: Neo4j meets needs, migration cost high, unclear benefit

🎯 Decision Matrix:
| Dimension | Neo4j | Memgraph | Winner |
|-----------|-------|----------|--------|
| Functionality | 9/10 | 8/10 | Neo4j |
| Performance | 8/10 | 9/10 | Memgraph |
| Maturity | 10/10 | 6/10 | Neo4j |
| Migration Cost | N/A | 3/10 | Neo4j |

🚀 When to Reconsider: If query performance drops below SLA

Example 3: Security Audit

Scenario: User asks: “Audit our dependencies for security issues”

Outcome: Security report for each dependency with:

• CVE status (none found / issues identified)
• Latest version check
• Upgrade recommendations
• Priority ranking (critical/high/medium/low)

Research Framework (6 Dimensions)

1. Official Documentation (Context7)

Question: Does it do what we need?

• Check pyproject.toml [tool.context7] FIRST for existing library ID
• If not found, use resolve-library-id
• Focus search on specific use case (e.g., “async patterns”, “performance tuning”)
• Extract: capabilities, limitations, best practices

2. Recent Developments (WebSearch)

Question: Is it actively maintained and improving?

• Last 6 months of updates (use current year from env)
• Changelog, release notes, roadmap
• Breaking changes, deprecations
• Extract: momentum, stability, future-proofing

3. Alternatives (WebSearch)

Question: Is this the best tool for the job?

Sources:

• General: best {category} 2025 comparison benchmark
• Head-to-head: {lib} vs {alternative} 2025 (db-engines.com, github.com)
• Community: {lib} vs {alternative} discussion (reddit.com, news.ycombinator.com)

Extract: competitive positioning, trade-offs, deal-breakers

4. Security (WebSearch)

Question: Is it safe to use?

Sources:

• CVEs: {lib} CVE vulnerability 2025 (nvd.nist.gov, snyk.io)
• Advisories: {lib} security advisory 2025 (github.com)

Extract: known vulnerabilities, security track record, response time

5. Community Health (WebSearch)

Question: Will we get support when we need it?

Trusted sources:

• reddit.com: Real-world experiences, gotchas, war stories
• news.ycombinator.com: Technical discourse, deep dives
• github.com: Issues, PRs, responsiveness, contributor activity
• stackoverflow.com: Common problems, solutions, adoption

Searches:

• Adoption: {lib} adoption trends 2025
• Experiences: {lib} experiences 2025 (reddit.com)
• Discussion: {lib} discussion 2025 (news.ycombinator.com)
• Issues: {lib} problems issues 2025 (github.com, stackoverflow.com)

Extract: community size, responsiveness, common pain points, maturity

6. Current Usage (Codebase)

Question: How does this fit with what we already have?

• Semantic search: {lib} usage patterns
• Import count: How widely used?
• Migration effort: What needs to change?

Extract: integration complexity, migration cost, team familiarity

Output Format: Decision Brief

Template Structure

# Library Evaluation: {Name}

## 📋 TL;DR (30-second read)

**Recommendation:** [ADOPT / MIGRATE / UPGRADE / STAY / AVOID]
**Confidence:** [High/Medium/Low] (8/10)
**Priority:** [Critical/High/Medium/Low]
**Time to Adopt:** [Days/Weeks/Months]

**One-sentence summary:** {What it is and why it matters}

**Key insight:** {The one thing that drives the recommendation}

---

## 🎯 Decision Matrix (3-minute read)

| Dimension | Score | Evidence | Impact |
|-----------|-------|----------|--------|
| **Functionality** | ⭐⭐⭐⭐⭐ 9/10 | Does X, Y, Z | ✅ Meets all requirements |
| **Performance** | ⭐⭐⭐⭐ 8/10 | 20% faster than current | ✅ Significant improvement |
| **Security** | ⭐⭐⭐⭐⭐ 10/10 | No CVEs, active patches | ✅ Production-ready |
| **Community** | ⭐⭐⭐⭐ 8/10 | 50K GitHub stars, active | ✅ Strong support |
| **Maturity** | ⭐⭐⭐ 6/10 | v2.0, 3 years old | ⚠️ Some rough edges |
| **Migration** | ⭐⭐⭐⭐ 8/10 | 2-3 days effort | ✅ Low risk |

**Overall:** 8.2/10 - **Strong recommendation**

---

## ✅ Recommendation

**{DECISION}**: {Clear action statement}

**Why:**
1. {Primary reason - the deal-maker}
2. {Secondary reason - supporting evidence}
3. {Tertiary reason - nice-to-have}

**Why not alternatives:**
- {Alt 1}: {Specific disqualifying reason}
- {Alt 2}: {Specific disqualifying reason}

**Risks & Mitigations:**
- ⚠️ {Risk}: {Mitigation strategy}
- ⚠️ {Risk}: {Mitigation strategy}

---

## 🚀 Action Items

**Immediate (this week):**
1. [ ] {Specific action with owner}
2. [ ] {Specific action with owner}

**Short-term (this month):**
1. [ ] {Specific action}
2. [ ] {Specific action}

**Long-term (this quarter):**
1. [ ] {Specific action}

---

## 📊 Supporting Evidence (10-minute read)

### What We Found

#### ✅ Strengths
- {Specific strength with evidence}
- {Specific strength with evidence}

#### ⚠️ Limitations
- {Specific limitation with workaround}
- {Specific limitation with impact}

#### 🔴 Deal-breakers (if any)
- {What would disqualify this library}

### Community Insights

**From Reddit/HN:**
- {Key insight from real users}
- {Common gotcha or pain point}

**From GitHub:**
- {Issue responsiveness}
- {Active development indicators}

### Alternatives Considered

| Alternative | Why Not? | When to Reconsider |
|-------------|----------|-------------------|
| {Alt 1} | {Disqualifying reason} | {Condition} |
| {Alt 2} | {Disqualifying reason} | {Condition} |

---

## 📚 References

- Official docs: {Context7 library ID}
- GitHub: {URL}
- Key discussions: {URLs to reddit/HN threads}
- Benchmarks: {URLs}
- Security: {CVE/advisory links}

---

## 🔄 Next Review

**When to revisit:**
- {Specific trigger, e.g., "v3.0 release"}
- {Time-based, e.g., "6 months from now"}
- {Condition-based, e.g., "if performance degrades"}

Synthesis Guidelines

What to Include

Include if:

• ✅ Directly impacts the decision
• ✅ User will ask “what about X?”
• ✅ Reveals a non-obvious trade-off
• ✅ Prevents a future mistake

Exclude if:

• ❌ Common knowledge or easily Googled
• ❌ Doesn’t differentiate from alternatives
• ❌ Historical context without current relevance
• ❌ Marketing fluff or hype

How to Synthesize

1. Find the signal in the noise

   • Look for consensus across sources
   • Weight trusted sources (docs, GitHub) > marketing
   • Real user experiences > benchmarks

2. Identify the decision driver

   • What’s the ONE thing that matters most?
   • What’s the deal-maker or deal-breaker?
   • What’s the non-obvious insight?

3. Quantify when possible

   • “20% faster” > “fast”
   • “50K GitHub stars” > “popular”
   • “v3.0, 5 years old” > “mature”

4. Be honest about gaps

   • “No benchmark data available” > assume/guess
   • “Limited community feedback” > extrapolate
   • “Confidence: Medium (6/10)” > overstate

Research Workflow

Phase 1: Gather (15 minutes)

1. Context: Read pyproject.toml for existing libraries and Context7 IDs
2. Docs: Get official docs (check pyproject.toml first, then resolve-library-id)
3. Updates: WebSearch for recent changes, changelogs
4. Alternatives: WebSearch for comparisons, discussions
5. Security: WebSearch for CVEs, advisories
6. Community: WebSearch on reddit, HN, GitHub, StackOverflow
7. Codebase: Search current usage patterns

Phase 2: Synthesize (10 minutes)

1. Score each dimension (1-10) with specific evidence
2. Identify decision driver (what matters most?)
3. Determine recommendation (ADOPT/MIGRATE/UPGRADE/STAY/AVOID)
4. Assess confidence (High/Medium/Low)
5. List action items (what happens next?)

Phase 3: Write (15 minutes)

1. TL;DR: Write last, summarize the decision
2. Decision Matrix: Fill in scores and evidence
3. Recommendation: State decision and rationale
4. Action Items: Specific, assignable tasks
5. Supporting Evidence: Details for skeptics
6. References: Links for deep dive

Total time: ~40 minutes (vs 1-2 hours of ad-hoc research)

Quality Checklist

Before delivering the report:

• Can user make decision from TL;DR alone? (30 seconds)
• Is recommendation clear and confident? (not “it depends”)
• Are scores backed by specific evidence? (not gut feel)
• Are action items specific and assignable? (not vague)
• Are risks identified with mitigations? (not ignored)
• Is it scannable? (bullets, tables, sections)
• Is it < 2 pages for main content? (not a wall of text)
• Would you read this if someone else wrote it? (readability test)

Integration Points

With @researcher Agent

@researcher delegates library research to this skill, focusing on broader architectural questions.

With ADR Creation

Use this skill’s output to populate the “Alternatives Considered” section of ADRs.

With Security Audits

Run this skill on all dependencies periodically to catch security issues early.

With pyproject.toml

When researching NEW library (not in [tool.context7]):

• ALWAYS suggest adding library ID to pyproject.toml [tool.context7]
• Format: {lib} = { id = "{id}", trust = {score}, snippets = {count}, desc = "{desc}" }
• Creates single source of truth for future research

Success Metrics

Adoption Metrics

• Time to decision: < 1 hour (vs 2-4 hours ad-hoc)
• Decision confidence: 8+/10 consistently
• Decision accuracy: Right call 90%+ of the time
• Time to adoption: Days (vs weeks of analysis paralysis)

Quality Metrics

• Readability: User can scan and understand in < 5 minutes
• Actionability: Clear next steps, not open questions
• Completeness: All 6 dimensions covered with evidence
• Reproducibility: Different researchers reach same conclusion

Impact Metrics

• Avoided mistakes: Caught security issues, performance problems before adoption
• Faster adoption: Evidence-based decisions reduce bike-shedding
• Better outcomes: Right tool for job, not latest hype

Common Pitfalls

❌ Don’t:

• Dump all search results into report
• List features without context
• Ignore migration cost
• Assume newer = better
• Skip security checks
• Write for yourself (write for tired, busy reader)

✅ Do:

• Synthesize findings into insights
• Explain WHY features matter
• Quantify migration effort
• Evaluate maturity and stability
• Always check CVEs
• Write for skimmers (bullets, tables, sections)

Expected Outcomes

Successful Research Completion

✅ Research Complete

Library: pydantic
Time: 38 minutes
Confidence: High (8/10)

Decision Brief Delivered:
  ✅ TL;DR (30-second read) - Clear recommendation
  ✅ Decision Matrix - 6 dimensions scored with evidence
  ✅ Action Items - Specific, assignable tasks
  ✅ Supporting Evidence - Detailed rationale
  ✅ References - All sources documented

Quality Checks:
  ✅ Can user make decision from TL;DR alone?
  ✅ Is recommendation clear and confident?
  ✅ Are scores backed by specific evidence?
  ✅ Are action items specific and assignable?
  ✅ Are risks identified with mitigations?
  ✅ Is it scannable? (bullets, tables, sections)
  ✅ Is it < 2 pages for main content?

Next Steps:
  1. User reviews decision brief
  2. Team discusses action items
  3. Decision documented in ADR (if adoption)

Insufficient Evidence

⚠️  Research Incomplete

Library: obscure-lib
Confidence: Low (4/10)

Issues Found:
  - No official documentation found
  - Limited community feedback (< 10 GitHub stars)
  - No security audit data available
  - No alternatives comparison possible

Recommendation:
  ❌ AVOID - Insufficient evidence for production use

When to Reconsider:
  - If library reaches 1000+ GitHub stars
  - If official documentation published
  - If security audit completed

Requirements

Tools needed:

• Read – Access pyproject.toml and codebase files
• WebSearch – Research updates, alternatives, security, community
• mcp__context7__get-library-docs – Fetch official documentation
• mcp__context7__resolve-library-id – Resolve library identifiers
• mcp__project-watch-mcp__search_code – Search codebase usage
• mcp__memory__create_entities – Store research findings

Environment:

• Context7 configured in pyproject.toml
• Internet access for WebSearch
• Neo4j running (for codebase search)

Knowledge:

• Understanding of decision brief format
• Ability to synthesize technical information
• Familiarity with project architecture
• Knowledge of Context7 library registry

Optional:

• pyproject.toml management experience
• ADR creation workflow familiarity

Troubleshooting

Issue: Context7 library ID not found

Symptom: resolve-library-id returns no results

Solutions:

1. Check library name spelling (exact match required)
2. Search Context7 registry manually: mcp__context7__search-libraries
3. Use WebSearch as fallback for official docs
4. ALWAYS suggest adding library ID to pyproject.toml [tool.context7] after resolving

Example:

[tool.context7]
pydantic = { id = "pydantic_pydantic", trust = 90, snippets = 20, desc = "Data validation" }

Issue: Conflicting recommendations from sources

Symptom: Reddit says “avoid”, GitHub shows active development, docs look professional

Solutions:

1. Weight trusted sources higher – Official docs, GitHub > Reddit opinions
2. Look for recency – 2025 opinions > 2023 opinions
3. Identify context – Reddit complaint about v1, you’re researching v2
4. Be honest about conflict – Note in decision brief: “Mixed community feedback”
5. Adjust confidence score – Lower from High to Medium if significant conflict

Issue: No recent updates found

Symptom: Last release was 2+ years ago

Questions to ask:

1. Is library mature/stable (intentionally low churn)?
2. Is library abandoned (no issue responses, PRs ignored)?
3. Check GitHub: Last commit date, issue response time
4. Check alternatives: Are competitors more active?

Decision impact:

• If mature: ✅ May still be good choice
• If abandoned: ❌ AVOID or MIGRATE away

Issue: Decision brief too long (> 2 pages)

Symptom: Supporting Evidence section has walls of text

Solutions:

1. Ruthlessly cut – If it doesn’t change the decision, remove it
2. Move to references – Create research-{lib}.md in session workspace
3. Use bullets – Convert paragraphs to scannable bullet points
4. Focus on signal – What’s the ONE thing that matters most?

Target length:

• TL;DR: 1 screen
• Decision Matrix: 1 screen
• Full brief: < 2 pages

Red Flags to Avoid

Research Process

1. Dumping raw data – Synthesize findings, don’t copy-paste search results
2. No clear recommendation – Always have ADOPT/MIGRATE/UPGRADE/STAY/AVOID decision
3. Vague action items – “Consider using” → “POC this week with John on auth module”
4. Ignoring security – Always check CVEs, even if library looks safe
5. Assuming newer = better – Maturity and stability matter
6. Skipping migration cost – Easy adoption ≠ easy migration
7. Cherry-picking evidence – Include both strengths and limitations
8. Writing for yourself – Write for tired, busy reader who skims

Decision Brief Quality

9. Walls of text – Use bullets, tables, sections for scannability
10. Missing confidence score – Always include High/Medium/Low with /10 rating
11. No “when to reconsider” – Document conditions for revisiting decision
12. Broken references – Verify all links work before delivering
13. Marketing fluff – Focus on evidence, not hype
14. No time estimate – Include adoption/migration time estimate

Integration

15. Not updating pyproject.toml – Always suggest adding new library IDs to Context7 config
16. Skipping ADR creation – Major adoption decisions should have ADRs
17. No memory storage – Store research findings in mcp__memory for future reference

Evolution & Improvement

This skill should improve over time:

1. Track decisions: Did we make the right call?
2. Refine criteria: What dimensions matter most?
3. Improve synthesis: What signal did we miss?
4. Optimize sources: Which sources had best signal/noise?
5. Update templates: What format worked best?

Feedback loop: After 6 months, review decisions and update methodology.

Expected Benefits

Validation Process

Step 1: Research Validation

# Verify all 6 dimensions covered
✓ Official Documentation (Context7)
✓ Recent Developments (WebSearch)
✓ Alternatives (WebSearch comparisons)
✓ Security (CVE search)
✓ Community Health (Reddit, HN, GitHub)
✓ Current Usage (codebase search)

Step 2: Decision Matrix Validation

# Each dimension scored 1-10 with evidence
# Overall score calculated
# Confidence level assigned (High/Medium/Low)

Step 3: Recommendation Validation

# Clear ADOPT/MIGRATE/UPGRADE/STAY/AVOID decision
# Rationale documented
# Action items specific and assignable

Step 4: Quality Checklist

# Can user make decision from TL;DR alone? (30 seconds)
# Is recommendation clear and confident?
# Are scores backed by specific evidence?
# Are action items specific and assignable?
# Are risks identified with mitigations?
# Is it scannable? (bullets, tables, sections)

Step 5: Report Delivery

# Decision brief < 2 pages
# Supporting evidence documented
# References included
# Next review conditions specified

See Also

• references/date-aware-search.md – WebSearch strategies for current, relevant information
• references/research-checklist.md – Comprehensive checklist for all 6 dimensions
• templates/decision-matrix-template.md – Copy-paste template for evaluations

Last Updated: 2025-10-16 Version: 2.0 (Renamed from research-sota-library, refocused on methodology)