陌讯skills聚合平台

data engineering

📁 majiayu000/claude-skill-registry 📅 Jan 1, 1970
1
总安装量
0
周安装量
#49270
全站排名
安装命令
npx skills add https://github.com/majiayu000/claude-skill-registry --skill Data Engineering

Skill 文档

Data Engineering Skill

Quick Reference

Role Focus Timeline Entry From
Data Engineer Pipelines, Infra 12-24 mo Backend Dev
ML Engineer Models, Features 12-24 mo Data Scientist
AI Engineer LLMs, Agents 6-12 mo Any Developer

Learning Paths

Data Engineer

[1] SQL Mastery (4-6 wk)
 │  └─ Window functions, CTEs, optimization
 │
 ▼
[2] Python for Data (4-6 wk)
 │  └─ Pandas, file formats, scripting
 │
 ▼
[3] ETL/ELT Pipelines (6-8 wk)
 │  └─ Extract, transform, load patterns
 │
 ▼
[4] Big Data: Spark (8-12 wk)
 │  └─ PySpark, DataFrames, partitioning
 │
 ▼
[5] Data Warehouse (4-6 wk)
 │  └─ Star schema, dbt, Snowflake/BQ
 │
 ▼
[6] Orchestration (4-6 wk)
    └─ Airflow/Prefect, scheduling, monitoring

2025 Stack: Python + Spark + Airflow + dbt + Snowflake/BigQuery

ML Engineer

[1] Python + NumPy (4-6 wk)
 │
 ▼
[2] Math Foundations (6-8 wk)
 │  └─ Linear algebra, calculus, statistics
 │
 ▼
[3] Classical ML (8-12 wk)
 │  └─ scikit-learn, XGBoost, evaluation
 │
 ▼
[4] Deep Learning (8-12 wk)
 │  └─ PyTorch, CNNs, Transformers
 │
 ▼
[5] MLOps (6-8 wk)
    └─ MLflow, model serving, monitoring

2025 Stack: Python + PyTorch + scikit-learn + MLflow + W&B

AI Engineer (2025 Hot Path)

[1] LLM Fundamentals (2-3 wk)
 │  └─ Tokens, embeddings, context windows
 │
 ▼
[2] Prompt Engineering (2-3 wk)
 │  └─ Few-shot, CoT, structured output
 │
 ▼
[3] RAG Systems (3-4 wk)
 │  └─ Embeddings, vector DBs, retrieval
 │
 ▼
[4] AI Agents (4-6 wk)
 │  └─ Tool calling, agent loops, memory
 │
 ▼
[5] Production Deploy (ongoing)
    └─ Evaluation, guardrails, monitoring

2025 Stack: Python + LangChain/LlamaIndex + OpenAI/Anthropic + ChromaDB

2025 Tool Matrix

Data Processing

Tool Scale Use Case
Pandas <10GB Prototyping, small data
Polars <100GB Fast local processing
Spark >100GB Distributed processing
dbt Any Transformations, testing

ML Frameworks

Framework Best For Complexity
scikit-learn Classical ML Low
XGBoost Tabular data Low
PyTorch Research, flexibility Medium
TensorFlow Production, mobile Medium

LLM/AI Tools

Tool Use Case
LangChain LLM orchestration
LlamaIndex RAG systems
Claude/OpenAI LLM APIs
ChromaDB Vector storage

Algorithm Reference

Classical ML

Type Algorithms
Regression Linear, Ridge, Lasso, ElasticNet
Classification Logistic, SVM, Decision Tree
Ensemble Random Forest, XGBoost, LightGBM
Clustering K-Means, DBSCAN, Hierarchical

Deep Learning

Architecture Use Case
CNN Images, vision
RNN/LSTM Sequences
Transformer NLP, LLMs
Diffusion Image generation

AI Agent Architecture (2025)

┌─────────────────────────────────────────┐
│            AGENTIC LOOP                  │
├─────────────────────────────────────────┤
│  PERCEIVE → REASON → ACT → REFLECT      │
│      │         │       │       │        │
│      │         │       │       └─► Loop │
│      │         │       └─► Execute tools│
│      │         └─► LLM decides action   │
│      └─► Gather context, observations   │
└─────────────────────────────────────────┘

Design Patterns (Anthropic 2025):
• Prompt Chaining - Sequential fixed steps
• Routing - Classify and dispatch
• Parallelization - Concurrent subtasks
• Orchestrator-Workers - Central delegation
• Evaluator-Optimizer - Generate + critique

Troubleshooting

Which path to choose?
├─► Love building infrastructure? → Data Engineer
├─► Love algorithms/math? → ML Engineer
├─► Want fastest AI entry? → AI Engineer
└─► Uncertain? → Start with Python + SQL

Model not performing well?
├─► Data quality issues? → Clean data first
├─► Feature engineering? → Create better features
├─► Wrong algorithm? → Try different models
├─► Overfitting? → More data, regularization
└─► Hyperparameters? → Grid/random search

LLM giving bad answers?
├─► Prompt too vague? → Be more specific
├─► Missing context? → Add relevant info
├─► Hallucinating? → Use RAG, verify facts
└─► Wrong tool? → Improve tool descriptions

Common Failure Modes

Symptom Root Cause Recovery
Model fails in prod Data drift Monitor distributions
Pipeline always late Unoptimized queries Profile, partition
RAG finds wrong docs Bad chunking Tune chunk size, overlap
Agent loops forever No exit condition Add max iterations

Portfolio Projects

Data Engineering

  1. ETL Pipeline (Airflow + dbt)
  2. Real-time Streaming (Kafka + Spark)
  3. Data Warehouse Design

ML Engineering

  1. Classification Model (scikit-learn)
  2. Deep Learning Model (PyTorch)
  3. ML Pipeline (MLflow)

AI Engineering

  1. RAG Chatbot (LangChain + ChromaDB)
  2. AI Agent with Tools
  3. Multi-Agent System

Next Actions

Specify your target role for a detailed learning plan.

GitHub 仓库 ↗ ← 返回陌讯 Skills 聚合平台
本页面由 陌讯 Skills 聚合平台 收录整理，数据定期更新。