Multi-Agent Workflow Engineering Skill

Table of Contents

• Quick Start
• Core Principle
• The Three Patterns
• Design Principles
• Decision Framework
• Failure Modes
• Anti-Patterns
• Bundled Resources

Quick Start

Three patterns form the foundation of reliable multi-agent systems. Each uses TypeScript and Zod to enforce contracts at agent boundaries.

1. Typed Schema – Validate Agent Handoff Data

import { z } from "zod";

const AgentHandoff = z.object({
  sourceAgent: z.string(),
  targetAgent: z.string(),
  timestamp: z.string().datetime(),
  payload: z.object({
    taskId: z.string().uuid(),
    status: z.enum(["pending", "in_progress", "completed", "failed"]),
    data: z.record(z.unknown()),
  }),
  metadata: z.object({
    correlationId: z.string().uuid(),
    retryCount: z.number().int().min(0).default(0),
    schemaVersion: z.literal("1.0"),
  }),
});

type AgentHandoff = z.infer<typeof AgentHandoff>;

// Validate on send
const validated = AgentHandoff.parse(outgoingData);

// Validate on receive
const received = AgentHandoff.parse(incomingData);

2. Action Schema – Constrain Agent Behavior

import { z } from "zod";

const AgentAction = z.discriminatedUnion("type", [
  z.object({
    type: z.literal("delegate"),
    targetAgent: z.string(),
    task: z.string(),
    deadline: z.string().datetime().optional(),
  }),
  z.object({
    type: z.literal("respond"),
    content: z.string(),
    confidence: z.number().min(0).max(1),
  }),
  z.object({
    type: z.literal("escalate"),
    reason: z.string(),
    severity: z.enum(["low", "medium", "high", "critical"]),
  }),
  z.object({
    type: z.literal("retry"),
    originalAction: z.string(),
    backoffMs: z.number().int().positive(),
  }),
]);

type AgentAction = z.infer<typeof AgentAction>;

// Exhaustive matching ensures no action is missed
function handleAction(action: AgentAction): void {
  switch (action.type) {
    case "delegate":
      return routeToAgent(action.targetAgent, action.task);
    case "respond":
      return sendResponse(action.content);
    case "escalate":
      return notifyHuman(action.reason, action.severity);
    case "retry":
      return scheduleRetry(action.originalAction, action.backoffMs);
  }
}

3. MCP Contract – Define Tool Interfaces

import { z } from "zod";

const SearchToolInput = z.object({
  query: z.string().min(1).max(500),
  filters: z.object({
    dateRange: z.object({
      from: z.string().datetime(),
      to: z.string().datetime(),
    }).optional(),
    maxResults: z.number().int().min(1).max(100).default(10),
  }).optional(),
});

const SearchToolOutput = z.object({
  results: z.array(z.object({
    id: z.string(),
    title: z.string(),
    relevance: z.number().min(0).max(1),
    snippet: z.string(),
  })),
  totalCount: z.number().int().min(0),
  executionMs: z.number().int().min(0),
});

const searchTool = {
  name: "search_documents",
  description: "Search documents with structured filters and validated results.",
  inputSchema: SearchToolInput,
  outputSchema: SearchToolOutput,
  handler: async (input: z.infer<typeof SearchToolInput>) => {
    const validated = SearchToolInput.parse(input);
    const results = await performSearch(validated);
    return SearchToolOutput.parse(results);
  },
};

Core Principle

Treat agents like distributed systems, not chat flows.

Most multi-agent failures stem from treating agents as cooperative conversation partners that naturally understand each other. In reality, agents are unreliable network services. This mental model shift changes everything about how you design multi-agent systems.

Chat flow thinking assumes agents share context, understand intent implicitly, pass data freely, and recover gracefully from errors. This works in demos but fails in production.

Distributed systems thinking assumes agents have no shared context, intent must be explicit in every message, data must be validated at every boundary, and failures are inevitable and must be handled. This is what makes multi-agent systems reliable.

The practical implications:

• Explicit contracts: Every agent boundary has a typed schema. No agent trusts data from another agent without validation.
• Validation at boundaries: Data is validated on send AND on receive. Schema mismatches surface immediately as errors, not as subtle downstream bugs.
• Failure handling: Every agent interaction has a timeout, a retry strategy, and an escalation path. No “happy path only” designs.
• Observability: Every message carries a correlation ID. Every state transition is logged. When something fails at 3 AM, you can trace exactly what happened.

The Three Patterns

Pattern 1: Typed Schemas

Problem: Agents pass unstructured data between each other. A planning agent emits a JSON blob, a coding agent consumes it, and somewhere in between a field name changes or a type shifts from string to number. The system doesn’t crash — it silently produces wrong results.

Solution: Define Zod schemas at every agent boundary. Validate data when sending and when receiving.

Key rules:

1. Validate on send AND receive – The sender validates it produced correct output. The receiver validates it got correct input. Both sides check.
2. Version your schemas – Include a schemaVersion field. When schemas evolve, old and new agents can coexist during rollout.
3. No any types – Every field has an explicit type. z.unknown() is acceptable only inside z.record() for truly dynamic data, and even then the consumer must narrow the type before use.
4. Fail loudly – Use .parse() not .safeParse() at boundaries. A schema violation should throw, not return a success-with-undefined.

const TaskResult = z.object({
  taskId: z.string().uuid(),
  agentId: z.string(),
  schemaVersion: z.literal("1.0"),
  result: z.discriminatedUnion("status", [
    z.object({ status: z.literal("success"), output: z.string() }),
    z.object({ status: z.literal("failure"), error: z.string(), retryable: z.boolean() }),
  ]),
});

Pattern 2: Action Schemas

Problem: Agents take ambiguous actions. An agent decides to “process” something, but what does “process” mean? Does it transform data, delegate to another agent, store a result, or call an external API? Ambiguity leads to unpredictable behavior and debugging nightmares.

Solution: Define a discriminated union type that enumerates every possible action an agent can take. Each action variant has its own typed payload.

Key rules:

1. Exhaustive matching – Use TypeScript’s never type or a switch statement to ensure every action variant is handled. If you add a new action, the compiler tells you everywhere it needs handling.
2. Explicit intent – The action type field is a literal string that describes exactly what the agent intends. No ambiguous verbs like “process” or “handle”.
3. No implicit actions – If an agent can do it, it must be in the union. Side effects that happen outside the action schema are bugs.
4. Typed payloads – Each action variant carries only the data it needs. A “delegate” action has a target agent and task. A “respond” action has content and confidence. No shared bag-of-properties.

const WorkflowAction = z.discriminatedUnion("type", [
  z.object({ type: z.literal("fetch_data"), source: z.string().url(), timeout: z.number() }),
  z.object({ type: z.literal("transform"), inputKey: z.string(), outputKey: z.string() }),
  z.object({ type: z.literal("store_result"), key: z.string(), value: z.unknown() }),
  z.object({ type: z.literal("notify"), channel: z.string(), message: z.string() }),
]);

Pattern 3: MCP Contracts

Problem: Tool interfaces drift over time. An MCP tool changes its expected input format, but the agents calling it still send the old format. Or a tool starts returning additional fields that downstream agents don’t expect. Integration breaks silently.

Solution: Define MCP tool contracts with Zod schemas for both input and output. Validate before execution (reject bad requests early) and after execution (ensure the tool produced what it promised).

Key rules:

1. Pre-execution validation – Validate input before calling the tool handler. Return a structured error if validation fails. Don’t let bad data reach the implementation.
2. Typed responses – The tool output schema is not optional. Every tool declares what it returns, and the output is validated before being sent back to the calling agent.
3. Contract tests – Write tests that verify the tool’s schema contract independently of its implementation. If the schema changes, the test fails before any agent integration breaks.
4. Versioned endpoints – When a tool’s contract changes, bump the version and support both old and new versions during migration.

function createMcpTool<I extends z.ZodType, O extends z.ZodType>(config: {
  name: string;
  description: string;
  inputSchema: I;
  outputSchema: O;
  handler: (input: z.infer<I>) => Promise<z.infer<O>>;
}) {
  return {
    ...config,
    execute: async (rawInput: unknown) => {
      const input = config.inputSchema.parse(rawInput);
      const result = await config.handler(input);
      return config.outputSchema.parse(result);
    },
  };
}

Design Principles

Six principles derived from engineering reliable multi-agent systems in production:

1. Design for Failure First

Before writing the happy path, answer: What happens when this agent times out? What happens when it returns garbage? What happens when it’s unavailable for 30 seconds? Design the failure path first, then build the success path on top of it.

2. Validate Every Agent Boundary

Every time data crosses from one agent to another, validate it with a schema. This applies to agent-to-agent handoffs, agent-to-tool calls, tool-to-agent responses, and external API interactions. No boundary is too trivial to validate.

3. Constrain Actions Before Adding Complexity

Start with the smallest possible action schema. An agent that can only respond or escalate is easier to reason about than one that can respond, delegate, transform, store, notify, and retry. Add actions only when you have a concrete need and a test for each one.

4. Log Intermediate State

Every agent should log its input, its decision, and its output as structured data with a correlation ID. When a 5-agent pipeline produces a wrong result, you need to trace the exact point where correct data became incorrect. Logging only the final output is not enough.

5. Make Agent Intent Explicit

An agent should never “just do something”. Every action it takes must be represented in its action schema with a clear type literal. If you’re debugging and an agent took an action that isn’t in its schema, you have a design bug, not a runtime bug.

6. Test Contracts, Not Just Outputs

Don’t only test that the system produces the right final output. Test that each agent’s input and output conform to their schemas. Test that action schemas are handled exhaustively. Test that MCP tool contracts are stable. Contract tests catch integration bugs before they reach production.

Decision Framework

When to Use Single Agent vs Multi-Agent

Score each dimension from 1 (single agent) to 5 (multi-agent). Sum the scores to guide your decision.

Scoring guide:

• 5-12: Use a single agent. The overhead of multi-agent coordination will slow you down without meaningful benefit.
• 13-18: Consider multi-agent. Evaluate whether the coordination cost is justified by the gains in specialization and isolation.
• 19-25: Use multi-agent. The task naturally decomposes, the domains are distinct, and you need failure isolation.

Important: A multi-agent system is not inherently better than a single agent. It is inherently more complex. Only add agents when the complexity buys you something concrete — better failure isolation, independent scaling, or domain specialization that a single agent cannot achieve.

Failure Modes

The six most common failure modes in multi-agent systems, with prevention strategies.

1. Schema Drift

What happens: Agent A’s output schema and Agent B’s input schema diverge over time. A field gets renamed, a type changes, or an optional field becomes required. Neither agent fails — they just silently produce wrong results.

Prevention: Shared schema definitions imported by both agents. Schema version fields that trigger explicit errors on mismatch. Contract tests that run on every build.

2. Action Ambiguity

What happens: Two agents interpret the same action type differently. Agent A thinks “process” means “transform data”. Agent B thinks “process” means “store result”. The system appears to work until edge cases reveal the divergence.

Prevention: Literal string type discriminators (z.literal("transform") not z.string()). One canonical action schema shared across all agents. Exhaustive switch statements that the compiler verifies.

3. State Corruption

What happens: Multiple agents modify shared state concurrently. Agent A reads state, Agent B modifies it, Agent A writes its changes based on stale data. The result is a state that neither agent intended.

Prevention: Immutable state with event sourcing. Each agent produces events, a single reducer applies them in order. Optimistic concurrency with version fields. No direct state mutation.

4. Cascade Failure

What happens: Agent A depends on Agent B, which depends on Agent C. Agent C fails. Agent B retries, creating load. Agent A times out waiting for B. The entire pipeline collapses because one leaf agent had a transient error.

Prevention: Circuit breakers at every agent boundary. Timeouts with escalation (don’t just retry — escalate to a human or fallback after N retries). Bulkhead isolation so one failing path doesn’t consume all resources.

5. Ordering Violation

What happens: The system assumes agents execute in a specific order, but under load or during retries, the order changes. Agent B processes before Agent A has finished, working with incomplete data.

Prevention: Explicit dependency declarations in the orchestrator. Agents declare their prerequisites. The orchestrator enforces ordering. No implicit ordering assumptions in agent logic.

6. Silent Data Loss

What happens: An agent receives data that doesn’t match its expected schema, silently drops the unrecognized fields, and continues processing. The dropped fields were critical for a downstream agent.

Prevention: Strict schema validation with .parse() (not .safeParse() with silent fallbacks). Schema rules like .strict() that reject unexpected fields. Logging of every validation error with the full payload for debugging.

Anti-Patterns

Bundled Resources

References (12 files)

Templates (14 files)

Specialized Agents (5)

Slash Commands (6)

Quick References

• Source Blog Post: https://github.blog/ai-and-ml/generative-ai/multi-agent-workflows-often-fail-heres-how-to-engineer-ones-that-dont/
• Zod Documentation: https://zod.dev/
• MCP Specification: https://modelcontextprotocol.io/
• TypeScript Handbook: https://www.typescriptlang.org/docs/