observability-testing

📁 apankov1/quality-engineering 📅 Today
0
总安装量
1
周安装量
安装命令
npx skills add https://github.com/apankov1/quality-engineering --skill observability-testing

Agent 安装分布

amp 1
cline 1
opencode 1
cursor 1
continue 1
kimi-cli 1

Skill 文档

Observability Testing

Test that your code produces correct logs — not just that it runs.

Logs are your only window into production behavior. If critical paths don’t log correctly, you’re flying blind during incidents. This skill teaches you to verify structured log output as part of your test suite.

When to use: Testing error logging with context, audit trails, monitoring integration, log level policy enforcement, any code where observability matters.

When not to use: Pure business logic tests, UI components, tests where logging is incidental not critical.

Rationalizations (Do Not Skip)

Rationalization Why It’s Wrong Required Action
“Logs are side effects, not behavior” Incorrect logs = blind in production Assert log output as first-class behavior
“I can see logs when I run it” Manual inspection doesn’t scale Automate with mock logger assertions
“Any log level is fine” Wrong levels = alert fatigue or missed incidents Enforce log level policy
“Context isn’t important” Context-free logs are useless for debugging Assert required context fields

Included Utilities

import {
  createMockLogger,
  assertLogEntry,
  assertNoLogsAbove,
  assertHasLogLevel,
  assertErrorLogged,
  LOG_LEVEL_POLICY,
  LOG_LEVEL_ORDER,
  classifyLogLevel,
} from './structured-logger.ts';

Core Workflow

Step 1: Create Mock Logger

Replace real loggers with mock loggers that capture entries for assertion:

const logger = createMockLogger();

// Pass to code under test
await myHandler({ logger });

// Assert on captured entries
assert.equal(logger.entries.length, 2);

Step 2: Assert Specific Log Entries

Verify that specific logs were recorded with correct level, message, and context:

it('logs error with full context on failure', async () => {
  const logger = createMockLogger();

  await assert.rejects(() => myHandler({ logger, shouldFail: true }));

  assertLogEntry(logger, 'error', 'Request failed', {
    component: 'Handler',
    path: '/api/test',
  });
});

Step 3: Verify Happy Path Has No Warnings/Errors

Happy paths should not produce warnings or errors — that’s alert fatigue:

it('happy path produces no warnings or errors', async () => {
  const logger = createMockLogger();

  await myHandler({ logger });

  assertNoLogsAbove(logger, 'info');  // Only debug/info allowed
});

Step 4: Verify Error Logs Include Error Instances

Error logs should include the actual Error object for stack traces:

it('error log includes Error instance', async () => {
  const logger = createMockLogger();

  try {
    await myHandler({ logger, causeError: true });
  } catch {}

  assertErrorLogged(logger, 'Database connection failed', {
    name: 'ConnectionError',
    message: 'timeout',
  });
});

Step 5: Follow Log Level Policy

Use the correct log level based on the nature of the message:

// Reference table
console.log(LOG_LEVEL_POLICY);
/*
{
  error: {
    description: "Actionable failures requiring investigation",
    examples: ["Database connection failed", "Authentication error"],
    production: "100% always",
  },
  warn: {
    description: "Degraded but recoverable",
    examples: ["Retry exhausted but fallback worked", "Deprecated API called"],
    production: "100% always",
  },
  info: {
    description: "Significant state changes",
    examples: ["User logged in", "Config loaded", "Session started"],
    production: "sampled (1-10%)",
  },
  debug: {
    description: "Routine operations",
    examples: ["Cache hit/miss", "Heartbeat tick", "Request timing"],
    production: "off (or LOG_LEVEL=warn)",
  },
}
*/

Step 6: Classify Log Levels Automatically

Use the classifier to suggest appropriate levels:

// Returns suggested level based on keywords
classifyLogLevel("Database connection failed");  // 'error'
classifyLogLevel("Retry attempt 3 of 5");        // 'warn'
classifyLogLevel("User logged in");              // 'info'
classifyLogLevel("Cache hit for key xyz");       // 'debug'

Step 7: Enforce Logger Interface

The mock logger implements a strict interface. Your production logger should match:

interface StructuredLogger {
  debug(message: string, context?: Record<string, unknown>): void;
  info(message: string, context?: Record<string, unknown>): void;
  warn(message: string, context?: Record<string, unknown>): void;
  error(message: string, error?: Error, context?: Record<string, unknown>): void;
}

Note: error() has a different signature — it accepts an optional Error instance as the second argument for stack trace capture.


Common Misclassifications

Log Message Wrong Level Correct Level Why
Cold start recovery warn info Recovery is expected, not degradation
Stale flush skip warn debug Expected race condition, not actionable
Periodic heartbeat tick info debug Routine operation, not state change
User authentication failed info error Security event requiring investigation
Config validation warning debug warn Human should know about config issues

Violation Rules

missing_error_log_assertion

Error paths MUST have log assertions verifying correct error logging with context. Severity: must-fail

happy_path_logs_error

Happy paths MUST NOT produce warn or error logs. Use assertNoLogsAbove(logger, 'info'). Severity: must-fail

log_level_misclassification

Log levels MUST follow the policy. Routine operations at warn/error = alert fatigue. Severity: should-fail

missing_context_fields

Error logs MUST include context fields for debugging (component, userId, requestId, etc.). Severity: should-fail

console_spy_instead_of_mock

DO NOT spy on console.* — use a proper mock logger interface. Severity: must-fail


Companion Skills

This skill provides testing utilities for structured logging, not logging architecture guidance. For broader methodology:

  • Search logging or observability on skills.sh for production logging architecture, wide events, and OpenTelemetry integration
  • Error paths tested here often originate from resilience scenarios — use fault-injection-testing for circuit breaker, retry policy, and queue preservation testing
  • State machine transitions should log state changes at correct levels — use model-based-testing for systematic transition matrix coverage

Quick Reference

Assertion When Example
assertLogEntry Verify specific log assertLogEntry(logger, 'error', 'Failed', { component: 'X' })
assertNoLogsAbove Happy path validation assertNoLogsAbove(logger, 'info')
assertHasLogLevel Verify level exists assertHasLogLevel(logger, 'error')
assertErrorLogged Verify Error instance assertErrorLogged(logger, 'Failed', { name: 'TypeError' })
classifyLogLevel Suggest correct level classifyLogLevel("Connection timeout") → ‘warn’

See patterns.md for correlation field patterns, log level migration guides, and framework integration.