Observability Tools

Query metrics, logs, and dashboards for diagnostics and incident response.

Prometheus (Metrics)

Query metrics for performance analysis and alerting.

# CPU usage by pod
prometheus_query({
  query: 'rate(container_cpu_usage_seconds_total{namespace="my-service"}[5m])'
})

# Memory usage
prometheus_query({
  query: 'container_memory_usage_bytes{namespace="my-service"}'
})

# HTTP request rate
prometheus_query({
  query: 'rate(http_requests_total{namespace="my-service"}[5m])'
})

# Error rate
prometheus_query({
  query: 'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])'
})

Loki (Logs)

Query logs for debugging and incident investigation.

# Application logs
loki_query({
  query: '{namespace="my-service", app="api"} |= "error"',
  limit: 100
})

# Structured log parsing
loki_query({
  query: '{namespace="my-service"} | json | level="error"'
})

# Time-based filtering
loki_query({
  query: '{namespace="my-service"}',
  start: "2024-01-01T00:00:00Z",
  end: "2024-01-01T01:00:00Z"
})

Common Queries

Incident Response Flow

1. Check alerts – What triggered?
2. Query metrics – Is it resource exhaustion?
3. Query logs – What errors are occurring?
4. Correlate – Match timestamps across metrics and logs
5. Identify root cause – Database? Network? Code bug?

Best Practices

1. Start broad, then narrow – Filter down to specific pods
2. Use time ranges – Don’t query unbounded
3. Correlate metrics + logs – Same time window
4. Check dashboard first – Grafana may have pre-built views