Deploy App Workflow

End-to-end orchestration for deploying applications to the Kubernetes homelab with full monitoring integration.

Workflow Overview

┌─────────────────────────────────────────────────────────────────────┐
│                      /deploy-app Workflow                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  1. RESEARCH                                                        │
│     ├─ Invoke kubesearch skill for real-world patterns              │
│     ├─ Check if native Helm chart exists (helm search hub)          │
│     ├─ Determine: native chart vs app-template                      │
│     └─ AskUserQuestion: Present findings, confirm approach          │
│                                                                      │
│  2. SETUP                                                           │
│     └─ task wt:new -- deploy-<app-name>                             │
│        (Creates isolated worktree + branch)                         │
│                                                                      │
│  3. CONFIGURE (in worktree)                                         │
│     ├─ kubernetes/platform/versions.env (add version)               │
│     ├─ kubernetes/platform/namespaces.yaml (add namespace)          │
│     ├─ kubernetes/platform/helm-charts.yaml (add input)             │
│     ├─ kubernetes/platform/charts/<app>.yaml (create values)        │
│     ├─ kubernetes/platform/kustomization.yaml (register)            │
│     ├─ .github/renovate.json5 (add manager)                         │
│     └─ kubernetes/platform/config/<app>/ (optional extras)          │
│        ├─ route.yaml (HTTPRoute if exposed)                         │
│        ├─ canary.yaml (health checks)                               │
│        ├─ prometheus-rules.yaml (custom alerts)                     │
│        └─ dashboard.yaml (Grafana ConfigMap)                        │
│                                                                      │
│  4. VALIDATE                                                        │
│     ├─ task k8s:validate                                            │
│     └─ task renovate:validate                                       │
│                                                                      │
│  5. TEST ON DEV (bypass Flux)                                       │
│     ├─ helm install directly to dev cluster                         │
│     ├─ Wait for pods ready (kubectl wait)                           │
│     ├─ Verify ServiceMonitor discovered (Prometheus API)            │
│     ├─ Verify no new alerts firing                                  │
│     ├─ Verify canary passing (if created)                           │
│     └─ AskUserQuestion: Report status, confirm proceed              │
│                                                                      │
│  6. CLEANUP & PR                                                    │
│     ├─ helm uninstall from dev                                      │
│     ├─ git commit (conventional commit format)                      │
│     ├─ git push + gh pr create                                      │
│     └─ Report PR URL to user                                        │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Phase 1: Research

1.1 Search Kubesearch for Real-World Examples

Invoke the kubesearch skill to find how other homelabs configure this chart:

/kubesearch <chart-name>

This provides:

• Common configuration patterns
• Values.yaml examples from production homelabs
• Gotchas and best practices

1.2 Check for Native Helm Chart

helm search hub <app-name> --max-col-width=100

Decision matrix:

1.3 User Confirmation

Use AskUserQuestion to present findings and confirm:

• Chart selection (native vs app-template)
• Exposure type: internal, external, or none
• Namespace selection (new or existing)
• Persistence requirements

Phase 2: Setup

2.1 Create Worktree

All deployment work happens in an isolated worktree:

task wt:new -- deploy-<app-name>

This creates:

• Branch: deploy-<app-name>
• Worktree: ../homelab-deploy-<app-name>/

2.2 Change to Worktree

cd ../homelab-deploy-<app-name>

All subsequent file operations happen in the worktree.

Phase 3: Configure

3.1 Add Version to versions.env

Add a version entry with a Renovate annotation. For annotation syntax and datasource selection, see the versions-renovate skill.

# kubernetes/platform/versions.env
<APP>_VERSION="x.y.z"

3.2 Add Namespace to namespaces.yaml

Add to kubernetes/platform/namespaces.yaml inputs array:

- name: <namespace>
  dataplane: ambient
  security: baseline                    # Choose: restricted, baseline, privileged
  networkPolicy: false                  # Or object with profile/enforcement

PodSecurity Level Selection:

If security: restricted: You MUST set full security context in chart values (see step 3.4a below).

Network Policy Profile Selection:

Optional Access Labels (add if app needs these):

    access.network-policy.homelab/postgres: "true"    # Database access
    access.network-policy.homelab/garage-s3: "true"   # S3 storage access
    access.network-policy.homelab/kube-api: "true"    # Kubernetes API access

For PostgreSQL provisioning patterns, see the cnpg-database skill.

3.3 Add to helm-charts.yaml

Add to kubernetes/platform/helm-charts.yaml inputs array:

- name: "<app-name>"
  namespace: "<namespace>"
  chart:
    name: "<chart-name>"
    version: "${<APP>_VERSION}"
    url: "https://charts.example.com"  # or oci://registry.io/path
  dependsOn: [cilium]  # Adjust based on dependencies

For OCI registries:

    url: "oci://ghcr.io/org/helm"

3.4 Create Values File

Create kubernetes/platform/charts/<app-name>.yaml:

# yaml-language-server: $schema=<schema-url-if-available>
---
# Helm values for <app-name>
# Based on kubesearch research and best practices

# Enable monitoring
serviceMonitor:
  enabled: true

# Use internal domain for ingress
ingress:
  enabled: true
  hosts:
    - host: <app-name>.${internal_domain}

See references/file-templates.md for complete templates.

3.4a Add Security Context for Restricted Namespaces

If the target namespace uses security: restricted, add security context to the chart values. Check the container image’s default user first — if it runs as root, set runAsUser: 65534.

# Pod-level (key varies by chart: podSecurityContext, securityContext, pod.securityContext)
podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

# Container-level (every container and init container)
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

Restricted namespaces: cert-manager, external-secrets, system, database, kromgo.

Validation gap: task k8s:validate does NOT catch PodSecurity violations — only server-side dry-run or actual deployment reveals them. Always verify security context manually for restricted namespaces.

3.5 Register in kustomization.yaml

Add to kubernetes/platform/kustomization.yaml configMapGenerator:

configMapGenerator:
  - name: platform-values
    files:
      # ... existing
      - charts/<app-name>.yaml

3.6 Configure Renovate Tracking

Renovate tracks versions.env entries automatically via inline # renovate: annotations (added in step 3.1). No changes to .github/renovate.json5 are needed unless you want to add grouping or automerge overrides. For the full annotation workflow, see the versions-renovate skill.

3.7 Optional: Additional Configuration

For apps that need extra resources, create kubernetes/platform/config/<app-name>/:

HTTPRoute (for exposed apps)

For detailed gateway routing and certificate configuration, see the gateway-routing skill.

# config/<app-name>/route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <app-name>
spec:
  parentRefs:
    - name: internal-gateway
      namespace: gateway
  hostnames:
    - <app-name>.${internal_domain}
  rules:
    - backendRefs:
        - name: <app-name>
          port: 80

Canary Health Check

# config/<app-name>/canary.yaml
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: http-check-<app-name>
spec:
  schedule: "@every 1m"
  http:
    - name: <app-name>-health
      url: https://<app-name>.${internal_domain}/health
      responseCodes: [200]
      maxSSLExpiry: 7

PrometheusRule (custom alerts)

Only create if the chart doesn’t include its own alerts:

# config/<app-name>/prometheus-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: <app-name>-alerts
spec:
  groups:
    - name: <app-name>.rules
      rules:
        - alert: <AppName>Down
          expr: up{job="<app-name>"} == 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "<app-name> is down"

Grafana Dashboard

1. Search grafana.com for community dashboards
2. Add via gnetId in grafana values, OR
3. Create ConfigMap:

# config/<app-name>/dashboard.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-<app-name>
  labels:
    grafana_dashboard: "true"
  annotations:
    grafana_folder: "Applications"
data:
  <app-name>.json: |
    { ... dashboard JSON ... }

See references/monitoring-patterns.md for detailed examples.

Phase 4: Validate

4.1 Kubernetes Validation

task k8s:validate

This runs:

• kustomize build
• kubeconform schema validation
• yamllint checks

4.2 Renovate Validation

task renovate:validate

Fix any errors before proceeding.

Phase 5: Test on Dev

The dev cluster is a sandbox — iterate freely until the deployment works.

5.1 Suspend Flux (if needed)

If Flux would reconcile over your changes, suspend the relevant Kustomization:

task k8s:flux-suspend -- <kustomization-name>

5.2 Deploy Directly

Install or upgrade the chart directly on dev:

# Standard Helm repo
KUBECONFIG=~/.kube/dev.yaml helm install <app-name> <repo>/<chart> \
  -n <namespace> --create-namespace \
  -f kubernetes/platform/charts/<app-name>.yaml \
  --version <version>

# OCI chart
KUBECONFIG=~/.kube/dev.yaml helm install <app-name> oci://registry/<path>/<chart> \
  -n <namespace> --create-namespace \
  -f kubernetes/platform/charts/<app-name>.yaml \
  --version <version>

For iterating on values, use helm upgrade:

KUBECONFIG=~/.kube/dev.yaml helm upgrade <app-name> <repo>/<chart> \
  -n <namespace> \
  -f kubernetes/platform/charts/<app-name>.yaml \
  --version <version>

5.3 Wait for Pods

KUBECONFIG=~/.kube/dev.yaml kubectl -n <namespace> \
  wait --for=condition=Ready pod -l app.kubernetes.io/name=<app-name> --timeout=300s

5.4 Verify Network Connectivity

CRITICAL: Network policies are enforced – verify traffic flows correctly:

# Setup Hubble access (run once per session)
KUBECONFIG=~/.kube/dev.yaml kubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Check for dropped traffic (should be empty for healthy app)
hubble observe --verdict DROPPED --namespace <namespace> --since 5m

# Verify gateway can reach the app (if exposed)
hubble observe --from-namespace istio-gateway --to-namespace <namespace> --since 2m

# Verify app can reach database (if using postgres access label)
hubble observe --from-namespace <namespace> --to-namespace database --since 2m

Common issues:

• Missing profile label → gateway traffic blocked
• Missing access label → database/S3 traffic blocked
• Wrong profile → external API calls blocked (use internal-egress or standard)

5.5 Verify Monitoring

Use the helper scripts:

# Check deployment health
.claude/skills/deploy-app/scripts/check-deployment-health.sh <namespace> <app-name>

# Check ServiceMonitor discovery (requires port-forward)
.claude/skills/deploy-app/scripts/check-servicemonitor.sh <app-name>

# Check no new alerts
.claude/skills/deploy-app/scripts/check-alerts.sh

# Check canary status (if created)
.claude/skills/deploy-app/scripts/check-canary.sh <app-name>

5.6 Iterate

If something isn’t right, fix the manifests/values and re-apply. This is the dev sandbox — iterate until it works. Update Helm values, ResourceSet configs, network policy labels, etc. and re-deploy.

Phase 6: Validate GitOps & PR

6.1 Reconcile and Validate

Before opening a PR, prove the manifests work through the GitOps path:

# Uninstall the direct helm install
KUBECONFIG=~/.kube/dev.yaml helm uninstall <app-name> -n <namespace>

# Resume Flux and validate clean convergence
task k8s:reconcile-validate

If reconciliation fails, fix the manifests and try again. The goal is a clean state where Flux can deploy everything from git.

6.2 Commit Changes

git add -A
git commit -m "feat(k8s): deploy <app-name> to platform

- Add <app-name> HelmRelease via ResourceSet
- Configure monitoring (ServiceMonitor, alerts)
- Add Renovate manager for version updates
$([ -f kubernetes/platform/config/<app-name>/canary.yaml ] && echo "- Add canary health checks")
$([ -f kubernetes/platform/config/<app-name>/route.yaml ] && echo "- Configure HTTPRoute for ingress")"

6.3 Push and Create PR

git push -u origin deploy-<app-name>

gh pr create --title "feat(k8s): deploy <app-name>" --body "$(cat <<'EOF'
## Summary
- Deploy <app-name> to the Kubernetes platform
- Full monitoring integration (ServiceMonitor + alerts)
- Automated version updates via Renovate

## Test plan
- [ ] Validated with `task k8s:validate`
- [ ] Tested on dev cluster with direct helm install
- [ ] ServiceMonitor targets discovered by Prometheus
- [ ] No new alerts firing
- [ ] Canary health checks passing (if applicable)

Generated with [Claude Code](https://claude.com/claude-code)
EOF
)"

6.4 Report PR URL

Output the PR URL for the user.

Note: The worktree is intentionally kept until PR is merged. User cleans up with:

task wt:remove -- deploy-<app-name>

Secrets Handling

For detailed secret management workflows including persistent SSM-backed secrets, see the secrets skill.

┌─────────────────────────────────────────────────────────────────────┐
│                      Secrets Decision Tree                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  App needs a secret?                                                │
│      │                                                              │
│      ├─ Random/generated (password, API key, encryption key)        │
│      │   └─ Use secret-generator annotation:                        │
│      │      secret-generator.v1.mittwald.de/autogenerate: "key"     │
│      │                                                              │
│      ├─ External service (OAuth, third-party API)                   │
│      │   └─ Create ExternalSecret → AWS SSM                         │
│      │      Instruct user to add secret to Parameter Store          │
│      │                                                              │
│      └─ Unclear which type?                                         │
│          └─ AskUserQuestion: "Can this be randomly generated?"      │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Auto-Generated Secrets

apiVersion: v1
kind: Secret
metadata:
  name: <app-name>-secret
  annotations:
    secret-generator.v1.mittwald.de/autogenerate: "password,api-key"
type: Opaque

External Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: <app-name>-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-parameter-store
  target:
    name: <app-name>-secret
  data:
    - secretKey: api-token
      remoteRef:
        key: /homelab/kubernetes/${cluster_name}/<app-name>/api-token

Error Handling

User Interaction Points

References

• File Templates – Copy-paste templates for all config files
• Monitoring Patterns – ServiceMonitor, PrometheusRule, Canary examples
• flux-gitops skill – ResourceSet patterns
• app-template skill – For apps without native charts
• kubesearch skill – Research workflow