How I Built a Shift-Left DevSecOps Pipeline for FastAPI & Kubernetes (And Caught 3 Real High-Severity CVEs with Trivy) published: true

When an e-commerce platform runs a flash sale, traffic can jump from 10 requests per second to 10,000 in under a minute. If your team hasn't tested how your containers and Kubernetes pods behave under a sudden traffic surge—or how Kubernetes self-heals when a pod returns HTTP 503—production is the worst place to find out.

Even before code reaches a Kubernetes cluster, there is a second problem: most CI pipelines only check if the code runs, not if the container image is secure. Warning-only security scans get ignored, leaked secrets hide in Git commit history, and default base images quietly ship with unpatched vulnerabilities.

To solve both problems, I built CloudPulse—a two-repository High-Availability Cloud Status & Incident Telemetry Platform designed around strict shift-left DevSecOps gates and GitOps delivery:

• Part 1 (App & DevSecOps CI): cloudpulse-devsecops-service — FastAPI telemetry & chaos microservice, multistage Docker hardening, and a 3-stage GitHub Actions pipeline.
• Part 2 (AWS EKS & GitOps Infra): cloudpulse-eks-gitops-infra — Modular AWS VPC & EKS Terraform IaC, Argo CD GitOps continuous delivery, Kubernetes HPA auto-scaling, and Prometheus/Grafana observability.

Here is how I architected the pipeline, how our Trivy security gate blocked a real build after catching 3 HIGH severity CVEs, and how I remediated them to reach a zero-CVE container build.

1. Designing a Microservice Built for Kubernetes Chaos Testing

Instead of deploying a static "Hello World" container, CloudPulse exposes real Prometheus telemetry (cloudpulse_http_requests_total, cloudpulse_flash_surge_events_total, cloudpulse_service_health_status, and request latency histograms) alongside two interactive chaos endpoints:

1. POST /simulate-surge: Launches a non-blocking 30-second background task (asyncio.create_task) that injects synthetic checkout traffic bursts into the Prometheus counters to trigger downstream Kubernetes HorizontalPodAutoscaler (HPA) scaling and Grafana alerts.

2. POST /simulate-outage: Flips the internal service health state so GET /healthz returns HTTP 503 Service Unavailable, allowing us to test Kubernetes Liveness and Readiness probe self-healing in real time.

Baseline Healthy State (200 OK)

Active Flash-Sale Traffic Surge

Simulated Server Outage (503 OUTAGE)

2. The 3-Stage Shift-Left DevSecOps Pipeline

Every push and pull request to main passes through three strict gates in .github/workflows/devsecops.yml before any container image can be published to GitHub Container Registry (ghcr.io).

1. Station 1 — Code Quality & Unit Tests (Ruff + Pytest): Lints Python syntax/import formatting and executes automated TestClient tests verifying /healthz, /metrics, surge increments, and 200 <-> 503 outage toggling.

2. Station 2 — Full-History Secret Scanning (Gitleaks): Runs gitleaks-action@v2 with fetch-depth: 0 to scan the entire Git commit history—not just the latest commit—so accidentally committed AWS keys or tokens block the build immediately.

3. Station 3 — Multistage Container Build, Trivy CVE Gate & Smoke Test: Builds the production Docker image, scans both OS packages and Python libraries with Trivy (exit-code: '1', severity: 'CRITICAL,HIGH'), spins up the live container to smoke-test /healthz and /metrics via curl --fail, and pushes to ghcr.io.

3. Real-World Remediation: When Trivy Blocked the Pipeline

Having exit-code: '1' configured on Trivy immediately proved its value on our initial pipeline run.

Even though our code passed all unit tests and used the official python:3.12-slim base image, Trivy failed the build and blocked the image from publishing after detecting three actionable HIGH severity vulnerabilities and vulnerable vendored SBOM metadata:

Additionally, Trivy flagged embedded CycloneDX SBOM files (bom.cdx.json) bundled inside setuptools and wheel in the base Python image that referenced older vulnerable package versions.

How We Fixed It for a 0-CVE Build

First, we pinned starlette>=1.3.1 alongside upgraded fastapi in requirements.txt so transitive dependencies could no longer resolve to vulnerable versions.

Second, we hardened our multistage Dockerfile to patch OS packages, strip unused packaging tools and embedded bom.cdx.json metadata from the runtime image, and execute Uvicorn under a locked-down non-root user (appuser, UID 10001):

# Stage 1: Builder FROM python:3.12-slim AS builder WORKDIR /build RUN python -m venv /opt/venv ENV PATH="/opt/venv/bin:$PATH" COPY requirements.txt . RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \  pip install --no-cache-dir -r requirements.txt && \  find /opt/venv -name "bom.cdx.json" -delete # Stage 2: Hardened Runtime FROM python:3.12-slim AS runtime RUN apt-get update && \  apt-get upgrade -y && \  apt-get install -y --no-install-recommends curl && \  rm -rf /var/lib/apt/lists/* && \  pip install --no-cache-dir --upgrade pip setuptools wheel && \  rm -rf /usr/local/lib/python3.12/site-packages/setuptools* \  /usr/local/lib/python3.12/site-packages/wheel* && \  find /usr/local/lib/python3.12 -name "bom.cdx.json" -delete RUN groupadd -g 10001 appuser && \  useradd -u 10001 -g appuser -s /sbin/nologin -M appuser WORKDIR /app COPY --from=builder /opt/venv /opt/venv COPY --chown=appuser:appuser app/ ./app/ ENV PATH="/opt/venv/bin:$PATH" \ PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 USER appuser EXPOSE 8000 HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \ CMD curl -f http://localhost:8000/healthz || exit 1 CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] 

On the next push (commit 688dfa7), Trivy reported 0 HIGH or CRITICAL vulnerabilities, the container smoke tests passed, and the verified image was published to ghcr.io.

4. Downstream GitOps Delivery & Observability (Preview of Part 2)

Because we separated our application repository from our infrastructure repository following enterprise GitOps practices, our second repository cloudpulse-eks-gitops-infra provisions the AWS VPC & EKS architecture with Terraform and uses Argo CD (selfHeal: true, prune: true) to deploy the verified container.

When we trigger the Flash-Sale Traffic Surge across the Kubernetes pods, Prometheus scrapes the /metrics endpoint and Grafana visualizes the real-time request spike (~300 req/s) and pod readiness.

Explore the Full Source Code

Both repositories—including the FastAPI source, unit tests, multistage Dockerfile, GitHub Actions workflows, AWS EKS Terraform modules, Argo CD manifests, and Grafana dashboard JSON—are open source on GitHub.

• Part 1 (App & DevSecOps CI/CD): cloudpulse-devsecops-service
• Part 2 (AWS EKS Terraform, Argo CD & Observability): cloudpulse-eks-gitops-infra