Stop Running Database Migrations in initContainers

How we moved Flyway into a Helm hook that runs once, before the deploy, and works the same in Argo CD

Our app ran 6 replicas. On every deploy, 6 pods tried to migrate the same MySQL database.

We had put Flyway in an initContainer. It looked clean: the migration runs before the app starts, so the app always sees the right schema. It's a common pattern, and for a single pod, it's fine.

With 6 replicas, it wasn't.

Here's why we moved away from it, the Helm hook we replaced it with, and what happens when a migration fails.

We didn't want to write this from memory. So we rebuilt the setup on a local kind cluster with MySQL, Helm, and Argo CD, reproduced the problems we'd had in production, and then tested the fix. Every YAML file and every error message you'll see below comes from that test setup.

Our example app is a Spring service on MySQL. Nothing here is Spring-specific, though. It works for any app that uses Flyway.

Why the initContainer had to go

1. Every replica ran the migration.

Not once per deploy. Once per pod. And it wasn't only deploys: every restart, every rescheduled pod, and every HPA scale-up ran Flyway again.

Flyway protects itself with a database lock, so the pods didn't corrupt anything. They waited for each other instead. That leads to the second problem.

2. New pods waited on the database lock.

If one pod is running a slow migration, every other new pod waits for the lock before its app can start. That includes pods the HPA adds because traffic just went up.

In our test setup, we reproduced it: a pod with nothing to migrate started 5 seconds after another pod began a 40-second migration. It waited 36 seconds before it could finish, even though it had no work to do.

Flyway itself isn't slow. With nothing to migrate, it finished in about a second in our tests. The problem is that scaling now depends on the database being free.

3. Failures were hard to see.

When a migration failed, we didn't get one clear "migration failed" step. We got pods stuck in Init:CrashLoopBackOff. Someone had to find the right pod and read the init container logs to see why.

The fix: one Job, before the deploy

We wanted the migration to run once per release, not once per pod. Kubernetes already has a tool for that. It's called a Job.

Helm can run a Job as a hook before it touches anything else in the release. If the Job fails, Helm stops the upgrade, and the old pods keep running.

The setup has three parts:

A small migration image with the SQL files inside
A Job with Helm hook annotations
The app, with its own Flyway turned off

Part 1: The migration image

Bake the SQL files into an image built on the official Flyway image:

FROM flyway/flyway:11 COPY sql /flyway/sql

migrations/ Dockerfile sql/ V1__create_users.sql V2__create_orders.sql V3__add_order_status.sql

Build it and tag it with the same version as the app. The migrations for app version 1.4.0 live in migration image 1.4.0. You don't need volume mounts or ConfigMaps, and you always know exactly which SQL ran in production.

Part 2: The migration Job as a Helm hook

apiVersion: batch/v1 kind: Job metadata: name: {{ .Release.Name }}-db-migrate annotations: helm.sh/hook: pre-install,pre-upgrade helm.sh/hook-weight: "0" helm.sh/hook-delete-policy: before-hook-creation spec: backoffLimit: 0 activeDeadlineSeconds: 600 template: spec: restartPolicy: Never containers: - name: flyway image: "{{ .Values.migrations.repository }}:{{ .Values.migrations.tag }}" args: ["migrate"] env: - name: FLYWAY_URL value: {{ .Values.db.url | quote }} - name: FLYWAY_USER valueFrom: {secretKeyRef: {name: {{ .Values.db.secretName }}, key: username}} - name: FLYWAY_PASSWORD valueFrom: {secretKeyRef: {name: {{ .Values.db.secretName }}, key: password}} - name: FLYWAY_CONNECT_RETRIES value: "10"

What each part does:

pre-install,pre-upgrade: the Job runs before Helm creates or updates anything else in the release, on the first install and on every upgrade.
before-hook-creation: Helm deletes the old Job only when the next release starts. A failed Job stays in the cluster, so you can still read its logs. Don't use hook-succeeded alone; you'll want those logs.
backoffLimit: 0: a failed migration fails once and stops. Retrying a half-applied migration on MySQL won't fix it (more on that below).
activeDeadlineSeconds: a stuck migration doesn't hang the release forever.
FLYWAY_CONNECT_RETRIES: if the database is briefly unreachable, Flyway retries the connection instead of failing straight away.

One thing to check: the Secret with the DB credentials must exist before the hook runs. In our test setup, on a fresh install where the Secret didn't exist yet, the Job sat in CreateContainerConfigError until Helm timed out. The app was never deployed. Create the Secret outside the app chart, or make sure it exists before the release.

Part 3: The app stops migrating itself

Spring Boot runs Flyway on startup by default when Flyway is on the classpath. If you leave that on, you're back to every pod migrating. Turn it off in the Deployment:

env: - name: SPRING_FLYWAY_ENABLED value: "false"

Now your app just connects. By the time it starts, the database is already at the right version.

What a deploy looks like now

The order with helm upgrade --wait is:

Helm creates the migration Job.
Flyway applies any new migrations and exits.
Only after the Job succeeds does Helm update the Deployment.
The rolling update starts, and the new pods start straight away.

We checked this on the test cluster. On the first install, the Deployment was created only after the Job had completed. On an upgrade with a new V3 migration and a new app version, V3 was applied first, then the pods rolled.

New pods from the HPA don't touch Flyway at all anymore.

When a migration fails

This is the part that matters most. So we broke a migration on purpose:

-- V4__broken.sql ALTER TABLE orderz ADD COLUMN note VARCHAR(50);

Helm stopped the upgrade:

Error: UPGRADE FAILED: pre-upgrade hooks failed: resource Job/shop/webapp-db-migrate not ready. status: Failed

Helm never touched the Deployment. Our 3 test pods kept running the old version and kept serving traffic. That's the whole point.

Then we fixed the SQL and deployed again. It still failed:

ERROR: Validate failed: Migrations have failed validation Detected failed migration to version 4 (broken). Please remove any half-completed changes then run repair to fix the schema history.

That's MySQL, not a Flyway bug. MySQL commits DDL statements on its own, so Flyway can't roll back a failed migration. It marks it as failed in its history table and won't go any further until someone looks at it. Flyway's documentation says the same.

The recovery is:

Check what the failed migration changed and undo any half-applied part by hand.
Run flyway repair. On MySQL, this removes the failed row from the history table.
Fix the migration file and deploy again.

After the repair, our next deploy applied the fixed V4 and updated the app.

We'd call that manual step a feature. If a schema change fails on production, you want a person to look at it before anything retries.

The same chart in Argo CD

We deploy through Argo CD, and the chart didn't need to change.

Argo CD reads Helm hook annotations and maps pre-install and pre-upgrade hooks to its PreSync phase. We tested this on the same cluster: Argo CD ran the migration Job as a PreSync hook, and the Deployment was synced only after the Job succeeded.

With the broken migration, the sync failed and Argo CD retried it, while the app stayed on the old version with all pods ready. If you use automated sync, the retries won't fix a failed MySQL migration, so expect the same repair step as above.

So one chart works with Argo CD and with plain helm upgrade from a pipeline.