Running DB migrations on Kubernetes with Helm

You probably have seen this before. Many of your backend services use Postgres, MySQL or some other sort of relational database. Developers are constantly making changes to the applications to improve performance, add new functionality, fix bugs… and sometimes that means making changes to the DB schemas.

The best way to go is using migrations. It’s safer and more reliable. You can track changes, keep them part of the code base… You can know who’s changed what, when, and so on.

Migrations can be run when the applications start. However, when you host them on Kubernetes, there are a couple of problems in doing it in this way.

  1. If the migrations are run when the application starts that means whenever a pod dies and is restarted the process will be triggered again. Even if doesn’t need to migrate anything it’ll need to check anyway.
  2. Same thing when scaling up.
  3. Two or more pods starting at the same time might run the same migration, which can cause problems.

Defining Kubernetes applications with Helm means we can use a very interesting mechanism, the Chart Hooks.

Using hooks allows us to control certain points of the release’s lifecycle. In our example the approach is the following:

  • Use a secrets hook to pass the information to the job that will run the migrations. The reason for this is that hooks are created before any resources, so the job won’t be able to use the “regular” secrets we might create as part of the release.
  • Use a job hook to run the migrations.

Let’s look at the secrets first.

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "some-service.fullname" . }}-migrations
  labels:
  {{ - include "some-service.someLabels"  . | nindent 4 }}
annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "-1"
    "helm.sh/hook-delete-policy": before-hook-creation, hook-succeeded
type: Opaque
data:
  {{- range $key, $value := .Values.config }}
  {{ $key }}: {{ $value | b64enc | quote }}
  {{- end -}}

As a summary. The values are read from the config section of the values file values.yaml. That file is encrypted and only decrypted when needed, but that’s another story. The important bits of this snippet are the annotations:

annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "-1"
    "helm.sh/hook-delete-policy": before-hook-creation, hook-succeeded
  • "helm.sh/hook": pre-install, pre-upgrade
    • pre-install: Executes after templates are rendered, but before any resources are created in Kubernetes.
    • pre-upgrade: Executes on an upgrade request after templates are rendered, but before any resources are updated.
  • "helm.sh/hook-weight": "-1"
    • This sorts the hook’s execution depending on its weight.
    • Default value is 0.
    • They are sorted negative to positive.
  • "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
    • before-hook-creation
      • Delete previous resource before a new hook is launched (default)
      • This is particularly useful if the hook fails and you want to go onto a pod to look at logs, check what the values were…
      • If something fails it won’t be deleted, it will be deleted next time you run the hooks.
    • hook-succeeded
      • Delete the hook when it finishes correctly
      • I like having this so I don’t keep completed stuff lying around in the cluster.

And now the job that runs the actual migrations:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "some-service.shortname" . }}-postgres-loader
  labels:
    {{- include "some-service.someLabels" . | nindent 4 }}
  annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation, hook-succeeded
spec:
  template:
    metadata:
      labels:
        {{- include "some-service.someLabels" . | nindent 8 }}
        app: postgres-loader
    spec:
      containers:
        - name: {{ template "some-service.shortname" . }}-postgres-loader-load
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: Always
          env:
          {{- range $key, $value := .Values.config }}
            - name: {{ $key }}
              valueFrom:
                secretKeyRef:
                  name: {{ template "some-service.fullname" $ }}-migrations
                  key: {{ $key -}}
          {{- end }}
          command:
            - sh
          args:
            - -c
            - yarn db:migrations:run
      restartPolicy: OnFailure
  backoffLimit: 5

The migrations will be run using the same image that we use to run our application. The important bit here is that the entrypoint has been replaced so the application doesn’t start, only the migrations are run.

Breaking the template up a little bit:

  annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "0"
    "helm.sh/hook-delete-policy": before-hook-creation, hook-succeeded

I explained the annotations when we looked at the secrets definition. The only difference is the weight is 0. This means the secrets (-1) were created before the job.

          env:
          {{- range $key, $value := .Values.config }}
            - name: {{ $key }}
              valueFrom:
                secretKeyRef:
                  name: {{ template "some-service.fullname" $ }}-migrations
                  key: {{ $key -}}
          {{- end }}
          command:

The environment variables are pulled from the secrets created by the hook.

          command:
            - sh
          args:
            - -c
            - yarn db:migrations:run
  • command is the Docker entrypoint. As you can see I have replaced it with sh so the default behaviour (starting the application) is ignored.
  • And then we pass the arguments we need to run the migrations. In our case yarn db:migrations:run

And that’s just a very simple example on how you can run migrations using Helm. Please, remember, you’re running them before you’ve deployed your new pods, so for a few seconds/minutes your old pods will be hitting the new version of the schema. Remember to make your migrations backwards compatible.


Thank you very much for reading my blog. I hope this post is useful for you. Please, let me know down below and leave a comment if you have any questions.

Leave a Reply

Your email address will not be published. Required fields are marked *