Encrypting Helm secrets with Mozilla SOPS and AWS KSM

One of important points we need to consider when using Helm is managing secrets. There’s many ways we can do that, using Hashicorp Vault, the AWS Parameter Store…

I find the combination of Mozilla SOPS and AWS KMS a great way to keep secrets as code safely and easily.

AWS KMS

AWS KMS allows us to create and manage cryptographic keys. There are different options of symmetric and asymmetric ones. For this example we will use a symmetric KMS key.

Creating an IAM role

I use Jenkins for CI/CD, with different IAM credentials to access different accounts on AWS. There’s also humans who edit the secrets and push them to repos.

When I was first trying SOPS and KMS and found a problem when encrypting/decrypting secrets depending on who, Jenkins or user, had encrypted it in the first place. The solution was adding a role that is allowed to use the KMS key. That role is used by both Jenkins and real life users.

For this I created a role with no permissions whatsoever and added the right AWS accounts to the Trusted entities list so my other users belonging to all my different accounts can assume it.

Take a note of the ARN, you will need it later.

Controlling who can assume the role

To control who can assume the role I created the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "123",
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRole"
      ],
      "Resource": [
        "arn:aws:iam::yourAWSAccount:role/yourRoleName"
      ]
    }
  ]
}

You can attach it directly to the users or use groups. Whatever works best for your situation.

Creating a symmetric key

I’m using some sample values here. Replace them with your choices.

  1. Create key
  2. Configure key
    • Key type: Symmetric
    • Advanced options:
      • Key material origin: KMS
  3. Create alias and description
    • Alias: helm-secrets
    • Description: This is used to encrypt Helm secrets
  4. Key administrators:
    • Choose the best options for your case here.
  5. Define key usage permissions:
    • Add the role you created earlier.
Take a note of the ARN, you will need it later.

SOPS

SOPS is an editor of encrypted files developed by Mozilla. It supports YAML (ideal for our values files), JSON, ENV… and it also supports encryption with AWS KMS, GCP KMS, Azure Key Vault and PGP.

Demo from SOPS’ GitHub page

Installing SOPS

You can find the latest stable release on the project’s GitHub page.

SOPS configuration file

We should now create a .sops.yaml config file at the root directory of our project. This will tell SOPS which KMS to use, which part of the yaml file we want to encrypt and the path those rules will apply to.

I find the possibility of encrypting only one part of the file is something extremely interesting. It allows for only the passwords and other secrets to be encrypted, while the rest of the values file is easily readable, for example, to review it before merging.

The config file would look like this.

---
creation_rules:
  - path_regex: values.*.yaml$
    kms: 'arn_of_your_kms+arn_of_the_role'
    encrypted_suffix: secrets

Going line by line:

  • path_regex: values.*.yaml$
    • The rules only apply to files matching this regex
  • kms: 'arn_of_your_kms+arn_of_the_role'
    • ARN of the KMS key you created + ARN of the role you use to access it.
    • You need to include the + symbol.
    • Don’t leave spaces between the ARNs and the symbol.
    • If you are not using a role you don’t need to include the second part, just use the arn of your KMS key.
  • encrypted_suffix: secrets
    • SOPS will only encrypt the secrets section of the yaml file

Encrypting our values file

Let’s create a sample values file:

minReplicas: 5
maxReplicas: 20
targetCPUUtilizationPercentage: 50
image:
    name: someImageName
    tag: v1.2.3
config:
    open:
        SOME_API_ENDPOINT: http://api.example.com/svc/some-service/
        DB_NAME: mydatabase
        DB_HOST: somedbhost
    secrets:
        DB_USER: mydbuser
        DB_PASSWORD: mysupersecretpassword

Since we’ve set a rule to only encrypt secrets everything else will be kept in plain text. We can run the following commands:

  • sops -e values.example.yaml
    • This will encrypt the file and print the result (but not save it)
    • You can redirect the output to a file
  • sops -e -i values.example.yaml
    • This will encrypt the file and save it with its original extension

Once the file has been encrypted it will look like this:

minReplicas: 5
maxReplicas: 20
targetCPUUtilizationPercentage: 50
image:
    name: someImageName
    tag: v1.2.3
config:
    open:
        SOME_API_ENDPOINT: http://api.example.com/svc/some-service/
        DB_NAME: mydatabase
        DB_HOST: somedbhost
    secrets:
        DB_USER: ENC[AES256_GCM,data:+LkJ1nV95lzCkrVbk4JNGH5HhH2R3WN9hmpdLFd1K6Q2Yxz5ZkDEPC1jogO2uXNPQONvN5LrS5VSytx14==,type:str]
        DB_PASSWORD: ENC[AES256_GCM,data:67oWHhA2l3kvzL8PCCbsDzYVeuSNTVo9p9CeFDIRlFkx39JIzOByWXhTWIxzawIXqO1WkwRPsreUXTRH==,type:str]
sops:
    kms:
    -   arn: arnOfMyKMS
        role: arnOfMyRole
        created_at: '2020-10-17T21:34:08Z'
        enc: wq4htZxKTIN0pu4r8DxYuyEfkmpRUzmy4uqSuv9GSIyWzARMXqyUhEm2qYAgFk2k6qzVuY2bdxLXoTbw7mIT1VvmtbKVCweJp6RlfGqb9AbKwl6HKNKwQs3Dnd2cPFNkB3G0KpIXC4RiIwDtzzZI6D9hqvxFphljGh1eDCkxrd3NG5mcKHDiivUBe7te64L8zQLL0kLW3Bm7rtSGVz6eVlSWs6kQnLviaXLWrdghlP94Apzk==
        aws_profile: ""
    gcp_kms: []
    azure_kv: []
    lastmodified: '2020-10-17T21:34:09Z'
    mac: ENC[AES256_GCM,data:/Qmx0X86psXm8eCnfoZLGWGP8bGlRBF8QvCYz9tPE7XkiQ15HAY9SM4ZdbY79RBfTvAuyazcUfQcC9WaC/KlwsTtUqqXpZUbBiA9F8RRdU0vhrAg6ChmWuoUR98yVbUOgUYXm049J5BDbe5RrgiKg0NuRmrRtO8Y0L/5DmwBc2AqcZWnztf30OkbU089aklnVMZhS8JaHAcRqLUUM61SUn9EzG1QqsDF0QaIhf2cCssyDITJFEr==,type:str]
    pgp: []
    encrypted_suffix: secrets
    version: 3.5.0

As you can see, only the entries under secrets have been encrypted and SOPS has added some metadata to the file.

Decrypting our values file

To decrypt a file simply run:

  • sops -d values.example.yaml (output on screen)
  • sops -d -i values.example.yaml (decrypt the file and save it)
Little extra. Running sops fileName will open the file in your preferred editor (vim, nano…) and will encrypt it automatically when you save and close it.

Creating the secrets with Helm

Now that our secrets are safely encrypted it’s time to use them with Helm. A sample template could be:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "my-chart.fullname" . }}
  labels:
    {{- include "my-chart.someLabels" . | nindent 4 }}
type: Opaque
data:
{{- range $key, $value := .Values.config.secrets }}
  {{ $key }}: {{ $value | b64enc | quote }}
{{- end -}}

How you decrypt the values file to use it with Helm depends on your what your process is. Zendesk have developed helm-secrets, but I have never used it so I cannot comment. I manage decrypting and cleaning up as part of the Jenkins job that handles the deployments.


Thank you very much for joining me again! How do you manage your secrets? Please, let me know in the comments below, and if you have any questions, go ahead and ask.

Leave a Reply

Your email address will not be published. Required fields are marked *