GitOps with GitLab: How to tackle secrets management

It is possible to use GitLab as a best-in-class GitOps tool, and this blog post series is going to show you how. These easy-to-follow tutorials will focus on different user problems, including provisioning, managing a base infrastructure, and deploying various third-party or custom applications on top of them. You can also view our entire "Ultimate guide to GitOps with GitLab" tutorial series.

In this article we will use our cluster connection to manage secrets within our cluster.

Prerequisites

This article assumes that you have a Kubernetes cluster connected to GitLab using the GitLab Agent for Kubernetes. If you don't have such a cluster, I recommend looking at the linked articles above so you have a similar setup from where we will start today.

A few words about secrets management

The Kubernetes Secret resource is a rather tricky one! By design, secrets should have limited access and should be encrypted at rest and in transit. Still, by default, Kubernetes does not encrypt secrets at rest and accessing them might not be restricted in your cluster. We will not go into detail about how to secure your cluster with respect to secrets in this article. Instead, we want to focus on getting some secrets configured in your cluster with a GitOps approach.

Managing secrets with GitOps means you store those secrets within your Git repository. Of course, you should never store unencrypted secrets in a repo, and some security people are even reluctant to store encrypted secrets in Git. We will not be that worried, but you should consider if this is an acceptable risk for you. There is an alternative we'll talk about, below, if you prefer to not manage your secrets in Git.

There are a few benefits of Git-based secrets management:

• you get versioning by default
• collaboration is supported using merge requests
• as secrets are in code, you push responsibilities towards the development team
• the tools used are well-known to developers

Secrets management with GitLab

When it comes to secrets, Kubernetes, and GitLab, there are at least 3 options to choose from:

• create secrets automatically from environment variables in GitLab CI
• manage secrets through HashiCorp Vault and GitLab CI
• manage secrets in git with a GitOps approach

Create secrets automatically from environment variables in GitLab CI

The Auto Deploy template applies every K8S_SECRET_ prefixed environment variable into your cluster as a Kubernetes Secret. Later, your applications can reference these secrets. This approach is the simplest to use, especially if you would like to use Auto DevOps. We will look into it in a future article.

While simple to use, with this approach your secrets are stored in the GitLab database, instead of Git. That means you lose versioning of the secrets, you need Maintainer rights to modify these secrets, and you lose the ability to approve a change of secret in a merge request.

Manage secrets through HashiCorp Vault and GitLab CI

GitLab CI/CD integrates with HashiCorp Vault to support advanced secrets management use cases. You can combine the K8S_SECRET_ prefixed use case even with Vault-based secrets, and have the secrets applied automatically.

With this approach, you get the all the benefits of HashiCorp Vault, but there is a question: why do you move secrets from Vault to GitLab just to move them to your cluster instead of retrieving the secrets directly from within your cluster? We recommend leaving GitLab out of this flow if you don't have a really good reason to provide secret access to GitLab too! Vault has really great Kubernetes support, thus retrieving secrets directly should be feasible.

Manage secrets in Git with a GitOps approach

To manage secrets in Git, we will need some kind of tooling to take care of the encryption/decryption of the secrets. In this article, I will show you how to set up and use Bitnami's Sealed Secrets, but you can try other tools, like SOPS too. We will look into Bitnami's approach as it targets Kubernetes exclusively, unlike SOPS that supports other use cases too, and might need a bit more setup for Kubernetes.

Bitnami's Sealed Secrets is composed of an in-cluster controller and a CLI tool. The cluster component defines a SealedSecret custom resource that stores the encrypted secret and related metadata. Once a SealedSecret is deployed into the cluster, the controller decrypts it and creates a native Kubernetes Secret resource from it. To create a SealedSecret resource, the kubeseal utility can be used. kubeseal can take a public key and transform and encrypt a native Kubernetes Secret into a SealedSecret, and kubeseal can help with retrieving the public key from the cluster-side controller too.

Setting up Bitnami's Sealed Secrets

As the GitLab Agent supports pure Kubernetes manifests to do GitOps, we will need the manifests for Sealed Secrets. Open the Sealed Secrets releases page and find the most recent release (Don't be fooled by the helm releases!). At the time of writing this article, the most recent release is v0.16.0. From there you can download the release yaml, if your cluster supports RBAC, I recommend the basic controller.yaml file.

• Save and commit the controller.yaml under kubernetes/sealed-secrets.yaml

Push the changes and wait a few seconds for them to get applied. Check that they got applied successfully using: kubectl get pods -n kube-system -l name=sealed-secrets-controller

Retrieving the public key

While the user can encrypt a secret directly with kubeseal, this approach requires them to have access to the Kube API. Instead of providing access, we can fetch the public key from the Sealed Secrets controller and store it in the Git repo. The public key can be used to encrypt secrets, but is useless for decrypting them.

kubeseal --fetch-cert > sealed-secrets.pub.pem

How to avoid storing unencrypted secrets

I prefer to have an ignored directory within my Git repo. The content of this directory is never committed to Git, and I put every sensitive data under this directory.

mkdir ignored
cat <<EOF > ignored/.gitignore
*
!.gitignore
EOF

Continue with setup - not needed if we use a box

Now, you can create sealed secrets with the following two commands:

echo "Very secret" | kubectl create secret generic my-secret -n gitlab-agent --dry-run=client --type=Opaque --from-file=token=/dev/stdin -o yaml > ignored/my-secret.yaml
kubeseal --format=yaml --cert=sealed-secrets.pub.pem < ignored/my-secret.yaml > kubernetes/

The first command creates a regular Kubernetes Secret resource in the gitlab-agent namespace. Setting the namespace is important if you use Sealed Secrets and every SealedSecret is scoped for a specific namespace. You can read more about this in the Sealed Secrets documentation.

The second command takes a Secret resource object and turns it into an encrypted SealedSecret resource. In my case, the secret file:

apiVersion: v1
data:
  token: VmVyeSBzZWNyZXQK
kind: Secret
metadata:
  creationTimestamp: null
  name: my-secret
  namespace: gitlab-agent
type: Opaque

got turned into:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-secret
  namespace: gitlab-agent
spec:
  encryptedData:
    token: AgC1m/D1UwliKD3C2QSv/g+zBi1qGz1YTLZfqnl5JJ4NydCatKzsp8LZr2stIlkwcS3f2YAo/ZIq1OUhOgSgkuNMwVdqsBx1zq7Z3xpGLMIMe7B3XhQ+ExWwqgrm1dTiTDHaH9eXsZWaNsruKQU0F8oGxgLfO/axEZeGWd4WngZRaed9B43dy2k05B6fZnxmwtUVSpr86MO52fX06/QdbvB8MZTrYb7qFuL14U0IDvdFl4l8sPl2rrXsriKg0fJHIV6XtlCwPpQGozTZTUX8nbvU0yXothBzPbaIUfXseFqaW8i/i0Ai+aKhWQAjPGooVAXGwKsuve16DxZ6GJPp1ymR1cEsBkEPlYKbVCKtH5VuptCYZuTXMM6OEPzjFabaIMIUVkkciHlUMcpKFfPnpf7XbBNqZCAKjt//9L99gc48dJRyO4pCrcpFnv6287d65UGnWjmcUJNQNBhEuh9k4esfEZuBNiYIz3Ouz7Wg5HQoT6v3i3J1X5LluWEcTK1G10T7UN+QrnklH4yUtx35yLp83B5/TGICo0Yq1QnARNbKhL5EXuwAO427XO65zzJ3Lh2ymUfrBY3bHO8NW4ykO7ZNDRdj/fsge1J8k4yaxeimQapDKs4XMhoNnKqUNPQYaiQzNPRoj9JwMvtvOH+WLJqEXHIc8RooWGkdo/SB7zp3q7OuHk6HRJM+AQVP3t0r3A1bVhHonUGlv1ApduM=
  template:
    metadata:
      creationTimestamp: null
      name: my-secret
      namespace: gitlab-agent
    type: Opaque

Just commit the SealedSecret and quickly start to watch for the event stream using kubectl get events --all-namespaces --watch to see when the sealed secret is unsealed and applied as a regular Secret.

Utility scripts

If you found the kubeseal command above to be quite complex, you can wrap it in a script.

• Create bin/seal-secret.sh with the following content:

#!/bin/sh

if [ $# -ne 2 ]
  then
    echo "Usage: $0 ignored/my-secret.yaml output-dir/"
    echo "This script requires two arguments"
    echo "The first argument should be the unsealed secret"
    echo "The second argument should be the directory to output the sealed secret"
  exit 1
fi


SECRET_FILE=$(basename $1)

kubeseal --format=yaml --cert=sealed-secrets.pub.pem < $1 > "$2/SealedSecret.${SECRET_FILE}"

echo "Created file $2/SealedSecret.${SECRET_FILE}"

This script takes a path to a vanilla Kubernetes secret and an output directory, and tranforms your Secret into a SealedSecret.

Winding it up

In this article, we have seen how you can install Bitnami's Sealed Secret into your cluster and set it up for static secrets management. Please note the installation method provided here works for all the other 3rd party, off-the-shelf applications that can be deployed using Kubernetes manifests only.

What is next?

In the next article, we will see how you can access a Kubernetes cluster using GitLab CI/CD and why you might want to do it even if you aim for GitOps.

Click here for the next tutorial.