‘Secrets’ in Kubernetes cannot be stored in a Git repository as that would expose confidential and sensitive information to everyone who can access this repository. ‘Sealed Secrets’ has been developed to address this specific challenge.
When deploying an application in Kubernetes, resources are generally described in a group of yaml files and stored in a Git repository. This works for all of the Kubernetes resources except ‘secrets’, as that exposes sensitive information to everyone having access to the repository.
‘Sealed Secrets’ is a controller developed by Bitnami to solve this particular problem. With Sealed Secrets in place, you can manifest it in your Git repository, which will be automatically decrypted by the controller running in your cluster.
Installation
There are various ways to install Sealed Secrets in your Kubernetes cluster: Kustomize, Helm Chart or via the Operator framework.
I prefer Kustomize as you can just use plain kubectl to perform the installation. Let’s create a kustomization file for Sealed Secrets:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: kube-system resources: - resources.yaml
The resources.yaml file contains the resources related to Sealed Secrets. Let’s apply it:
> kubectl apply -k sealed-secrets/ customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created serviceaccount/sealed-secrets-controller created role.rbac.authorization.k8s.io/sealed-secrets-key-admin created role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created clusterrole.rbac.authorization.k8s.io/secrets-unsealer created rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created service/sealed-secrets-controller created deployment.apps/sealed-secrets-controller created
Usage
Now let’s say that you have the following secret resource:
apiVersion: v1 kind: Secret metadata: name: mysecret namespace: addon-system type: Opaque data: username: c2liaQ== password: bXlzZWNyZXQ=
Both c2liaQ== and bXlzZWNyZXQ= are base64 encoded values of the secrets:
> echo -n c2liaQ== | base64 -d sibi
We want to convert these to Sealed Secrets type so that they can be stored in a public place. Make sure you have the kubeseal executable in your $PATH. You can create a Sealed Secrets manifest like this:
> kubeseal -o yaml < secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: mysecret
namespace: addon-system
spec:
encryptedData:
password: AgBjKaKjdpxf080pWIcUKKbHMOt0p/29miuKEgMzTJsmSQFE4mAgX5S/RAjQYOhwXKQoa1Vhbwou3S379yxy+BiUVZsSeFub8gl TNDx6RPO1UUyo+R2kO88qGT1ULqiB39+/INS1u68R18Q2MzlnpP+anHDtCds0twMs51 aSbuBDXvkws6TGfxjTmttgCeYBOSfsZ33rsgDCesPAxxsSf2y WcdofAWvDIh5QH5HDIdveZ8qsL3mPVq9JpDrwvxdArLqPARu3uV8GHVA 5dAH9i92vhEbsEQZmayz/4l8128AwZ8ZuUmIKH/SIwHzqbMOnb+2yxCqRw bBRcpq7pd4ygJ+Zph9P4tyl+YQ9YBgHBqUQhRISZLrJURbdcTAZlUr WIGOenVcEsXOUj9AscjveE4sy+PDgsOdm5TOO/wfUinAqV5Ncc1Swc1YvrVqn6A0 J6sSeHdA3zblYtQZhL9OwfPj1nOyWTvRhnqhh+8SHODJnrrBBf 0h73qg1 FXJmAK46qx P5ZQcUQG0CxxPj7S1RYPk5QjakOMTOkh36UxZ0nwqq G8c7V5RVUNvbkuVWi9Lp8ilPSkBHAYt9agoB/+iJzI64jbMGMqdP7pL1OPmZ2 FsI2yaLAlrkYJy03sibiF3VgA5v8Ox1OeTa2ux9VjaedkMSHIbcTo0ewDW kXzc5iY3qH7bMM0AvlZLDjy1DCayzk/iYOWRZD3Cgw==
username:
AgAq/gkwK04XAViAuNGNnApJKQIGzn9V4yf1ziLj+/vXtCu37bj27nei46RS 1n5SZ/1AC1YM2lLYScW+ZB7oZDK2bXNFGYxv9XgA6t0 3+vgnN1v07VxrdDDI kpUeAizddfrmQfUzL2al8Erm3DZzCiGio6ZLrVfg52trh tMeKZqqamewkQ0n ZkTPNGKgWaAu2X8RtSD2CbQrWm3DJQ4z9gwqXl9gTraMJl 3xcMs7BynIfliE5yuERlslVlDQgD+ZHvwP5V1aDX1g3H20is4iAhP0qZzy0dNf/Lc11yFmrwZHl2trIGtedBtllQBo6PkYsqNbrbEQheMoiAn+djR2i191lS7LNw6 9N4w4VMGwc8VDcBoXXwd17iczyjhCn9G4q4b0Pj7Mw6mcpmdirGMYXSbDk0JGui
FwJeXVl/ql00OCOYp8PR2c/fimQi98WpUrfAa+jkqzw0StJYl/tmdiSNIqulf7jiObspi8ujuNB1/5FgHcxPLw7sZBRXINY83CQitNv+NOW+8qt
gAAX1jyNPEAj/25tPbFaNbml+f+x/naMG3Xwbl4a9QFPkdUqnGrIYPxAllEQ/
mNdmzbQYAS7ZFq2CU/iYPH5Jh4iNCxdZ+AS9EQG9L/oVXGnq3NsVCzT35C SORD+tFo55D5ebzi+UiOfy/Gl3n/QiJJZ2V3YxdnxbQsf73KR4VGMObf24NriIBfOx+a
template:
data: null
metadata:
creationTimestamp: null
name: mysecret
namespace: addon-system
type: Opaque
Now you can have the above yaml file as part of your repository and even share it publicly without worrying about any leak. Let us save this yaml file as sealed-secret.yaml and apply the above manifest:
> kubectl apply -f sealed-secret.yaml sealedsecret.bitnami.com/mysecret created
You can observe that your secret will get populated by the controller in a couple of seconds:
> kubectl get secrets -n addon-system | rg mysecret NAME TYPE DATA AGE mysecret Opaque 2 69s
You can even check the kubeseal controller logs to verify that it unsealed your secret:
> kubectl -n kube-system logs -f pods/sealed-secrets-controller-78476684bf-jfbsp ... 2021/12/03 14:44:13 Updating addon-system/mysecret
Let’s verify that our secrets are indeed what we stored:
> kubectl -n addon-system get secret mysecret -o=jsonpath={.data}
{“password”:”bXlzZWNyZXQ=”,”username”:”c2liaQ==”}
And that confirms that the unsealing did happen properly!
You can also save the certificate offline and generate Sealed Secrets without having access to the controller. Let’s fetch the certificate:
> kubeseal --fetch-cert
The output will give the complete certificate, which can be stored locally:
> kubeseal --fetch-cert > sealed-secrets.pem > file sealed-secrets.pem sealed-secrets.pem: PEM certificate
Now let’s create the same Sealed Secret using the above sealed-secrets.pem:
❯ kubeseal --cert ./sealed-secrets.pem -o yaml < secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: mysecret
namespace: addon-system
spec:
encryptedData:
password: AgCsa+ihNMcSpMrPrWnRO0VZQ6XZ0bl0crgOqQDTsimODeBW
t5hHiGk0utl0zDyvOCofziWBf3uQpoMsB+MZCquBL7sF7FQiNLFK5EbZfWV3Gr
kT3DN9YNyjDYunvy7TM9qlgJGLtHhQt3gTRwRozz2nz6UfJ92LI4mRfdm2eHTa
eTQy230pZJPp1Kw7bQWbrEq6rOkHbzQf3vsORlLUnmQO9DeJJM4cHddG7szIs
qUEeDJ6+uOMErmjOTvgPIoy1VVFnjcs9U5zdIVzsh6H28TErmcm+3krn+8DP/
ae0KpT+1WpSpmTsNU4EcXwYQHOZFP1eP0YBYUJ5yreg7Fu/LAOqlPs6GUWYc0
hN9bRZsPwcC5TfahO1nW8hvrtBjY9Zw0iTDNZQGcmcFE2BAQaQ/ri8VW9U8t+
yXWx7BQlmM071majTMBH9/d245FxUj4jtJ4NVE+wnk3fYxzNF0zChxni72EXW
Uww3PeI/QfSUfpqceMGnpbRud+qa1ycp24n6C6+pnqYQX+xGGBU6oLx5IK6l4
0eOnQbZzx2Qv6kH4Q8zLLqiIuF86FvXYP2St3FdzBZWecHK2uJuyR83gRDVn9
4Ksxv6wbALv5T6o+8AeQiSfdcE2gBVZbv20tmr0IrMzs961GSyT7LFLd+ujo
HyORj+c1yUYAUzsRksYI4wFokcwoweRPC+ZP0cK+XRYyzX5CErgl/s6KNYQ==
username: AgCrtM/W5hJd8Ihg5GCh0lKnUK/rhqU/4fNzKCVcnF98w1C+
VIy9M33sx2xGe8U4wqeYs1aPgPeqWznudHMGoLIuMYCt4X89dAF06o7w/Hw0QHu /CbFFo3gToMj/V8uPRirhg09bCCyXQfmamVLYkGms JWMLHs3uf9LATxpXZP35j EZvtaTd9uk4+oLVzc886R5H8ltFdwj myPHCM76gXUSobomIMgds7EbHL6zYdbAOS1GOn25XZ4uCqIA ThtAcKM5njcm9dNlgL2LNaabPrR9wqug8vm+jjoJV59sBJaghKLyuuUH4vyaiqnfWh sFnCnnucTwGlh/bQPU5xg+MSJgeE/aSvBhF/7syv1d+JXuQbdW1iQDjKE/rjsVC5OHTd/x9/RyghyT0xDlUKTiy/xkzrtfQN3UOBwVGTqSPB5OUw1Rw0WVXhM60tB0I+7OUxA1xtusYCwJKiJs7S7r0tMQ7MalS1o4/oiEm1fO/rdxGiYD1KyaELLVDbudWKO9KXZ2lBgyplXqpdO3IuW0owGO
dWkBpn0XhQZ2SIKjec1boJ4GSIvEl8xQ5XExfzBenYyXoexQETsyyII38zLJ
kFACygJrh4SO1W6wixXWm9MAvDbySPxvKUHWgIuvHr5jTd+1N/kPy2Noz/GFH7WYpIrPgoZfpFH/fJgjgvJEAzTad+PI+S74JA1VyQICO08ssCEkrS75
template:
data: null
metadata:
creationTimestamp: null
name: mysecret
namespace: addon-system
type: Opaque
The above workflow will allow any of your teammates to create new sealed secrets without interacting with the cluster credentials. In fact, you can store the certificate publicly in a repository to allow others to create sealed secrets.
How it works
The underlying principle of Sealed Secrets is the usage of public key cryptography. The public certificate is used for sealing secrets. The private key the controller has is used for decrypting the sealed secrets.
Note that there are other implementation details that govern how sealed secrets work:
- The name space is used during the encryption process by default. Thus, two same secrets on different name spaces will have a different set of encrypted data within it.
- The secret name is also used during the encryption by default. This design decision disallows Sealed Secrets resources to be renamed. This improves the security as RBAC (role based access control) can enforce limitation of secret access by name and Sealed Secrets follows it by disallowing the secret to be renamed.
The above default behaviour can be configured. There is a concept of ‘scopes’ which has these rules:
- strict scope requires both name space and secret name as part of the encryption process.
- namespace-wide scope requires only a secret name as part of the encryption process.
- cluster-wide doesn’t require either.
You could pass the —scope command line argument to the controller if you want to customise it.
By default, sealing keys are automatically renewed every 30 days. Note that this doesn’t mean you have to re-encrypt your secrets every 30 days. All it means is that the controller will generate a new set of key pairs after 30 days. Any new sealed secret should be generated via the new certificate. But the old sealed secrets will continue to function since the old keys are not deleted — they are kept around and the new keys are just appended to them.
