Encrypt Confidential Data At REST

ETCD, the key-value data store used by Kubernetes, supports at-rest encryption to secure data. However, this feature is not enabled by default.

While not all data requires encryption, storing sensitive information like secrets without proper encryption can lead to security vulnerabilities. Ensuring that confidential data is encrypted is a critical step toward protecting your Kubernetes environment.

This blog will guide you through enabling encryption for sensitive data, such as secrets, in the ETCD datastore.

CREATING A GENERIC SECRET

To understand how secret data is stored in the ETCD datastore, let’s start by creating a generic secret.
In this example, we’ll create the secret in the default namespace:

kubectl create secret generic my-secret --from-literal=key1=supersecret

Note: Kubernetes secrets are encoded in Base64 by default. While this obscures the data, it does not provide actual encryption. Anyone with access to the secret can easily decode it.

CHECK HOW THE DATA IS STORED IN ETCD

Note: You should have ETCDCTL installed in your system before getting access to the ETCD pod using command line. To install use this command.: apt-get install etcd-client

Use the below command

ETCDCTL_API=3 etcdctl \
>    --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
>    --cert=/etc/kubernetes/pki/etcd/server.crt \
>    --key=/etc/kubernetes/pki/etcd/server.key  \
>    get /registry/secrets/default/my-secret | hexdump -C

Ensure the required certificates and keys are available in the specified paths. If they are located in a different location, update the command accordingly.
Replace my-secret with the name of your secret and default with your desired namespace if needed.

The output of the command will reveal that the secret value is stored in plain text without any encryption. You can observe this in the bottom-right section of the output, where key1..supersecret is clearly visible. This highlights the importance of enabling at-rest encryption to protect sensitive data.

ENABLING ENCRYPTION in ETCD

To secure sensitive data in the ETCD datastore, we can configure encryption by specifying an encryption-provider-config in the kube-apiserver manifest file and passing the configuration file's location.

Check if enabled by default.

Run the following commands to check whether encryption is enabled:
```
  ps -aux | grep kube-api
  ps -aux | grep kube-api | grep "encryption-provider-config"
```
If encryption is not enabled, the second command will return no results.
Generate a 32-Byte Encryption Key
```
  head -c 32 /dev/urandom | base64
```
Copy the generated key, as you will need it for the configuration file.

Create the Encryption Configuration File

Create a configuration file named enc.yaml in the /etc/kubernetes/enc directory:

  mkdir -p /etc/kubernetes/enc
  vi /etc/kubernetes/enc/enc.yaml

Add the following content to the file:

  ---
  apiVersion: apiserver.config.k8s.io/v1
  kind: EncryptionConfiguration
  resources:
    - resources:
        - secrets
        - configmaps
        - pandas.awesome.bears.example
      providers:
        - aescbc:
            keys:
              - name: key1
                # See the following text for more details about the secret value
                secret: <BASE 64 ENCODED SECRET>
        - identity: {}

We are using the aescbc encryption algorithm. Other algorithms are available, but aescbc is commonly used.
The providers section defines the order of encryption methods. The first provider is used for encryption. If identity is listed first, no encryption will be applied. Ensure that identity is not the first provider.
Replace <BASE64_ENCODED_SECRET> with the 32-byte Base64-encoded key you generated earlier.

Update the Kube-API Server Configuration
Edit the kube-apiserver manifest file:
```
  vi /etc/kubernetes/manifests/kube-apiserver.yaml
```
Add the following flag to the kube-apiserver command:
```
  - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
```

Configure Volumes and Volume Mounts
Since we are adding a custom configuration file, update the volumes and volumeMounts sections of the manifest file:

Volume Mount Section:

  # Add this is volume mount section
  - name: enc                           
      mountPath: /etc/kubernetes/enc      
      readOnly: true

Volumes Section:

    # Add this to volumes section
    - name: enc                             
      hostPath:                             
        path: /etc/kubernetes/enc           
        type: DirectoryOrCreate

Save and Restart
Save the changes to the kube-apiserver manifest file. Kubernetes will automatically detect the changes and restart the kube-apiserver. Allow a few minutes for the restart process to complete.

VALIDATING ETCD ENCRYPTION

After enabling encryption, follow these steps to validate that it is working correctly:

Verify Encryption is Enabled
Use the following command to check if the encryption-provider-config flag is set in the kube-apiserver process:
```
  ps -aux | grep kube-api | grep "encryption-provider-config"
```
If the flag is present, it confirms that the encryption configuration is applied.

Create a New Secret
Create a new secret to test the encryption:

  kubectl create secret generic my-secret2 --from-literal=key1=supersecret --from-literal=key2=topsecret

Inspect the Secret in ETCD
Run the following command to retrieve and view the stored secret in the ETCD datastore:

  ETCDCTL_API=3 etcdctl \
  >    --cacert=/etc/kubernetes/pki/etcd/ca.crt   \
  >    --cert=/etc/kubernetes/pki/etcd/server.crt \
  >    --key=/etc/kubernetes/pki/etcd/server.key  \
  >    get /registry/secrets/default/my-secret2 | hexdump -C

The secret data should now be stored in encrypted form. You will no longer see the plain text values (key1=supersecret, key2=topsecret) in the output.

Rotate Existing Secrets
To encrypt previously stored secrets in ETCD, use the following command to reapply all secrets with the updated encryption configuration:
```
  kubectl get secrets --all-namespaces -o json | kubectl replace -f -
```
This ensures that all existing secrets are encrypted according to the new configuration.

CONCLUSION

Encrypting sensitive data held in ETCD is paramount in keeping your Kubernetes cluster confidential as well as maintaining its integrity. It is CCIM compliant with the added bonus of checking resources like secrets against unauthorized accesses. By properly configuring and validating sensitive information, practitioners are able to greatly mitigate a potential breach on the ETCD datastore.

Take note, the secrets are meant to be rotated ceaselessly while access control as well as certificates management best practices are followed to ensure a secure and healthy Kubernetes environment.

Encrypt Confidential Data At REST

CHECK HOW THE DATA IS STORED IN ETCD

Use the below command

ENABLING ENCRYPTION in ETCD

VALIDATING ETCD ENCRYPTION

CONCLUSION