9
9
Introduction
The Azure Key Vault Provider for Secrets Store CSI Driver facilitates the seamless integration of an Azure Key Vault as a secure secret store with an Azure Kubernetes Service (AKS) cluster through the utilization of Container Storage Interface (CSI) volumes. This guide demonstrates how to implement and use the Azure Key Vault Provider for Secrets Store CSI Driver in AKS clusters. It covers integration steps, benefits, and best practices for secure secret management, including setup, configuration, and practical use cases to enhance Kubernetes application security.
How it works
The diagram below illustrates how Secrets Store CSI volume works:
Just like Kubernetes secrets, upon the initialization and restart of a pod, the Secrets Store CSI driver interacts with the external Secrets Store through gRPC to fetch the secret content specified in the SecretProviderClass custom resource. This content is then mounted within the pod as a `tmpfs` volume, and the secret data is written into that volume.
When a pod is deleted, the corresponding volume is cleaned up and deleted.
Features
While Kubernetes secrets provide a basic mechanism for storing such data, they may not offer the robust security measures required for certain scenarios. Secrets Store CSI Driver’s notable features make it a preferred choice for organizations seeking a robust and flexible secret management solution:
1. Mounts secrets, keys, and certificates to a pod using a CSI volume. Secrets are directly mounted in the pod’s volume so secrets cannot be accessed directly outside the pod.
2. Kubernetes secrets are not secure. Secrets Store CSI Driver uses external secret stores to store the secrets which are more secure.
3. Secrets Store CSI Driver does not store secrets in etcd and supports CSI inline volumes.
4. Supports mounting multiple secrets store objects as a single volume.
5. Supports pod portability with the SecretProviderClass CRD.
6. Supports Windows containers.
7. Syncs with Kubernetes secrets.
8. Supports autorotation of mounted contents and synced Kubernetes secrets.
9. Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support
Solution approach
Implementing the Azure Key Vault Provider for Secrets Store CSI Driver involves a strategic approach to ensure secure and effective deployment within your AKS cluster. Follow these steps for a successful setup:
- Install the driver and providers in either the kube-system namespace or a dedicated namespace. The driver is deployed as a DaemonSet and requires permissions to mount kubelet hostPath volumes and access pod service account tokens. It should be handled as a privileged component, and regular cluster users should not have the authority to deploy or modify it.
- The above point can be ignored if you are installing the CSI driver by enabling azure-keyvault-secrets-provider as add-ons (which we’ll be using in this blog); the AKS handles this on its own.
Prerequisites:
- Check that your version of the Azure CLI is 2.30.0 or later. If it's an earlier version, install the latest version.
- If you're restricting Ingress to the cluster, make sure ports 9808 and 8095 are open.
Steps:
1. Create an Azure resource group using the [az group create] command.
2. Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the az aks create command and enable the azure-keyvault-secrets-provider add-on.
If you are using Azure Portal then the option to enable secret store CSI driver can be found in the `Advanced` options.
Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver support:
Upgrade an existing AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability using the az aks enable-addons command and enable the azure-keyvault-secrets-provider add-on. The add-on creates a user-assigned managed identity you can use to authenticate to your Azure key vault.
If you are using Azure Portal, the same addon can be enabled in Setting -> Cluster configurations.
3. Verify the Azure Key Vault Provider for Secrets Store CSI Driver installation:
Verify the installation is finished using the kubectl get pods command, which lists all pods with the secrets-store-csi-driver and secrets-store-provider-azure labels in the kube-system namespace.
4. Create or use an existing Azure key vault:
Create an Azure key vault using the az keyvault create command. The name of the key vault must be globally unique.
Your Azure key vault can store keys, secrets, and certificates.
In this example, the az keyvault secret set command is used to set a plain-text secret called secretName.
You can also create the Key Vault and Key vault secret using Azure portal.
Note: Azure role-based access control (Azure RBAC) should be configured for the key vault.
RBAC can be found in Access Configuration settings while creating the key vault.
5. Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver:
Access with a user-assigned managed identity-
Find the Client ID of the managed identity created by azure-keyvault-secrets-provider addon.
Use the following command to find the same-
The same can be found through Azure portal in managed identities. The name of the managed identity has a syntax: azurekeyvaultsecretsprovider-<clusterName>
Create a role assignment that grants the identity permission to access the key vault secrets, access keys, and certificates using the az role assignment create command.
6. Create a SecretProviderClass:
You can use the following YAML to create a SecretProviderClass-
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kv-spc
spec:
provider: azure
secretObjects:
- data:
- key: username
objectName: secretName
secretName: mysecret
type: Opaque
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: <Managed_Identity's_Client_ID>
keyvaultName: <Azure_Key_vault_name>
objects:
- |
array:
- |
objectName: secretName
objectType: secret
objectVersion: ""
tenantId: <tenant_ID>
Note:
a) No need to create k8s secret object separately. It will be created automatically when you create a pod with CSI driver volume.
b) The secretObjects block in the above YAML is optional and is only needed if you need to synchronize mounted content with a Kubernetes secret. You will still get the key vault object mounted to the pod if you do not use this block.
7. Set an environment variable to reference Kubernetes secrets:
Use the following YAML to reference your newly created Kubernetes secret by setting an environment variable in your pod:
kind: Pod
apiVersion: v1
metadata:
name: nginx-csi
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets-store"
readOnly: true
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kv-spc"
8. Validate the secrets:
a) Show secrets held in the secrets store using the following command.
b) Display a secret in the store using the following command. This example command shows the test secret secretName.
9. Enable and disable autorotation:
When the Azure Key Vault Provider for the Secrets Store CSI Driver is enabled, it periodically updates the mounted secrets within your pods and the corresponding Kubernetes secrets defined in the secretObjects field of your SecretProviderClass. This update process is initiated based on the rotation poll interval that you have configured. By default, the rotation poll interval is set to two minutes.
When a secret updates in an external secrets store after initial pod deployment, the Kubernetes Secret and the pod mount periodically updates.
a) Mount the Kubernetes Secret as a volume: Use the autorotation and Sync K8s secrets features of Secrets Store CSI Driver. The application needs to watch for changes from the mounted Kubernetes Secret volume. When the CSI Driver updates the Kubernetes Secret, the corresponding volume contents automatically update as well.
b)Application reads the data from the container’s filesystem: Use the rotation feature of Secrets Store CSI Driver. The application needs to watch for the file change from the volume mounted by the CSI driver.
c)Use the Kubernetes Secret for an environment variable: Restart the pod to get the latest secret as an environment variable. Use a tool such as Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods.
Enable auto-rotation on a new AKS cluster:
Enable auto-rotation on an existing AKS cluster:
Specify a custom rotation interval:
Disable auto-rotation:
To disable autorotation, you first need to disable the add-on azure-keyvault-secrets-provider. Then, you can re-enable the add-on without the enable-secret-rotation parameter.
You can see the automatic change in value of the secretName after the autorotation interval.
But the same change won’t be reflected in the environment variable that is using this ‘secretName’ value. To reflect the change in this case you need to restart the pod.
Additional Examples
The following example illustrates the usage of multiple secrets from Azure Key Vault in a single SecretProviderClass YAML file:
secretProviderClass.yaml-
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kv-spc
spec:
provider: azure
secretObjects:
- secretName: bucketsecret
type: Opaque
data:
- key: DB-USER
objectName: DB-USER
- key: DB-PASSWORD
objectName: DB-PASSWORD
- key: DATABASE-HOST
objectName: DATABASE-HOST
- key: DATABASE-PORT
objectName: DATABASE-PORT
- key: DATABASE-NAME
objectName: DATABASE-NAME
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: <Managed_Identity's_Client_ID>
keyvaultName: <key_vault_name>
objects: |
array:
- |
objectName: DB-USER
objectType: secret
objectVersion: ""
- |
objectName: DB-PASSWORD
objectType: secret
objectVersion: ""
- |
objectName: DATABASE-HOST
objectType: secret
objectVersion: ""
- |
objectName: DATABASE-PORT
objectType: secret
objectVersion: ""
- |
objectName: DATABASE-NAME
objectType: secret
objectVersion: ""
tenantId: <tenant_id>
pod.yaml-
kind: Pod
apiVersion: v1
metadata:
name: nginx-csi
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets-store"
readOnly: true
env:
- name: DB_USER
valueFrom:
secretKeyRef:
name: bucketsecret
key: DB-USER
- name: DB-PASSWORD
valueFrom:
secretKeyRef:
name: bucketsecret
key: DB-PASSWORD
- name: DATABASE-HOST
valueFrom:
secretKeyRef:
name: bucketsecret
key: DATABASE-HOST
- name: DATABASE-PORT
valueFrom:
secretKeyRef:
name: bucketsecret
key: DATABASE-PORT
- name: DATABASE-NAME
valueFrom:
secretKeyRef:
name: bucketsecret
key: DATABASE-NAME
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kv-spc"
Limitations
a) Enable Secret autorotation feature has been released in v0.0.15+ and is not available in release v0.0.14 and earlier of Secrets Store CSI Driver.
b) Secrets not rotated when using subPath volume mount.
volumeMounts:
- mountPath: /app/spapi/settings.ini
name: app-config
subPath: settings.ini
...
volumes:
- csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: app-config
name: app-config
For more details refer to https://secrets-store-csi-driver.sigs.k8s.io/known-limitations.html
Conclusion
This guide provided a comprehensive walkthrough for implementing the Azure Key Vault Provider for Secrets Store CSI Driver in AKS clusters. It covered integration, benefits, and best practices for secure secret management, empowering users to enhance their Kubernetes application security and efficiently leverage Azure Key Vault.
