EKS Best Practice - Kubernetes Secrets Management Part II

Read Time 2 mins | Written by: Daoqi | Celeste Shao

In the previous EKS Secrets Management post, we talked about three options for securing your secrets in EKS. In this post, we will walk through the details of KMS for envelope encryption.

KMS Envelope Encryption

Use AWS KMS for envelope encryption of Kubernetes secrets and audit the use of Kubernetes Secrets. For secrets that are like login credentials, use an external secrets provider to rotate your secrets periodically. If you have secrets that cannot be shared between applications in a namespace, create a separate namespace for those applications.


In ASCENDING, we implement our EKS cluster using the infrastructure as code(IaC) tool - Terraform. Creating an EKS secret in Terraform is easy, we can just use the kubernetes_secret_v1 resource to create an empty secret in EKS. The more important part is how to pass the sensitive data into the EKS secret.

There’re some requirements and factors we take into consideration when designing the secret store approach:

  • We’re okay with storing the encrypted secrets in the GitHub repo for version control purposes.

  • We want to know who uses the secrets.

  • We want to minimize the cost.

  • We want to make the secret retrieval process easier.

The secret store approaches should be designed accordingly based on different situations. Considering the above factors, we decide to use Amazon KMS to encrypt the data and push the encrypted data to version control. We’ll also turn on EKS audit logs and create a CloudWatch metrics filter to log the secret usage.

STEP 1 - Create a KMS service in AWS

    resource "aws_kms_key" "eks" {
      description                           = "EKS Secret Encryption Key"
      deletion_window_in_days = 7
      policy                                     = coalesce(data.aws_iam_policy_document.this[0].json)
    # Add an alias to the key
    resource "aws_kms_alias" "eks" {
      name                = "alias/${var.alias}"
      target_key_id = aws_kms_key.eks.key_id

STEP 2 - Use the KMS key to encrypt the data

First, we save our data in a file called db-creds.yml, then we use the generated KMS to generate the encrypted file db-creds.yml.encrypted. We can check the encrypted file into version control and delete the original file.

  aws kms encrypt \
    --key-id <YOUR KMS KEY> \
    --region <AWS REGION> \
    --plaintext fileb://db-creds.yml \
    --output text \
    --query CiphertextBlob \
    > db-creds.yml.encrypted

STEP 3 - Decrypt the file in Terraform

To use Terraform to create EKS secret and pass value in, we need to decrypt the file

  data "aws_kms_secrets" "creds" {
    secret {
      name    = "db"
      payload = file("${path.module}/db-creds.yml.encrypted")
  locals {
    db_creds = yamldecode(data.aws_kms_secrets.creds.plaintext["db"])

  resource "kubernetes_secret_v1" "mysql-root-secret" {
    metadata {
      name = "mysql-root-pass"
      namespace = "poll"
    data = {
      username = local.db_creds.username
      password = local.db_creds.password
    type = "kubernetes.io/basic-auth"



DevOps Engineer @ASCENDING

Celeste Shao

Data Engineer @ASCENDING