EKS Best Practice - Kubernetes Secrets Management Part II

In the previous EKS Secrets Management post, we talked about three options for securing your secrets in EKS. In this post, we will walk through the details of KMS for envelope encryption.

KMS Envelope Encryption

Use AWS KMS for envelope encryption of Kubernetes secrets and audit the use of Kubernetes Secrets. For secrets that are like login credentials, use an external secrets provider to rotate your secrets periodically. If you have secrets that cannot be shared between applications in a namespace, create a separate namespace for those applications.

ASCENDING Approach

In ASCENDING, we implement our EKS cluster using the infrastructure as code(IaC) tool - Terraform. Creating an EKS secret in Terraform is easy, we can just use the kubernetes_secret_v1 resource to create an empty secret in EKS. The more important part is how to pass the sensitive data into the EKS secret.

There’re some requirements and factors we take into consideration when designing the secret store approach:

• We’re okay with storing the encrypted secrets in the GitHub repo for version control purposes.

• We want to know who uses the secrets.

• We want to minimize the cost.

• We want to make the secret retrieval process easier.

The secret store approaches should be designed accordingly based on different situations. Considering the above factors, we decide to use Amazon KMS to encrypt the data and push the encrypted data to version control. We’ll also turn on EKS audit logs and create a CloudWatch metrics filter to log the secret usage.

STEP 1 - Create a KMS service in AWS

    resource "aws_kms_key" "eks" {
      description                           = "EKS Secret Encryption Key"
      deletion_window_in_days = 7
      policy                                     = coalesce(data.aws_iam_policy_document.this[0].json)
    } 
    # Add an alias to the key
    resource "aws_kms_alias" "eks" {
      name                = "alias/${var.alias}"
      target_key_id = aws_kms_key.eks.key_id
    }    

STEP 2 - Use the KMS key to encrypt the data

First, we save our data in a file called db-creds.yml, then we use the generated KMS to generate the encrypted file db-creds.yml.encrypted. We can check the encrypted file into version control and delete the original file.

  aws kms encrypt \
    --key-id <YOUR KMS KEY> \
    --region <AWS REGION> \
    --plaintext fileb://db-creds.yml \
    --output text \
    --query CiphertextBlob \
    > db-creds.yml.encrypted
  

STEP 3 - Decrypt the file in Terraform

To use Terraform to create EKS secret and pass value in, we need to decrypt the file

  data "aws_kms_secrets" "creds" {
    secret {
      name    = "db"
      payload = file("${path.module}/db-creds.yml.encrypted")
    }
  }
  locals {
    db_creds = yamldecode(data.aws_kms_secrets.creds.plaintext["db"])
  }

  resource "kubernetes_secret_v1" "mysql-root-secret" {
    metadata {
      name = "mysql-root-pass"
      namespace = "poll"
    }
    data = {
      username = local.db_creds.username
      password = local.db_creds.password
    }
    type = "kubernetes.io/basic-auth"
  }