EKS Best Practice - Store Secrets for Sensitive Data(Kubernetes Secrets Management)

A Kubernetes Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret means that you don't need to include confidential data in your application code.

Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. On Amazon EKS, the Amazon EBS volumes for etcd nodes are encrypted with EBS encryption.

There are many options to create secrets in Amazon EKS, you can use kubectl, eksctl, config files, tools like Kustomize or infrastructure as code tools like Terraform. It has become a common practice for a DevOps Team to manage the YAML manifests for various Kubernetes resources and version control them using a Git repository. In this article, we will focus on different ways of passing secrets into Amazon EKS with version control.

How to securely pass Secrets into Amazon EKS

There are three major options to securely pass secrets into Amazon EKS:

• Use Environment Variables

• Use Amazon KMS or Sealed Secrets for envelope encryption of Kubernetes secrets

• Use an external secrets provider (e.g., Vault, AWS Secrets Manager)

OPTION 1: Use Environment Variables

This option lets you save your secret data as environment variables and read from your local environment when you need to create K8s secrets. You can use secrets management tools like pass to avoid leaving data in your BASH history.

This option should only be used when you’re doing demo or test projects, and should not be used in production.

Here’s an example of using pass and terraform to create Kubernetes’s secret.

First, you’ll need to store your secrets by using the pass insert command:

$ pass insert db_username
Enter password for db_username: admin
$ pass insert db_password
Enter password for db_password: password  

Second, export secrets as environment variables:

# Read secrets from pass and set as environment variables
export TF_VAR_username=$(pass db_username)
export TF_VAR_password=$(pass db_password)

Third, in your terraform code use variables to create Kubernetes’s secret:

variable "username" {
  type        = string
  default     = ""
}
variable "password" {
  type        = string
  default     = ""
}

resource "kubernetes_secret_v1" "mysql-root-secret" {
  metadata {
    name = "mysql-root-pass"
  }
  data = {
    username = var.username
    password = var.password
  }
  type = "kubernetes.io/basic-auth"
}  

PROS:

• Keep plain text secrets out of your code and version control system.

• An easy solution to get started with and easy to test.

• Integrates with most other secrets management solutions: e.g., if your company already has a way to manage secrets, you can typically find a way to make it work with environment variables.

CONS:

• Everyone using your code has to know to take extra steps to either manually set these environment variables or run a wrapper script.

• Hard to keep track of the secrets or rotate the secrets.

• Not very secure

OPTION 2: Use Amazon KMS or Sealed Secrets for envelope encryption

This option relies on encrypting the secrets, storing the cipher text in a file, and checking that file into version control. AWS KMS and Bitnami's Sealed Secrets are popular tools to encrypt secrets that can be stored in version control.

AWS KMS allows you to encrypt your secrets with a unique data encryption key (DEK). The DEK is then encrypted using a key encryption key (KEK) from AWS KMS which can be automatically rotated on a recurring schedule. With the AWS KMS plugin for Kubernetes, all Kubernetes secrets are stored in etcd in ciphertext instead of plain text and can only be decrypted by the Kubernetes API server.

Sealed Secrets provides a mechanism to encrypt a Secret object so that it is safe to store - even in a public repository. A SealedSecret can be decrypted only by the controller running in the Kubernetes cluster and nobody else is able to obtain the original Secret from a SealedSecret.

The use case for this option is when your secrets will be effective for a long time and don’t need to be rotated often, such as sensitive lab data. Our next blog post will talk about detailed configuration and sample code.

Here’s an example of using AWS KMS to encrypt you secrets:

aws kms encrypt \
--key-id <YOUR KMS KEY> \
--region <AWS REGION> \
--plaintext fileb://db-creds.yml \
--output text \
--query CiphertextBlob \
> db-creds.yml.encrypted

Then you can decrypt the secrets in terraform and use it to create Kubernetes’s secret:

data "aws_kms_secrets" "creds" {
  secret {
    name    = "db"
    payload = file("${path.module}/db-creds.yml.encrypted")
  }
}
locals {
  db_creds = jsondecode(
    data.aws_secretsmanager_secret_version.creds.secret_string
  )
}
resource "kubernetes_secret_v1" "mysql-root-secret" {
  metadata {
    name = "mysql-root-pass"
  }
  data = {
    username = local.db_creds.username
    password = local.db_creds.password
  }
  type = "kubernetes.io/basic-auth"
}

PROS:

• Keep plain text secrets out of your code and version control system.

• Your secrets are stored in an encrypted format in version control, so they are versioned, packaged, and tested with the rest of your code. This helps reduce configuration errors, such as adding a new secret in one environment (e.g., staging) but forgetting to add it in another environment (e.g., production).

• Works with a variety of different encryption options: AWS KMS, GCP KMS, Sealed Secrets, etc.

• Everything is defined in the code. There are no extra manual steps or wrapper scripts required

CONS:

• Encrypting the data requires extra work.

• The secrets are now encrypted, but as they are still stored in version control, rotating and revoking secrets is hard. If anyone ever compromises the encryption key, they can go back and decrypt all the secrets that were ever encrypted with it.

• The ability to audit who accessed secrets is minimal. If you’re using a cloud key management system (e.g., AWS KMS), it will likely maintain an audit log of who used a key to decrypt something, but you won’t be able to tell what was actually decrypted.

• Not as test friendly

OPTION 3: Use an external secrets provider (e.g., Vault, AWS Secrets Manager)

This option is to store your secrets in a dedicated secret store: that is, a database that is designed specifically for securely storing sensitive data and tightly controlling access to it.

There are several viable secret stores to using Kubernetes secrets, including AWS Secrets Manager and Hashicorp's Vault. These services offer features such as fine-grained access controls, strong encryption, and automatic rotation of secrets that are not available with Kubernetes Secrets.

The use case for this option would when you need to rotate the secrets often or secrets have short session time, such as your AWS credentials or database credentials.

Here’s an example of using AWS Secrets Manager and Terraform to create Kubernetes secret:

First, log in to the AWS Secrets Manager UI, click “store a new secret,” and enter the secrets you wish to store and save it:

Then you can use terraform to pull the secret down and use it to create Kubernetes’s secret:

data "aws_secretsmanager_secret_version" "creds" {
  # Fill in the name you gave to your secret
  secret_id = "db-creds"
}
locals {
  db_creds = jsondecode(
    data.aws_secretsmanager_secret_version.creds.secret_string
  )
}
resource "kubernetes_secret_v1" "mysql-root-secret" {
  metadata {
    name = "mysql-root-pass"
  }
  data = {
    username = local.db_creds.username
    password = local.db_creds.password
  }
  type = "kubernetes.io/basic-auth"
}

PROS: