EKS Best Practice - Infrastructure As Code Using Terraform

Read Time 4 mins | Written by: Daoqi | Celeste Shao | Publish Date: 01-26-2023

Infrastructure As Code Overview

Infrastructure as Code (IaC) is a method of managing and provisioning infrastructure through code instead of manual processes. This approach involves creating configuration files that contain all of the necessary infrastructure specifications, making it easy to edit and distribute configurations. By using IaC, you can ensure that you are provisioning the same environment every time, and it helps you to avoid undocumented, ad-hoc configuration changes.

Terraform Overview

One popular tool for implementing IaC is Terraform. It allows you to define both cloud and on-premise resources in human-readable configuration files that can be versioned, reused, and shared. Using Terraform, you can implement a consistent workflow for provisioning and managing all of your infrastructure throughout its lifecycle. Some of the benefits of using Terraform include:

Unified Workflow: Terraform can be integrated into your existing workflow for deploying infrastructure to AWS, and it can also be used to deploy applications into your Amazon EKS cluster.
Full Lifecycle Management: Terraform not only creates resources, but it also updates and deletes tracked resources without requiring you to inspect the API to identify those resources. The “state” of the infrastructure is maintained in a file that can be saved in s3, which makes it easy to roll back changes if necessary.
Graph of Relationships: Terraform understands the dependency relationships between resources, so it can create resources in the correct order and avoid creating resources that depend on other resources that have failed to create.

IaC is currently the most effective way to manage a Kubernetes cluster. Some of the pain points of managing infrastructure manually include:

Why need this

Infrastructure drift, where the actual infrastructure running is different than what is described in the documentation.
Difficulty in planning and sharing changes with teammates before actual deployment.
Having to repeat the same process multiple times for every environment with mixed results.
No quick way to roll back if an issue is detected.

Using Terraform for IaC can simplify the process of creating and managing Amazon EKS clusters on AWS. By using Terraform templates, you can create new clusters with the same settings easily and track changes over time. Additionally, Terraform provides useful functions for migrating yaml files to its own language, making it easy to transition from other tools.

Here is a table listing 6 main Kubernetes tools and their pros and cons. kubectl and eksctl are great tools to view cluster resources but you can’t track changes.

EKS IaC comparsion

ASCENDING Approach

To implement this approach, we will create Terraform templates to create and provision Amazon EKS clusters. The Terraform state file will be stored in an Amazon S3 bucket, and Kubernetes deployments or other resources will also be provisioned using Terraform templates.

An example of Terraform code to create an Amazon EKS cluster with one self-managed NodeGroup and one EKS-managed NodeGroup is provided.


#################################################################################
# This terraform template creates an EKS cluster with a self-mananged nodegroup 
# and one EKS-managed nodegroup. The cluster admin is set during cluster creation.
#################################################################################

module "eks" {

  source  = "terraform-aws-modules/eks/aws"
  version = "19.4.2"
  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  cluster_endpoint_public_access  = var.cluster_endpoint_public_access
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
  }

  vpc_id                   = var.vpc_id
  subnet_ids               = var.subnet_ids
  control_plane_subnet_ids = var.control_plane_subnet_ids
  # Self Managed Node Group(s)
  self_managed_node_group_defaults = {

    instance_type                          = "t3.small"
    update_launch_template_default_version = true
    # enable discovery of autoscaling groups by cluster-autoscaler
    autoscaling_group_tags = {
      "k8s.io/cluster-autoscaler/enabled" : true,
      "k8s.io/cluster-autoscaler/${var.cluster_name}" : "owned",
    }
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

  self_managed_node_groups = {
    sng-1 = {
      name         = "self-nodegroup-1"
      bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
      min_size     = 1
      max_size     = 2
      desired_size = 1
    }
  }
  # EKS Managed Node Group(s)
  eks_managed_node_groups = {
    mng-1 = {
      name         = "managed-nodegroup-1"
      min_size     = 1
      max_size     = 2
      desired_size = 1
      instance_types = ["t3.medium"]
    }
  }
  # aws-auth configmap
  manage_aws_auth_configmap = true
  aws_auth_roles = [
    {
      rolearn  = var.cluster_admin_role
      username = "Admin:{{SessionName}}"
      groups   = ["system:masters"]
    },
  ]
  tags = {
    Environment = "test"
    Terraform   = "true"
  }
}

Where Terraform State Becomes the Real Problem

The overview above mentions storing state in S3 almost in passing, but state handling is where most Terraform-managed clusters actually get into trouble, so it deserves more than a footnote.

The state file is the authority on what Terraform believes exists. Lose it and Terraform no longer recognises your cluster — a subsequent apply will try to create everything a second time. Corrupt it, or let two engineers write it concurrently, and the file can end up describing infrastructure that never existed.

The remote backend addresses both risks. Point the backend at an S3 bucket with versioning enabled, so any bad write can be rolled back to a known-good version, and enable state locking so a second apply waits rather than racing. Because the state file contains resource attributes in plaintext — including values that originated as secrets — enable bucket encryption and treat read access to that bucket as equivalent to read access to the cluster.

Separate state per environment. A single state file spanning dev, staging, and production means every production apply carries a plan that could touch dev, and one mistaken -target reaches further than intended. Separate backends, or at minimum separate keys, keep the blast radius bounded.

Modules, Versions, and Reproducibility

The promise of IaC is that the same configuration produces the same environment. Two things quietly break that promise.

The first is unpinned versions. Both the provider and any external module should be constrained to an explicit version range. Leave them open and a colleague running terraform init next month resolves a newer provider, generates a different plan from identical code, and the reproducibility guarantee is gone. Commit the lock file so the resolved versions travel with the repository.

The second is out-of-band change. Someone fixes an incident through the console, the cluster now differs from the code, and the next plan proposes to revert the fix. This is infrastructure drift, and IaC does not prevent it — it only makes it visible. Running terraform plan on a schedule and alerting on a non-empty diff turns drift from a surprise during the next deploy into a routine notification.

Terraform and Kubernetes Manifests

Terraform provisions the cluster well. Whether it should also manage the workloads inside it is a separate question, and the honest answer is that it is a trade-off rather than a best practice.

Managing manifests through the Kubernetes provider keeps everything in one plan and one dependency graph, which is genuinely convenient for the bootstrap layer — the CNI configuration, the ingress controller, the cluster-wide RBAC that has to exist before anything else can deploy.

It fits application workloads much less well. Application deployments happen far more often than infrastructure changes, and routing them through a Terraform apply couples release cadence to infrastructure cadence. It also puts ephemeral, controller-managed fields into Terraform state, which produces perpetual diffs that engineers learn to ignore — and ignored diffs are how real changes get missed.

A common split is to let Terraform own the cluster and its bootstrap add-ons, and let a Kubernetes-native delivery tool own the applications. The boundary sits where the change rate changes.