EKS Best Practice - Infrastructure As Code Using Terraform

Infrastructure As Code Overview

Infrastructure as Code (IaC) is a method of managing and provisioning infrastructure through code instead of manual processes. This approach involves creating configuration files that contain all of the necessary infrastructure specifications, making it easy to edit and distribute configurations. By using IaC, you can ensure that you are provisioning the same environment every time, and it helps you to avoid undocumented, ad-hoc configuration changes.

Terraform Overview

One popular tool for implementing IaC is Terraform. It allows you to define both cloud and on-premise resources in human-readable configuration files that can be versioned, reused, and shared. Using Terraform, you can implement a consistent workflow for provisioning and managing all of your infrastructure throughout its lifecycle. Some of the benefits of using Terraform include:

• Unified Workflow: Terraform can be integrated into your existing workflow for deploying infrastructure to AWS, and it can also be used to deploy applications into your Amazon EKS cluster.

• Full Lifecycle Management: Terraform not only creates resources, but it also updates and deletes tracked resources without requiring you to inspect the API to identify those resources. The "state" of the infrastructure is maintained in a file that can be saved in s3, which makes it easy to roll back changes if necessary.

• Graph of Relationships: Terraform understands the dependency relationships between resources, so it can create resources in the correct order and avoid creating resources that depend on other resources that have failed to create.

IaC is currently the most effective way to manage a Kubernetes cluster. Some of the pain points of managing infrastructure manually include:

Why need this

• Infrastructure drift, where the actual infrastructure running is different than what is described in the documentation.

• Difficulty in planning and sharing changes with teammates before actual deployment.

• Having to repeat the same process multiple times for every environment with mixed results.

• No quick way to roll back if an issue is detected.

Using Terraform for IaC can simplify the process of creating and managing Amazon EKS clusters on AWS. By using Terraform templates, you can create new clusters with the same settings easily and track changes over time. Additionally, Terraform provides useful functions for migrating yaml files to its own language, making it easy to transition from other tools.

Here is a table listing 6 main Kubernetes tools and their pros and cons. kubectl and eksctl are great tools to view cluster resources but you can’t track changes.

ASCENDING Approach

To implement this approach, we will create Terraform templates to create and provision Amazon EKS clusters. The Terraform state file will be stored in an Amazon S3 bucket, and Kubernetes deployments or other resources will also be provisioned using Terraform templates.

An example of Terraform code to create an Amazon EKS cluster with one self-managed NodeGroup and one EKS-managed NodeGroup is provided.

#################################################################################
# This terraform template creates an EKS cluster with a self-mananged nodegroup 
# and one EKS-managed nodegroup. The cluster admin is set during cluster creation.
#################################################################################

module "eks" {

  source  = "terraform-aws-modules/eks/aws"
  version = "19.4.2"
  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  cluster_endpoint_public_access  = var.cluster_endpoint_public_access
  cluster_addons = {
    coredns = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
  }

  vpc_id                   = var.vpc_id
  subnet_ids               = var.subnet_ids
  control_plane_subnet_ids = var.control_plane_subnet_ids
  # Self Managed Node Group(s)
  self_managed_node_group_defaults = {

    instance_type                          = "t3.small"
    update_launch_template_default_version = true
    # enable discovery of autoscaling groups by cluster-autoscaler
    autoscaling_group_tags = {
      "k8s.io/cluster-autoscaler/enabled" : true,
      "k8s.io/cluster-autoscaler/${var.cluster_name}" : "owned",
    }
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

  self_managed_node_groups = {
    sng-1 = {
      name         = "self-nodegroup-1"
      bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"
      min_size     = 1
      max_size     = 2
      desired_size = 1
    }
  }
  # EKS Managed Node Group(s)
  eks_managed_node_groups = {
    mng-1 = {
      name         = "managed-nodegroup-1"
      min_size     = 1
      max_size     = 2
      desired_size = 1
      instance_types = ["t3.medium"]
    }
  }
  # aws-auth configmap
  manage_aws_auth_configmap = true
  aws_auth_roles = [
    {
      rolearn  = var.cluster_admin_role
      username = "Admin:"
      groups   = ["system:masters"]
    },
  ]
  tags = {
    Environment = "test"
    Terraform   = "true"
  }
}