We have posted a few ways to cut down your AWS bill earlier. Most of examples were given through cloudformation template, because it is effective and straight forward way to manage environment in AWS. Lots of our enterprise clients start to rise awareness to cloudformation template, as they realize it is one of the critical way to save cost in AWS. Through automating customer’s application environment with cloudformation, we have stumbled upon some limitation, e.g. loop over all available instances type to create launch configuration for each instance type, parameterize template for different application environment to reduce duplication in cloudformation template. It took us some unnecessary effort to maintain those duplication code. Our team has struggled a few time for a programmatic cloudformation template solution or tool.

We found some tool like troposphere ,but  it’s a little heavy for us. After some internet investigation , we have found a fairly simple programmatic cloudformation template solution. We hope it can reduce complexity of automating your application environment through cloudformation, also reduce management effort on cloudformation template. Both can save tremendous $$$ cost on cloud operation.

The Problem:

As we started automating client’s DEV,QA,PROD environment in cloudformation, one of the problem is too much copy&paste and writing duplication code in cloudformation template. We realized that those code would potentially cost lots of manual effort to maintain down the road. And sometime if engineer change in one place, he/she may have to change multiple places. As cost optimization is our primary business objective to clients, we decide to improve the situation.

The other problem we have stumbled upon is trying to loop over all available instance types of creating spot fleet configuration. The more available instance type you can subscribe in configuration, the less cost it will be at spot market bidding (see this post). We certainly don’t want to manually duplicate 20+ launch configuration in cloudformation.

The Solution:

Our requirement is simple, the tool needs:

1. lightweight, easy to use
2. We can programmatically code cloudformation template.
3. We prefer native cloudformation language either JSON or YML instead of converting existing template to a new language syntax.

After doing some research, we found following options:

1. troposphere is fairly easy to use and support for almost all AWS resources, but we need to spend some effort to convert existing YML syntax to python. As we know AWS service has evolved pretty quickly, we have a concern if the tool is able to keep up with AWS release rhythm… +__+
2. terraform is more like full blown solution if you want to manage your infrastructure with multiple cloud provider. It has a nice object oriented inheritance features, so you reduce lots of code while writing your infrastructure as code. We’d recommend to give a try if you just start on automating infrastructure or new project. We have fairly large amount of existing cloudformation template, it can’t fulfill needs this time.

Because we weren’t able to find existing tool and the requirement wasn’t too difficult but rather custom. Our team has come up a solution. In order for programmatic cloudformation template management, we need some template engine for YML or JSON with “fragment management” and “variable interpolation” feature etc. We felt ejs template engine might be perfect fit for the job.

Lightweight

EJS engine itself is fairly lightweight. In order to generate target cloudformation YML, we also need a scripting tool load EJS template and manipulate variables. Bash is the go to choice, thanks for ejs-cli which provides seamless integration to bash. If you’d like to be more sophistic , you can also try grunt or gulp.

Programmatic Cloudformation

As it was mentioned earlier, we’d like to reduce duplication code in cloudformation templates for different environments. EJS template provides the flexibility to do those little tricks. We are going to list some of our particle use case.

Fragment management

In some case, one AWS resource is used in multiple environments. We can use EJS include features. DEV,QA,PROD cloudformation template can include same resources.

Resources:
<% include ./partials/resources.yml %>
  ...

Variable interpolation

We can interpolate variable from options.json through <%= %> in EJS.

  apiTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn: ECSTaskRole
    Properties:
      ContainerDefinitions:
      - Name: api
        Essential: true
        Image: <%= container_path %>/api
        Memory: <%= container.api.memory %>
        Environment:
        - Name: REGION
          Value: !Ref "AWS::Region"
      TaskRoleArn: !Ref 'ECSTaskRole'

Loop over list

There is a case we need to loop over all defined instances type(15+ instances) to create spot fleet instance launch configuration.

    Properties:
      SpotFleetRequestConfigData:
        IamFleetRole: !GetAtt iamFleetRole.Arn
        SpotPrice: !Ref 'ECSSpotPrice'
        TargetCapacity: !Ref 'DesiredCapacity'
        TerminateInstancesWithExpiration: false
        AllocationStrategy: lowestPrice
        LaunchSpecifications:
        <% for(var i in instancesTypes) {%>
        <% include ./partials/instance-launch-specification.yml %>
        <% } %>

Those are plenty ways to manipulate EJS template awaiting for you to explore.

Native Cloudformation Language

There are two official syntax supported in cloudformation template JSON,YML. YML was very concise to us but EJS support both JSON and YML template. Being able to continue writing YML cloudformation template is a huge effort saving for us. As we mentioned earlier, other tools are good but our goal is to reduce clients cost from rewriting cloudformation template while there are some simple solution to achieve our goal out there.

Working example:

Last but not least, this is one simple working sample for reference. If you find this approach useful, please tell us. As always, your feedback are always welcome!