How we created a serverless version of cfn-flip

20 May, 2018
Xebia Background Header Wave

The original version of cfn-flip is a command line tool that converts CloudFormation templates from JSON to YAML and vice versa. This tool has existed for some time and does the job magnificently. Most people find YAML more readable. I belong to the minority that finds JSON easier to read. It was ritten by Steve Engledow and you can find the original cfn-flip Github repository at
In my experience new idea’s aren’t born trying to think up what doesn’t yet exist, but rather by observing something lacking and building something around that. This was definitely the case for serverless cfn-flip.
The idea to create a serverless version of the tool that could do more than just convert the two formats was born when I noticed was not registered.


Before I wrote a single line of code I thought of what I wanted to achieve and narrowed it down to the following:

  1. As little as possible interaction with the AWS console!
  2. Created fully in CloudFormation.
  3. As I am a Ruby fanatic, it must output to CfnDsl as well.
  4. Each conversion function in its own Lambda function.
  5. With pretty colours (syntax highlighting)
  6. Backed with API Gateway, of course.
  7. Giving back to the community by open sourcing the entire project.


CfnFlip consists of an S3 bucket, four Lambda functions, an API Gateway, a CloudFront distribution and Route53. I’ve crafted the architecture to provide you with a visual overview:
How we created a serverless version of cfn-flip

CloudFormation Dependency Stack

I really like how the serverless framework creates an S3 bucket on the fly. It can store your artifacts there without the need to perform any manual bucket creation in the console or with the CLI. Inspired by that simple yet eloquent awesomeness I decided that this project needed that kind of magic as well. This came in the form of a dependency stack where I create some immutable resources that subsequent stacks import. The dependency stack took this form:

AWSTemplateFormatVersion: '2010-09-09'
    Type: String
    content: Domain name for the hosted zone
    Type: String
    content: Name of your project, e.g. cfnflip
    Default: CfnFlip

I’ve used parameters here so you can easily re-use this snippet for another project. It takes the domain name and project name as parameters. Then onto the resources:

    Type: AWS::S3::Bucket

We let CloudFormation create a bucket with a dynamic name. The name will be outputted and exported. Then we let CloudFormation create our HostedZone.

    Type: AWS::Route53::HostedZone
        Comment: !Join
          - ''
          - - 'HostedZone for '
            - !Ref 'DomainName'
      Name: !Ref 'DomainName'

Next up is a Lambda function. I wanted to automatically configure the Route53 domain name ( with the DNS servers of the dynamically created Route53 HostedZone, so I would have to have the least amount of interaction with the AWS console. A custom resource invokes this Lambda function and passes on the DNS servers and then proceeds to configure the domain name using the NodeJS AWS SDK:

    Type: AWS::Lambda::Function
      Handler: index.handler
      Role: !GetAtt 'LambdaFunctionRole.Arn'
        ZipFile: !Join
          - "\n"
          - - var AWS = require('aws-sdk');
            - 'AWS.config.update({region: ''us-east-1''});'
            - var response = require('cfn-response');
            - exports.handler = function(event, context) {
            - '    var nameservers = event.ResourceProperties.Nameservers;'
            - '    var domainname  = event.ResourceProperties.HostedZone;'
            - ''
            - '    console.log(nameservers);'
            - '    console.log(domainname);'
            - '    if (event.RequestType == ''Delete'') {'
            - '        var responseData = { Value: ''Go ahead.'' };'
            - '        response.send(event, context, response.SUCCESS, responseData);'
            - '    }'
            - '    else'
            - '    {'
            - ''
            - '        var route53domains = new AWS.Route53Domains();'
            - '        var ServerList = [];'
            - '        nameservers.forEach(function(server){'
            - '          ServerList.push({'
            - '            Name: server'
            - '          })'
            - '        });'
            - '        route53domains.updateDomainNameservers('
            - '            {'
            - '                DomainName: domainname,'
            - '                Nameservers: ServerList'
            - '            }, function(err, data) {'
            - '                if (err) console.log(err, err.stack);'
            - '                else     console.log(data);'
            - '                var responseData = { Value: ''Hurray!'' };'
            - '                response.send(event, context, response.SUCCESS, responseData);'
            - '            }'
            - '        )'
            - '    }'
            - '};'
      Runtime: nodejs6.10
      MemorySize: '128'
      Timeout: '25'
    DependsOn: LambdaFunctionRole

Of course our Lambda function needs permissions. Our execution role permits the Lambda function to update the DNS servers and to log to CloudWatch:

    Type: AWS::IAM::Role
      Path: /
        Version: '2012-10-17'
          - Effect: Allow
              - sts:AssumeRole
        - PolicyDocument:
            Version: '2012-10-17'
              - Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Effect: Allow
                Resource: arn:aws:logs:*:*:*
              - Action:
                  - route53domains:UpdateDomainNameservers
                Effect: Allow
                Resource: '*'
          PolicyName: Route53LambdaUpdateRole

Finally, we define a custom resource and pass on the DNS servers to our Lambda function:

    Type: Custom::UpdateRoute53
      HostedZone: !Ref 'DomainName'
      Nameservers: !GetAtt 'HostedZone.NameServers'
      ServiceToken: !GetAtt 'LambdaFunction.Arn'
      - LambdaFunction
      - HostedZone

We now have a HostedZone and an S3 bucket that should never change. We output and export these immutable values for our application stack to use:

    Value: !Ref 'HostedZone'
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - HostedZoneId
    Value: !Ref 'DomainName'
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - HostedZoneName
    Value: !Ref 'CloudformationBucket'
      Name: !Join
        - ''
        - - !Ref 'ProjectName'
          - CloudformationBucket

You can import an exported value in another stack by referring to it like this: !ImportValue ‘MyExportedValueName’

A word of caution: If you use an exported value in another stack you can’t modify the value of what you’ve exported until no stack refers to it anymore. In other words, use exports only for data that never ever changes.

The Lambda functions

cfn-flip is a Python script that ordinarily takes a flat file as argument and transforms it to either YAML or JSON, whereas cfn2dsl is a Ruby script that does something similar but instead outputs to Ruby code that the cfndsl gem uses and outputs a CloudFormation template when you feed it to a Ruby interpreter. That meant I had to create two custom Lambda deployment packages: One for cfn-flip and one for cfn2dsl.
Creating a deployment package for Python is a straightforward process. You can simply clone the cfn-flip repository and install the dependencies to your deployment package directory as follows:

pip install -r requirements.txt -t ~/your/deployment_package
On MacOS X, however, this doesn’t work out of the box. You will need to create (or add to) setup.cfg in the cloned directory with the following content:


By default you invoke the cfn-flip tool with some arguments to specify which file you want to convert. In this case, I wanted to have cfn-flip to use the content provided to the Lambda function. So essentially: whatever code the user submitted on the website. Looking at the source code there was just a single function in cfn-flip I needed, called flip.
My Lambda function then looked like this:

import json
import sys
import cStringIO
import urlparse
from cfn_flip import flip

def lambda_handler(event, context):
    data = urlparse.parse_qsl(event["body"])
    for k, v in data:
        if k == 'code':
            plaintext = v
            plaintext = ""

    stdout_ = sys.stdout
    stream = cStringIO.StringIO()
    sys.stdout = stream
    hndlr = sys.stdout
    sys.stdout = stdout_
    data = stream.getvalue()

    return {
        'statusCode': 200,
        'body': json.loads(json.dumps(data)),
        'headers': {
                'Content-Type': 'plain/text',
                'Access-Control-Allow-Origin': '*',
                'Cache-control': 'private, max-age=0, no-cache'

The event body expects the template’s body being posted with code as key. The Lambda function then captures the output of cfn-flip’s flip function, and finally returns the flipped version.
Porting Ruby’s “cfn-dsl” was a different challenge altogether. I’ve used a portable version of Ruby 2.2.x called Traveling Ruby to prime my deployment package on a Linux environment. The cfn2dsl gem (rightfully) pinned to Ruby 2.3 as a minimum version, so I had to get creative.
I started with installing Ruby with an identical version of traveling ruby using rbenv. Then, I created a Gemfile with the dependencies of cfn2dsl, but without the versions pinned to a minimum version. This allowed me to get all of cfndsl’s gem dependencies that were compatible with Ruby 2.2. I then moved all the dependencies into the traveling ruby’s gem directory and created a cfn2dsl_lib directory and stuck the cfn2dsl source code there. With that out-of-the-way I copied over the deployment package to my trusty Macbook and started to work on the Lambda function.
Now, AWS does not support Ruby natively, so the deployment package consists of a NodeJS function to invoke the Ruby script, and the Ruby script to transform the template into CfnDsl code. The ruby code looked like this:


require "uri"
require_relative 'cfn2dsl_lib/cfn2dsl'

payload = URI.decode(JSON.parse(ARGV[0])["body"].tr('+', ' ')).gsub(/^code\=/,'')
template = JSON.parse(payload)
cfndsl   =
dsl =

dsl = "require 'cfndsl'\n\n" \
      "# Generated by\n" \
      "# The 'cfndsl' gem is required. Type `gem install cfndsl` to install it.\n\n" \
      "#{dsl.gsub(/^CloudFormation\ do/, 'template = CloudFormation do')}\n" \
      "puts template.to_json"

puts dsl

There are more eloquent ways to parse POST data, but I didn’t include the required gems in the deployment package to avoid as much bloat as possible. Ruby can be notoriously slow. I broke down the cfn2dsl functionality to just rendering a template from a given string and outputting the result.
The NodeJS wrapper function looks like this:

const exec = require('child_process').exec;

exports.handler = function(event, context, callback) {
    const child = exec('./lambdaRuby.rb ' + '\'' + JSON.stringify(event) + '\'', function(error, stdout, stderr) {
        const response = {
          statusCode: 200,
          headers: {
            'Access-Control-Allow-Origin': '*',
            'Content-Type': 'text/html'
          body: stdout
        callback(null, response);
    child.stdout.on('data', console.log);
    child.stderr.on('data', console.error);

The NodeJS function spawns the Ruby script as a child process, passing a JSON-encoded string as argument to the Ruby script, with a function that waits for the child process to terminate and fetch stdout. The output is then modified so it’s ready to run out of the box and returned.
Last but not least, I needed to create the front-end which consists of a static HTML page and a Lambda function that returns the content:

'use strict';

var fs = require('fs');

module.exports.frontPage = (event, context, callback) => {
  var html = fs.readFileSync("index.html", "utf8");
  const response = {
    statusCode: 200,
    headers: {
      'Access-Control-Allow-Origin': '*',
      'Content-Type': 'text/html',
    body: html

  callback(null, response);

The Lambda function reads the contents of index.html and returns it.

Anatomy of API Gateway

With the Lambda functions in place, it is time to build an API gateway that terminates different end-points to their corresponding Lambda functions. We have a Lambda for the main page, and two Lambda’s to convert the submitted templates to a different format. I will highlight some parts of the CloudFormation template that are key. If you’re interested in the full template you can find it on GitHub.
First we create our API Gateway:

    Type: AWS::ApiGateway::RestApi
      Name: cfnflip

We create our /cfn2dsl/ endpoint as follows:

    Type: AWS::ApiGateway::Resource
      ParentId: !GetAtt 'ApiGatewayRestApi.RootResourceId'
      PathPart: cfn2dsl
      RestApiId: !Ref 'ApiGatewayRestApi'

For each Lambda I’ve created a log group to log to a location that’s predictable and consistent. In this case the log group name is identical to the name of the Lambda function. In case of the Ruby Lambda function the log group resource looks like this:

    Type: AWS::Logs::LogGroup
      RetentionInDays: 7
      LogGroupName: /aws/lambda/cfn2dsl

The LogGroupName determines where in CloudWatch you’ll be able to find back the log files, and we don’t want to keep them forever so we throw away the logs after 7 days. Remember the LogGroupName. We’ll get back to it when creating the IAM AssumeRole policy.
To deploy the Lambda function we need to create an AWS::Lambda::Function resource:

    Type: AWS::Lambda::Function
        S3Bucket: !ImportValue 'CfnFlipCloudformationBucket'
      Handler: ruby.handler
      FunctionName: cfn2dsl
      MemorySize: 256
      Role: !GetAtt 'IamRoleLambdaExecution.Arn'
      Runtime: nodejs6.10
      Timeout: 10
      - IamRoleLambdaExecution
      - CfnFlipRubyLogGroup

As you can see we import the bucket from the dependency stack and pass it to S3Bucket. In our deployment package we’ve created ruby.js, which we use as the function’s handler. MemorySize is all about balance. The higher the memory size, the more CPU the function has access to, so it’s all about balance between speed and cost. For our Ruby lambda function 256 MB of memory provided the best balance. When sizing Lambda functions you have to keep in mind that smaller sizing might mean more costs per invocation, and that sizing high might mean the increased speed doesn’t justify the additional costs.
Then, finally, we need to create a POST method for our endpoint:

    Type: AWS::ApiGateway::Method
      HttpMethod: POST
      RequestParameters: {}
      ResourceId: !Ref 'ApiGatewayResourceRuby'
      RestApiId: !Ref 'ApiGatewayRestApi'
      AuthorizationType: NONE
          application/x-www-form-urlencoded: "#set($allParams = $input.params())\n\
            {\n  \"params\" : {\n    #foreach($type in $allParams.keySet())\n    #set($params\
            \ = $allParams.get($type))\n    \"$type\" : {\n      #foreach($paramName\
            \ in $params.keySet())\n      \"$paramName\" : \"$util.escapeJavaScript($params.get($paramName))\"\
            \n      #if($foreach.hasNext),#end\n      #end\n    }\n    #if($foreach.hasNext),#end\n\
            \    #end\n  }\n}\n"
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Join
          - ''
          - - 'arn:aws:apigateway:'
            - !Ref 'AWS::Region'
            - :lambda:path/2015-03-31/functions/
            - !GetAtt 'CfnFlipRubyLambdaFunction.Arn'
            - /invocations
      MethodResponses: []

The RequestTemplate in the above snippet transforms the request into an event body that we can use in the function, and then we pass it on to the Lambda function.
So did you remember to LogGroupName ? It will come into play now. As you can see in the Lambda function snippet, there is a reference to the IamRoleLambdaExecution role. This role provides the Lambda function with relevant permissions to perform actions on your behalf. You want to limit these privileges to the least amount required for the function to work. Our Lambda’s fortunately don’t require a lot of permissions. It doesn’t interact with any AWS service nor does it require any VPC to talk to. So what does it need? It creates CloudWatch log streams and write to them! So let’s have a look:

    Type: AWS::IAM::Role
          - Action:
              - sts:AssumeRole
            Effect: Allow
        Version: '2012-10-17'
        - PolicyDocument:
              - Action:
                  - logs:CreateLogStream
                Effect: Allow
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfnflip:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/pyflip:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfn2dsl:*'
              - Action:
                  - logs:PutLogEvents
                Effect: Allow
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfnflip:*:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/pyflip:*:*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cfn2dsl:*:*'
            Version: '2012-10-17'
          PolicyName: !Join
            - '-'
            - - cfnflip
              - lambda
      Path: /
      RoleName: !Join
        - '-'
        - - cfnflip
          - eu-west-1
          - lambdaRole

We allow the Lambda function to assume this role with all the permissions listed in the PolicyDocument. We grant the CreateLogStream and the PutLogEvents privilege to the LogGroups we’ve created before. This is all our functions require.


It’s 2:27 AM and I’m typing this very epilogue. It took me longer to write this blog post than it took me to create the MVP for It was great fun to make and I’m already brewing some new ideas. My esteemed colleague Mark is toying with the thought of whipping up some Terraform transformer. If you have any cool ideas do let us know, or better yet: Fork the project and submit a PR for your improvements.

Dennis Vink
Crafting digital leaders through innovative AI & cloud solutions. Empowering businesses with cutting-edge strategies for growth and transformation.

Get in touch with us to learn more about the subject and related solutions

Explore related posts