Intoduction

AWS CloudFormation is a powerful tool to allow you to get environments in the cloud up and running as quickly as possible and with minimum user error. It allows you to build templates that can create almost anything in your cloud environment automatically. Well, almost…

There are some places where CloudFormation is lacking in customization, especially when it comes to processing multiple items in a list. To get around this, CloudFormation allows the creation of custom resources which you can use to extend your template with additional functionality.

AWS Lambda Function based Custom Resources

For those of you that are not familiar with AWS Lambda, here is a brief overview:

AWS Lambda is a platform that allows you to run code in a “serverless” manner, meaning without needing to deploy a server or any kind of runtime. A serverless function is usually stateless and short lived. In our particular use case, it is similar to running a script.

In CloudFormation, a special kind of AWS Lambda function can be created and called during the stack create / update / delete process to perform any kind of action. It can perform all kinds of tasks such as running some sort of calculation, looking up a value from a file in an S3 bucket, or calling AWS API functions to provision resources.

Creating a Custom Resource

To show you how to create a custom resource and add it to your template, we will use a relatively simple example:

In our use case, we want to get a list of VPC IDs as a parameter and then create VPC Flow Logs for all of the VPCs. CloudFormation supports creating VPC Flow Logs, but each flow log has to be defined separately, and as we do not know how many we need to create ahead of time (since we are getting it as a parameter), there is no way to create a dynamic number of resources in our template. To solve this, we will create an AWS Lambda function that will create the Flow Logs for us. Then we will use this function as a Custom Resource to have it created and deleted along with the rest of our stack.

The Lambda Function

The first step, is to create a Lambda function that creates our Flow Logs. We will be defining the function inline in our template so that we end up with a single template file that deploys everything without needing to rely on anything to already be set up in the account. Also, we will be using node.js as the programming language, though Python is supported as well.

First of all, we need to include the AWS SDK in our function so that we can call the AWS APIs to create flow logs. We will also add the cfn-response package which contains helper functions for CloudFormation Custom Resources (Note: The cfn-response package is only available when creating an inline Lambda function. For other cases, you can find the source code here).

const AWS = require(‘aws-sdk’);
var response = require(‘cfn-response’);

In our handler function, we first need to determine which operation we need to perform. This can be either “Create”, “Delete”, or “Update”.

if (event.RequestType == “Delete”) {
       // Delete the Flow Logs
      …
} else if(event.RequestType == “Create”) {
     // Create the Flow Logs
     …
};

To Create the flow logs, there are a few parameters we need to get from the CloudFormation Template. In our case, we need the S3 bucket ARN to which the Flow Logs will be sent to as well as the list of VPC IDs to enable. All the parameters that are sent to a Lambda Custom Resource are accessible from the event object under ResourceProperties. The API for creating Flow Logs supports receiving multiple resource IDs saving us the need to loop over the VPC IDs we sent. There are also some parameters that are sent automatically that identify the CloudFormation stack and resource. We will tag the Flow Logs with these – we will need them later.

let tags = [{ResourceType: “vpc-flow-log”,
        Tags: [ {Key: “vnt:cloudformation:logical-id”, Value: event.LogicalResourceId},
                    {Key: “vnt:cloudformation:stack-id”, Value: event.StackId}] }];
let params = {
        LogDestination: event.ResourceProperties.BucketArn,
        LogDestinationType: “s3”,
        ResourceIds: event.ResourceProperties.ResourceIds,
        ResourceType: ‘VPC’,
        TrafficType: ‘ACCEPT’,
        TagSpecifications: tags

};

ec2.createFlowLogs(params, function(err, data) {
       if(err) {
               responseData = {Error: “Failed to create VPC flow logs”};
               console.log(responseData.Error + “\n” + err);
      } else {
              responseStatus = response.SUCCESS;
              responseData = data;
     }

     response.send(event, context, responseStatus, responseData);
});

We use the response package that we included earlier to send the result back to CloudFormation. Make sure that all the paths in your function return a response! If you do not return a response, the stack will be stuck in the creating / updating / deleting status until it times out (1 hour by default).

When a delete request is received, we need to clean up everything we created. In our case, we need to delete the Flow Logs that we created. Here is where the tags we defined earlier come in, we can use them to identify the Flow Logs we created so that we know what to delete.

let params = {

       Filter: [{Name: “tag:vnt:cloudformation:logical-id”, Values: [event.LogicalResourceId]},

                {Name: “tag:vnt:cloudformation:stack-id”, Values: [event.StackId]}]

};

ec2.describeFlowLogs(params, function(err, data) {

       if(err) {

              responseData = {Error: “Failed to get existing flow logs information”};

              console.log(responseData.Error + “\n” + err);

              response.send(event, context, responseStatus, responseData);

       } else {

              let flowLogIds = [];

              if(data.FlowLogs && data.FlowLogs.length > 0) {

                     for(let i = 0; i < data.FlowLogs.length; ++i) {

                             flowLogIds.push(data.FlowLogs[i].FlowLogId);

                      }

                     

                      ec2.deleteFlowLogs({FlowLogIds: flowLogIds}, function(err, data) {

                             if(err) {

                                    responseData = {Error: “Failed to delete flow logs”};

                                    console.log(responseData.Error + “\n” + err);

                             } else {

                                    responseStatus = response.SUCCESS;

                                    responseData = data;

                             }

                            

                             response.send(event, context, responseStatus, responseData);

                      });

              } else {

                      responseData = {Error: “Couldn’t find existing flow logs”};

                      response.send(event, context, responseStatus, responseData);

              }

       }

});

That’s it for our Lambda function, now back to the CloudFormation template to put it all together.

Adding the Custom Resource to a Template

To add the Custom Resource we created to a CloudFormation template, we need to add it as a resource with a type starting with “Custom::”. We will be using the YAML format for our template as it is easier to work with embedded Lambda functions in YAML templates.

CreateFlowLogsFunction:

    Type: ‘AWS::Lambda::Function’

    Properties:

      Code:

        ZipFile: |

            …

      Handler: index.handler

      Runtime: nodejs12.x

      Timeout: ’30’

      Role: !GetAtt

        – CreateFlowLogsRole

        – Arn

CreateFlowLogs:

    Type: ‘Custom::CreateFlowLogs’

    DependsOn: FlowLogsBucket

    Properties:

      ServiceToken: !GetAtt

        – CreateFlowLogsFunction

        – Arn

      Region: !Ref ‘AWS::Region’

      ResourceIds: !Ref MappingVpcId

      BucketArn: !GetAtt

        – FlowLogsBucket

        – Arn

The ServiceToken is the only required property for a Custom Resource, we will use the ARN from our function as the ServiceToken. All other properties that we define are sent in the event to the Lambda function under ResourceProperties. Note that we are also referencing additional CloudFormation resources in the snippet above. This includes the Role that defines the permissions for the function, the MappingVpcId parameter which is the list of VPC IDs we got from the user, and the FlowLogsBucket which is a bucket we are creating to save the logs in. All these will be included in the full template file that you can find below.

That’s it, our template file is complete.  It will now automatically create and delete the flow logs for all the VPCs that were requested automatically when the stack is created or deleted.

Conclusion

Using CloudFormation Custom Resources can make this powerful utility even more useful. You can use Lambda functions to extend CloudFormation with almost anything you can imagine whether that’s creating dynamic resources based on the environment or looking up AMI IDs from a database table. Building them can be a bit tricky and the existing documentation is not the best, but I hope that this has helped you get started.

Just a note that there is also another option to extend the functionality of CloudFormation templates using Macros. Macros can basically duplicate parts of your template to create multiple resources. In my use case, a Macro was not a good fit since these require additional setup in your account and I wanted a single template file that could be deployed without needing anything specific existing in the account.

The full template file is available for download below so you can try it out yourself.

Download Customer Resource Templates
White Paper Dependency Discovery Banner LinkedIn