CloudFormation vs. Terraform

CloudFormation vs. Terraform Battle of AWS cloud automation tools

The cloud is an exciting technology. One of the main reasons is that almost every aspect can be automated. Traditional data centres have continued to automate infrastructure tasks and push the boundaries of what is possible, but the cloud brings new opportunities with enormous potential.

The objective

It is November 2018. We have an Amazon Web Services (AWS) cloud platform and about 30 accounts and five applications in production. When we started setting up the platform, one of our key objectives was to automate the provision of the infrastructure as much as possible. The most logical way to achieve this was to use AWS CloudFormation.

With growing experience and constant use of the tool, we have often wondered if there were more efficient ways to provide an infrastructure in AWS. What followed was inevitable: an analysis of the Infrastructure-as-Code (IaC) Tool Terraform. An important aspect of Terraform is that it supports multiple clouds – a major advantage in light of our upcoming multi-cloud initiative.

We compared Terraform 0.11 with CloudFormation. The results are described in detail as follows. (Note: 1) We only evaluated Terraform’s capabilities in comparison with AWS. 2) At the time of writing this post, Terraform 0.12 with HCL 2.0 had already been announced, but this version will not be available in the next few months. Some of the tool’s characteristics may need to be reassessed in the future. AWS has also recently introduced some new features for CloudFormation. In this blog we can only deal with these superficially.)

The battle

Support of AWS services

CloudFormation supports almost all AWS services. CloudFormation often supports completely new services either immediately or shortly after their release. However, it can take some time before it supports new features for existing services.

Terraform also supports almost all AWS services. Since it is developed by a large open source community and the developers often want to use new features themselves, completely new services and new features for existing services are quickly available in Terraform. However, since the Terraform community can only implement entirely new services after they have been officially introduced by AWS, it can happen that CloudFormation supports these new services before Terraform.

User Experience

AWS CloudFormation provides a user interface where users can create, modify and graphically display resource dependencies. It also offers a way to validate the template syntax. The console provides an overview of all activities and lots of information to track them. This is especially helpful in case of an error.

Terraform does not have a user interface. The open source version is controlled exclusively via the command line interface (CLI). This can be challenging for inexperienced users. Terraform SaaS and Enterprise provide a user interface.

Programming language

In CloudFormation, the infrastructure is defined in JSON or YAML files known as CloudFormation templates. They can become quite long and confusing for more complex resources. As far as conditions, branches and loops go, CloudFormation supports explicit “if” and “and” constructs, but it is not possible to use “for-loop” or “if-else” constructs.

Terraform has its own domain-specific language, the Hashicorp Configuration Language (HCL). Terraform templates are easy to read and efficient. On average, you need less code than with CloudFormation to achieve the same result. One reason is the “count” meta-parameter that can be used to create constructs like “if”, “if-else” and “for-loop”. But this requires some creativity, as can be seen in the article Terraform tips & tricks: loops, if-statements, and gotchas.

Modularisation

Terraform modules support a reproducible infrastructure that can be controlled with parameters. Modules support the export of template values in output parameters that can be used as input parameters for another module.

Terraform supports a multitude of data sources for input parameters. If the desired parameter is not available as an internal data source, scripts can be used to dynamically calculate the parameter as an external data source.

Terraform also offers a module registry with verified community modules. A helpful registry feature is that modules can be accessed via their version number. This helps prevent unwanted changes.

In CloudFormation, modularisation is limited to the use of nested stacks and cross-stack references. Cross-stack references make it possible to export values from a stack generated at runtime. These values can be referenced as parameters in subsequent stacks. Nested stacks consist of CloudFormation templates with hierarchical dependencies.

CloudFormation also allows users to use custom resources to define lambda functions that dynamically provide input parameters.

There is no AWS equivalent to the Terraform module registry. However, there are many CloudFormation templates online that have been provided by AWS and users.

Administrative overheads

Terraform offers well-established ways to create an infrastructure, but does not specify a process. The user is responsible for managing this process and all necessary artefacts. Terraform stores the last updated state of the infrastructure in a JSON file called a state file. The user is responsible for managing this file. If this is not done properly, it can lead to simultaneous parallel updates to the same infrastructure, which will be discussed in the next point. It is also somewhat confusing and there is a risk that the file will be corrupted by manual changes, which can lead to related resources being deleted.

CloudFormation is an AWS-managed service that makes some decisions for the user and makes it easier to use. Though this limits the possibilities, it reduces administrative overheads. CloudFormation manages the current state directly in the AWS account – in the stack. The stack represents the infrastructure provided by CloudFormation. It is easy to view the content of the stack on the user-friendly CloudFormation console. The stack can only be changed by a regular CloudFormation update.

Locking of resources to prevent parallel processing

Since CloudFormation is a managed AWS service, there is no risk of a stack being updated or modified multiple times in parallel.

Since Terraform is not (necessarily) managed from a central location, parallel updates of the infrastructure can occur. Measures must be taken by the user to prevent this. Terraform SaaS or Terraform Enterprise offer established ways to accomplish this.

Importing of infrastructures

It is easy to import resources created outside Terraform into Terraform and create a template from them.

CloudFormation cannot be used to manage or integrate resources that were not created with CloudFormation.

State verification, change overview and change management

This is perhaps the most important function. The Terraform plan command lets the user view and compare the current state of the infrastructure with the planned new state and display a visual representation of the differences. It also detects previous changes to the existing infrastructure that were not made with Terraform. This allows the user to check and confirm planned changes. It facilitates the traceability of changes and creates a high degree of confidence that they will have the desired effect.

Terraform will always change the infrastructure to the state defined in the template. This allows unwanted changes made by others to be undone. On the other hand, Terraform offers no way to keep changes that were made outside of Terraform. But depending on the application, both functions are required.

The drift status function recently introduced by AWS compares the stack with the actual state of the infrastructure to detect changes made outside of CloudFormation. However, drift status does not help undo changes.

When making a change with CloudFormation, there is no way to check it against the actual state of the infrastructure. Changes outside CloudFormation can only be overwritten by replaying a template, provided the template and stack are different. This has the disadvantage that external changes cannot simply be undone. On the other hand, it has the advantage that it is somewhat tolerant to external changes.

Error handling and rollback

If an error occurs while handling a resource, Terraform isolates the error to dependent resources. Non-dependent resources will continue to be created, updated or destroyed as usual. This can result in the infrastructure being in an unusable and unstable state. If the user wants to reset the entire infrastructure to the previous state, this has to be done manually or automated. It is helpful that Terraform backs up the state each time the state is changed.

CloudFormation always (or mostly) tries to keep the infrastructure in a usable and stable state. If a change fails, CloudFormation rolls the infrastructure back to the last stable state. There are some situations where this does not happen, but most of the time the user still has a working infrastructure despite errors.

Rolling updates of Auto Scaling groups

CloudFormation supports rolling updates of Auto Scaling Groups with a rollback should the update fail. Terraform does not support this.

Configuration of wait conditions

CloudFormation allows wait conditions to be defined. This means that all dependent resources are only handled after the resource with the wait condition is created or updated. This makes it possible, for example, to implement a workflow that waits until certain software is installed and available before continuing with the next steps.

Terraform does not inherently support wait conditions, but it is possible to implement them with an external data source.

Multi-cloud support

Terraform currently supports more than 90 providers. This means that a single tool can be used to provide infrastructures for all major public cloud providers such as AWS, Azure, Google Cloud Platform and a variety of cloud native tools such as Kubernetes and Docker.

The use of CloudFormation is limited to AWS.

Tool support and licensing

AWS CloudFormation is free of charge. AWS provides support through its Support Plans.

Terraform is available either as a free open source tool or as a paid business tool. The open source version has no support SLA, but most requests will eventually be answered.

Terraform Enterprise is an integrated IaC tool and features built-in integration with code repositories such as Git. Each commit triggers the plan command, which can be shared and applied. It has well-established methods for managing user and resource states.

Terraform Enterprise is available as a Pro version (SaaS install) and a Premium version (private install). The Pro version comes with 9-to-5 support and the Premium version with 24/7 support. The Premium version includes an unlocked Hashicorp Sentinel for precise control of resource provisioning. It is possible, for example, to specify when and under what circumstances changes can be made.

Terraform recently announced a free version of the SaaS solution that is ideal for small teams and developers. It provides a solution for managing the state file, locking the state, and providing a change history on a user interface.

Personal experience

Something that perhaps only I noticed about Terraform is that at some point users will accidentally delete or modify resources. This is not due to a bug or a lack of documentation from Terraform. In my case, it was due to my initial inexperience with the tool. Terraform provides a resource type called “aws_iam_policy_attachment” that creates exclusive attachments of IAM policies. I used this resource to give an AWS user “admin rights”. It took me a while to figure out why all other AWS users had been deprived of their admin rights – until I read the documentation, which clearly describes the use and implications of using this type of resource. This is by no means a shortcoming or problem. It is simply something to remember because CloudFormation does not provide features that grant exclusivity, and it could also be detrimental to other resources outside the tool.

Conclusion

It is hard to say which tool is better and can be considered the winner. Both tools have their disadvantages, but also workarounds to fix them. In a direct comparison like this, we think Terraform would be the right choice. But we also believe that we cannot only use Terraform and do completely without CloudFormation. Since we also operate an OpenShift cluster as a private cloud, where we use Ansible, OpenShift templates and HELM for IaC, we already have a wide range of tools for similar tasks in the Kubernetes environment. This multitude of tools represents a certain burden for the different teams. Therefore, the next step is to examine whether we can reduce the number of tools. We need a holistic automation concept for all platforms that takes into account not only the tools, but also the teams and their capabilities. Only when we have such a strategy will we possibly introduce Terraform.

Cool tools on the horizon

There is hope for all those who are sceptical about infrastructure as code due to the use of JSON, YAML or HCL. New emerging tools such as Pulumi for Terraform and AWS CDK for CloudFormation, allow infrastructures to be defined with common programming languages and object-oriented methods. Both tools are still new and have their teething pains – but they look promising.