Anti-patterns of using layers with Terraform

In the ever-evolving landscape of cloud computing and infrastructure management, Terraform has emerged as a transformative force, empowering organizations to define, provision, and manage their infrastructure as code (IaC). The versatility of Terraform extends to its project structuring, offering a myriad of approaches, from modular designs and remote state management to utilizing workspaces and version control, providing users with a spectrum of options to tailor their IaC projects to diverse needs.

In Terraform, the typical organizational structure involves modules, which are reusable units of code that encapsulate infrastructure components. These modules can be composed to create more complex infrastructure. There isn’t a standardized or official concept referred to as “Terraform layers” in the Terraform documentation. However, the term has gained popularity in the Terraform community. If you search the tern “Terraform layers” various articles will pop up discussing the concept.

Early stages of Terraform layers

One of the first references of using layers with Terraform was a talk by Armin Coralic (you can find it here).

In his talk, Armin discussed the evolution of using Terraform for IaC and introduced a layered approach to address challenges at different stages. He emphasized three key layers in Terraform code: the bootstrap layer, focusing on setting up the infrastructure; the foundation layer, handling global and foundational aspects; and the service layer, responsible for implementing specific services. This layering strategy aims to provide flexibility, simplify testing, and effectively manage organizational changes in large and complex environments. He finally pointed out the significance of considering both the tools as well as the organizational practices for successful implementation of Terraform in varying contexts.

The layered approach in Terraform can work exceptionally well in scenarios where the infrastructure and organizational needs become more complex and sophisticated. This approach is particularly effective in medium-to-large organizations with multiple teams, diverse responsibilities, and varying lifecycles of infrastructure components. It becomes especially beneficial when addressing challenges such as separating work and responsibilities between teams, managing different lifecycles, and avoiding a monolithic structure. The layering concept is advantageous in facilitating easier testing, promoting flexibility, and adapting to changes that commonly occur in complex organizational settings. Therefore, the layered approach is most suited for environments where the scale and complexity of infrastructure demand a structured and modular approach to Terraform code. One, though, would argue that the same could be achieved with using multiple repositories, abstracting and shifting the responsibility for each module completely to a team.

Evolution of Terraform layers

In the realm of Terraform architecture, two primary approaches emerge: the single monolith (single repo/module) and multiple modules, either split in different repositories or in multiple layers. Each approach comes with its own set of implications, and it becomes imperative to weigh them against the unique needs of your project. Understanding the intricacies of these alternatives empowers you to make informed decisions based on the specific requirements of your target infrastructure’s size, the dynamics of your team, and the desired level of immutability.

Monolith

The single monolith approach offers inherent advantages, such as safety, readability, and a straightforward implementation process. It provides a solid foundation, ensuring the stability of your infrastructure code. This approach is well-suited for scenarios where simplicity and ease of management are paramount. With the monolithic approach, all references to other resources can be passed in a native way without the need to hardcode values in environmental variables, or even worse share them through the various parameter and secret stores.

Layers

On the other hand, opting for multiple layers introduces a new dimension to IaC projects. This approach fosters collaboration, enhances agility, and provides flexibility to cater to future needs. By distributing responsibilities across layers, each dedicated team can focus on a specific aspect of the infrastructure, fostering specialization and efficiency. This approach is particularly valuable for larger-scale projects where the complexity demands a more modular and scalable solution. This is what we discussed in the previous paragraph, right? So where is the problem?

In an era that monolithic approaches do not seem to much the distributed cloud paradigm (unrightfully so as this example points out), there is a trend to split everything into subcomponents. To that end, many teams working with terraform have split their monolithic projects using layers, in an attempt to add modularity. In many cases, teams are using different layers for their networking, compute, database and application components. It could be a lot more layers as well. These layers ([sub-]modules?) have different states  and they have dependencies between each other passed as references to the state further up.

This creates a lot of complexity during the development process. This as an example the case of making a complex change as a result of a new feature request. Such a request could include touching almost all the layers. Due to the nature of the layer structure and the dependency mapping, upper layers need to be applied before lower layers can be planned/applied. Otherwise references to the state may not be found or the actual components may be missing. It also create an issue with carrying out the first deployment which needs to be treated as an edge case making the deployment in brand new environment cumbersome. As a reminder, the initial idea of the layers was to only split in layers components that have different teams as owners, or components that have completely different lifecycles. And as it becomes evident, strong coupling of components between layers can be a really bad idea.

All the previous also complicate the build and deployment process, the flexibility to automate using CI/CD tools. When you need to apply each layer before you can plan the next, you need to split your changes over multiple pull/merge requests, fragmenting next to the IaC also the development process.

A solid use case for using layers

Despite all the above, there are some edge cases that the layered approach can really shine. Terraform plan and apply can end up taking really long when a lot of resource changes need to be calculated or applied. Imagine a situation in the thousands of resources of a certain type changed at once. Could be thousands of buckets or dns records or anything else managed with Terraform. When (and only when) these resources do not carry references between each other, they can be partitioned in separate layers, with their name being the partition key. When combining this with the power from certain CI/CD tools that can identify the subfolders that carry changes, then only a sub-portion of the infrastructure needs to be planned and applied speeding things up by a lot.

Conclusion

Based on all the above, my advise would be that the layered approach in a single repository is an edge case that should be used with caution and only when it fits the target use case a lot better than the alternative options.

When considering the organization of infrastructure in Terraform, it’s crucial to align your approach with the complexity of your project and the dynamics of your team. For simpler projects managed by a single team, opting for a monolithic approach is often the most straightforward choice. It provides a solid foundation for stability, allowing all references to resources to be handled natively without the need for complex configurations or environment variable management. In contrast, for larger and more complex projects involving multiple teams, consider using separate repositories to promote collaboration, specialization, and efficiency across various aspects of the infrastructure. Ultimately, the choice between monolithic and multi-repository approaches should be based on the unique requirements of your project, team structure, and scalability goals.

Keep it simple.

Main image by vectorjuice on Freepik