Refactoring Terraform with a little help from Python

Terraform gives you a nice way to orchestrate your Cloud resources. It allows you to organise multiple resources into reusable modules or even separate Terraform runs with data resource as the glue.

Maintainability

Just like all code it can be challenging to keep maintenance of it from being someone or your own worse nightmare. Typically I find myself moving resources as much as creating them.
Especially at the beginning of a project or product, when you create lots of resources. Requirements change over time and the change in infrastructure is the effect.

Refactoring

After some time I usually realise that some resources would be better off in another module or even in another Terraform run. Advantage of using the Cloud is that you can do everything immutable, so just throw it all away you could say. But it’s not always convenient to do this using the immutable mindset.
Simply destroying and recreating is not wanted in various cases, for example:
VPN connections with already shared keys, shared VPC where lots of instances depend upon, components used in Live running apps or resources that just take forever to create and you don’t feel like waiting.

Example: Mutating State Manually

• You make it work for dev.
• Then you need a test environment, you create a module out of it and deploy 2 versions.
• Some time later the thing breaks in dev and your test env state is blocked.
• You want to move one env to another state file location.
  This is a fairly simple to task to do with state move.

Usage: terraform state mv [options] SOURCE DESTINATION
Perfectly fine when you’re moving 1 or 2 resources, changing some more can cause serious headaches. Typo’s can cause strange things to happen, automation again is your friend.

How to Automate Lots of Changes

When you are moving around multiple resources a script to extract the from config is easier and less error prone.
Terraform uses HCL, a configuration language authored by HashiCorp. The language was created with the goal of being both human and machine friendly.
Thanks to the python-hcl2 Python package we can read the structure easily and generate terraform command line commands.

TL;DR;

I just want to move resources to different state locations without recreating them and not be typing all the resource names.
Installing virtual environment and dependencies in a directory of you choice:

virtualenv -p python3 venv
. venv/bin/activate
pip install fire python-hcl2

Copy over script to be used:

import fire
import hcl2

class Parser():
    def hcl2(self, file, state="terraform.tfstate", state_out="to.tfstate"):
        with(open(file, 'r')) as file:
            dict = hcl2.load(file)

        for modules in dict.get("module", []):
            for module in modules.keys():
                print(modules.keys())
                print(f"terraform state mv -state {state} -state-out {state_out} module.{module} module.{module}")

        for resources in dict.get("resource", []):
            for resource in resources.keys():
                for resource_name in resources[resource].keys():
                    print(
                        f"terraform state mv -state={state} -state-out={state_out} {resource}.{resource_name} {resource}.{resource_name}")

if __name__ == '__main__':
    fire.Fire(Parser)

python ~/parse.py hcl2 src/saml-roles.tf

Generates state mv command for terraform to be used, piping them to bash (python ~/parse.py hcl2 src/saml-roles.tf | bash) will execute the move straight away,

terraform state mv -state terraform.tfstate -state-out to.tfstate module.iam module.iam
terraform state mv -state=terraform.tfstate -state-out=to.tfstate aws_vpc.this aws_vpc.this
…
terraform state mv -state=terraform.tfstate -state-out=to.tfstate aws_default_vpc.this aws_default_vpc.this

Conclusion

The goal for all of this is to see you more comfortable in changing resource organisation. If you don’t keep Infra as Code clean it might be that you shoot yourself in the foot.
If that doesn’t scare you, please do it for your teammembers. Hope the script helps, Terraform provides an easy way to move resources and using this feature keeps your infrastructure automation in shape.