How to terminate lingering packer instances on AWS

HashiCorp Packer is a great tool for building virtual machine images for a variety of platforms including AWS. Normally Packer starts an EC2 instance, builds the AMI on it and terminates the instance on completion. However sometimes the process is aborted and the instance is left running, racking up useless cloud spend. In this blog I present a utility to get rid of old Packer instances.

The aws-hashicorp-packer-reaper utility stops or terminates all Packer Builder EC2 instances. You use it from the command line utility or install it as an AWS Lambda function so that old instances are automatically reaped.

install the reaper

to install the Packer reaper, type:

$ pip install aws-hashicorp-packer-reaper

The utility allows you to list, stop or terminate any EC2 instance which has been tagged with the name Packer Builder. There is no other validation that the instance is indeed a machine started by packer. The name is pretty distinctive, but if you tag your application server with the same name it will be selected too.

show running packer instances

To show running packer instances, type:

$ aws-hashicorp-packer-reaper list

i-06ac951992dbc11a1 (Packer Builder) launched 13 minutes ago - terminated
i-035ebe427a538c829 (Packer Builder) launched 4 minutes ago - running
INFO: 2 packer builder instances found

stop running packer instances

To stop running packer instances older than 2 hours, type:

$ aws-hashicorp-packer-reaper stop --older-than 2h

INFO: stopping i-035ebe427a538c829 (Packer Builder) created 2 hours ago
INFO: total of 1 running instances stopped

If you want to see which instances will be stopped, specify --dry-run.

terminate running packer instances

To terminate stopped and running packer instances older than 24 hours:

aws-hashicorp-packer-reaper --verbose terminate --older-than 24h

INFO: terminating i-035ebe427a538c829 (Packer Builder) created 25 hours ago
INFO: total of 1 instances terminated

If you want to see which instances will be terminated, specify --dry-run.

deploy as a lambda

To deploy the packer reaper as an AWS Lambda, type:

git clone https://github.com/binxio/aws-hashicorp-packer-reaper.git
cd aws-hashicorp-packer-reaper
aws cloudformation deploy 
    --capabilities CAPABILITY_IAM 
    --stack-name aws-hashicorp-packer-reaper 
    --template-file ./cloudformation/aws-hashicorp-packer-reaper.yaml

This will install the packer reaper in your AWS account. It has two schedules which runs the reaper every hour. One to stop instances older than 2 hours:

 AWSHashicorpPackerReaperStopSchedule:
    Type: AWS::Events::Rule
    Properties:
      Name: stop-old-aws-hashicorp-packer-machines
      Description: reap expired Hashicorp Packer Builders
      ScheduleExpression: 'rate(1 hour)'
      State: ENABLED
      Targets:
        - Id: aws-hashicorp-packer-build-reaper
          Arn: !GetAtt AWSHashicorpPackerReaper.Arn
          Input: '{"dry_run": false, "older_than": "2h", "mode": "stop"}'

and one to terminate instances older than 24 hours.

  AWSHashicorpPackerReaperTerminatorSchedule:
    Type: AWS::Events::Rule
    Properties:
      Name: terminate-old-aws-hashicorp-packer-machines
      Description: reap expired Hashicorp Packer Builders
      ScheduleExpression: 'rate(4 hours)'
      State: ENABLED
      Targets:
        - Id: aws-hashicorp-packer-build-reaper
          Arn: !GetAtt AWSHashicorpPackerReaper.Arn
          Input: '{"dry_run": false, "older_than": "24h", "mode": "terminate"}'

Of course you can change these schedules to meet your requirements in the corresponding [CloudFormation template]()

conclusion

The aws-hashicorp-packer-reaper utility stops or terminates all virtual machines with the name tag Packer Builder. I recommend to deploy the Lambda in your AWS account, so that you will never pay for lingering packer instances.

Want to reduce more cost? Read my blog on how to minimize accidental cloud cost in your AWS account