How to create a private terraform provider registry using Google Cloud Storage

Since version 0.13 terraform supports the notion of a provider registry. You can build and publish your own provider in the public registry. At the moment, there is no private registry implementation available. In this blog, I will show you how to create a private terraform provider registry on Google Cloud Storage. A utility will generate all the required documents to create a static registry.

Before I explain how to create your own terraform provider registry, I will show you the expected protocol observed by a registry. Then I will show the anatomy of a terraform provider release. Finally, I will show you how to create the registry and release a provider to it.

terraform provider registry protocol

The protocol of the registry, consists of the three document types:

• discovery document per registry
• versions document per provider
• download document per document

In the following sections I will show samples of these document types.

the discovery document

This document allows terraform to query the endpoint of the resources::

$ curl $REGISTRY_URL/.well-known/terraform.json
{"providers.v1": "/v1/providers/"}

the versions document

The versions documents returns the available versions and platform binaries per provider.

$ curl $REGISTRY_URL/v1/providers/jianyuan/sentry/versions
{
  "versions": [{
      "version": "0.6.0",
      "protocols": ["5.0"],
      "platforms": [{"os": "darwin","arch": "amd64"}, {"os": "linux","arch": "amd64"}]
    }]
}

the download document

Finally, the download document contains the metadata needed to download a platform binary.

$ curl $REGISTRY_URL/v1/providers/jianyuan/sentry/0.6.0/download/darwin/amd64
{
  "protocols": [
    "5.0"
  ],
  "os": "darwin",
  "arch": "amd64",
  "filename": "terraform-provider-sentry_0.6.0_darwin_amd64.zip",
  "download_url": "$REGISTRY_URL/binaries/jianyuan/terraform-provider-sentry/v0.6.0/terraform-provider-sentry_0.6.0_darwin_amd64.zip",
  "shasums_url": "$REGISTRY_URL/binaries/jianyuan/terraform-provider-sentry/v0.6.0/terraform-provider-sentry_0.6.0_SHA256SUMS",
  "shasums_signature_url": "$REGISTRY_URL/binaries/jianyuan/terraform-provider-sentry/v0.6.0/terraform-provider-sentry_0.6.0_SHA256SUMS.sig",
  "shasum": "a2c5881ea67e1c397cb26c6162d81829e058d5a993801bcb69df9982412d27e9",
  "signing_keys": {
    "gpg_public_keys": [
      {
        "key_id": "8B15B898C0AA84DC7A7B0E46B851229EAFE0F521",
        "ascii_armor": "-----BEGIN PGP PUBLIC KEY BLOCK-----...-----END PGP PUBLIC KEY BLOCK-----",
        "trust_signature": "",
        "source": "",
        "source_url": null
      }
    ]
  }

So, now we know what is required lets take a look at a terraform provider release.

terraform provider releases

A terraform provider release is a collection of zipped binaries. A new release of a provider which meets the requirements of terraform looks like this:

terraform-provider-sentry_0.6.0_SHA256SUMS
terraform-provider-sentry_0.6.0_SHA256SUMS.sig
terraform-provider-sentry_0.6.0_darwin_amd64.zip
terraform-provider-sentry_0.6.0_freebsd_386.zip
terraform-provider-sentry_0.6.0_freebsd_amd64.zip
terraform-provider-sentry_0.6.0_freebsd_arm.zip
terraform-provider-sentry_0.6.0_freebsd_arm64.zip
terraform-provider-sentry_0.6.0_linux_386.zip
terraform-provider-sentry_0.6.0_linux_amd64.zip
terraform-provider-sentry_0.6.0_linux_arm.zip
terraform-provider-sentry_0.6.0_linux_arm64.zip
terraform-provider-sentry_0.6.0_windows_386.zip
terraform-provider-sentry_0.6.0_windows_amd64.zip

A release consists of a:

• zip file containing the built provider binary for each architecture
• checksum file with the checksum for each of the zip files
• signature file containing the signature of the checksum file.

The only thing missing in the distribution is the public key used to sign the checksum file.

the terraform provider registry generator

As shown in the previous paragraph, a release contains everything to implement the terraform provider registry protocol. The terraform provider registry api generator generates all required JSON documents. It reads a distribution on the bucket, loads the PGP public key and writes all required documents.

create your own private terraform provider registry

To create your own private terraform provider registry, configure a cloud storage bucket to host a static website as follows:

$ git clone https://github.com/mollie/tf-provider-registry-api-generator.git
$ cd tf-provider-registry-api-generator/example/terraform
$ terraform init
$ read -p "project id:" TF_VAR_project_id
$ read -p "dns zone name:" TF_VAR_dns_managed_zone
$ terraform apply

To obtain the registry URL, type:

$ REGISTRY_URL=https://registry.$(gcloud dns managed-zones 
     describe $TF_VAR_dns_managed_zone 
    --format 'value(dns_name)' | sed -e 's/.$//')

After a few minutes you should get a 404 on$REGISTRY_URL.

build and release to your own private registry

For the purpose of simplicity, I am going to build and release an existing terraform provider to my private registry. To do so, I need to:

• build the provider
• copy the release the bucket
• generate the terraform provider registry api documents

build the terraform provider

To build the terraform provider, type:

$ git clone https://github.com/jianyuan/terraform-provider-sentry.git
$ cd terraform-provider-sentry
$ git checkout v0.6.0

$ curl -sS -o /tmp/goreleaser.yaml  https://raw.githubusercontent.com/hashicorp/terraform-provider-scaffolding/main/.goreleaser.yml
$ read -p  "pgp key fingerprint:" PGP_FINGERPRINT
$ goreleaser release -f /tmp/goreleaser.yaml --rm-dist

copy the release the bucket

To copy the newly built release onto the bucket, type:

$ TF_REGISTRY_BUCKET=${TF_VAR_project_id}-tf-registry
$ gsutil cp . 
  gs://$TF_REGISTRY_BUCKET/binaries/terraform-provider-sentry/v0.6.0/

generate the terraform provider registry API documents

To generate the terraform provider registry API documents, type:

$ go get github.com/mollie/tf-provider-registry-api-generator
$ $GOPATH/bin/tf-provider-registry-api-generator 
  --bucket-name $TF_REGISTRY_BUCKET 
  --prefix binaries/terraform-provider-sentry/v0.6.0/ 
  --namespace jianyuan 
  --fingerprint $PGP_FINGERPRINT 
  --protocols 5.0 
  --url $REGISTRY_URL

Now you should be able to query the registry:

$ curl $REGISTRY_URL/.well-known/terraform.json
$ curl $REGISTRY_URL/v1/providers/jianyuan/sentry/versions
$ curl $REGISTRY_URL/v1/providers/jianyuan/sentry/0.6.0/download/darwin/amd64

Note that all the above steps can be combined into a single go release specification.

referencing your private registry

To use the provider from your private registry, add the following snippet to your terraform definition:

terraform {
  required_providers {
    sentry = {
      source = "<DOMAIN_NAME>/jianyuan/sentry"
    }
  }
}

conclusion

For most purposes, I recommend to use the public terraform provider registry. It is easy to use, turn-key solution. If for some reason you want to keep your providers private, it is quite easy to make your own private provider registry.

Photo by Dayne Topkin on Unsplash