Unpacking the possibilities of GitHub Packages

Some time ago I forked a github project and published it on Bintray, so it could be used as a dependency by one of my other projects. Because publishing to Bintray was quite a hassle, I decided to take a look at GitHub Packages, which was released last year. My question was, would this be an easier way to publish a package and use it as dependency in another project? Let’s find out.

Publishing to GitHub Packages

There are multiple languages and frameworks supported by GitHub Packages. My project uses Java as programming language and Gradle as build tool. The first thing that had to be done was modifying the "build.gradle" file.

As you can see, credentials are required to publish a package to GitHub. These credentials can be put in a separate file called "gradle.properties" (added to ".gitignore" to keep it private) which contains the following text:

gpr.user=[GITHUB_USERNAME]
gpr.key=[GITHUB_TOKEN]

Getting a GitHub token

To get this GitHub token, it is easy enough to go to your GitHub settings page and generate one, per GitHub’s documentation. It needs to have the correct scopes to publish packages.

Now it is just a matter of opening a terminal in the directory of the project and execute (assuming you are using the Gradle wrapper):

./gradlew publish

That’s it, the package will be published to github. Not too complicated at all.

Adding a GitHub Package as a dependency

Of course, this is only half of the story. Publishing a package is one thing, depending on it from a different software project is another matter. But hey, I thought, this can’t be too complicated. The dependency can stay the same, since the groupId and artifactId haven’t changed, I only need to change the version in the "pom.xml", since I was using Maven as build tool in the project which depended on the first one.

<dependency>
    <groupId>benjaminkomen</groupId>
    <artifactId>jwiki</artifactId>
    <version>${jwiki.version}</version>
</dependency>

Oh and I’ll add my GitHub package location as a repository, so Maven knows from which url to download my the dependency.

<repositories>
    <repository>
        <id>github</id>
        <url>https://maven.pkg.github.com/benjaminkomen/jwiki</url>
    </repository>
</repositories>

I committed and pushed my changes, but then my Travis CI build errored: Failed to collect dependencies at benjaminkomen:jwiki:jar:2.2.0: Failed to read artifact descriptor for benjaminkomen:jwiki:jar:2.2.0: Could not transfer artifact benjaminkomen:jwiki:pom:2.2.0 from/to github (<a href="https://maven.pkg.github.com/benjaminkomen/jwiki"><a href="https://maven.pkg.github.com/benjaminkomen/jwiki">https://maven.pkg.github.com/benjaminkomen/jwiki</a></a>): Authentication failed for <a href="https://maven.pkg.github.com/benjaminkomen/jwiki/benjaminkomen/jwiki/2.2.0/jwiki-2.2.0.pom"><a href="https://maven.pkg.github.com/benjaminkomen/jwiki/benjaminkomen/jwiki/2.2.0/jwiki-2.2.0.pom">https://maven.pkg.github.com/benjaminkomen/jwiki/benjaminkomen/jwiki/2.2.0/jwiki-2.2.0.pom</a></a> 401 Unauthorized -> [Help 1]

Unauthorized? I came to the horrifying realisation that you actually need a GitHub token with a read:packages scope, to fetch a dependency from GitHub packages during the (Maven) build process. This is actually quite inconvenient.. But let’s not give up, let’s try to solve this.

Authenticating to GitHub Packages

Of course it is all explained in the documentation (but who reads that upfront?). From your terminal simply execute:

subl ~/.m2/settings.xml

Note: ‘subl’ stands for ‘Sublime Text’, you can alternatively use ‘vim’ or any other text editor.

Then copy-paste the settings XML code from the documentation into your local "setting.xml" file and replace some placeholders. It is advisable to not reuse the GitHub token previously created to publish to GitHub Packages. Following the principle of least privilege, it is better to create a new GitHub token, which only has the scope to read packages and not publish them.

Saving this "settings.xml" file makes your local build work, but to make the Travis CI build succeed, more work needed to be done. The solution I found was simple in theory: during a build, copy the "settings.xml" file to the correct location on the build server. This way the GitHub package can be fetched during the build. However, the repository which needed to be built is public, so the "settings.xml" file should not contain the GitHub token. Even though it only provides read access, it feels weird to have a personal token publicly available. Luckily Travis allows to add environment variables via their settings page.

So I added a file called ".travis.settings.xml" to the project, which did not contain the GitHub token, but simply a reference to the environment variable ${env.GITHUB_TOKEN} (see it here). Furthermore the ".travis.yml" file had to be changed to copy the ".travis.settings.xml" file to the correct location during the build.

Interestingly enough, this needs to happen during the install step, not the script step.

And voila, this works! During a Travis build, a "settings.xml" is copied to the build server and an environment variable is used to store the required GitHub token.

The saga continues – Google Cloud

Of course, Travis was only one of the places where this project was built. When merging to the master branch, a trigger causes the project to be built on the Google Cloud by Cloud Build. This creates a Docker image to be run in Cloud Run. Yet another place which needed authorization to download my proudly published GitHub package.

I decided to go with a solution where I could reuse the "settings.xml" file I used for Travis and store the GitHub token in Cloud KMS, the Google Cloud way to store secrets (Note: since January 2020 the Secret Manager could be used alternatively).

Using Cloud Key Management Service

First of all the Cloud KMS API should be enabled. Then a keyring and a key need to be created. I just conveniently used the Cloud Shell from the Google Cloud Dashboard. To create a keyring with the name "my-secrets", you execute in the Cloud Shell terminal:

gcloud kms keyrings create my-secrets --location global

Then to create a key named "github-token" in this keyring and view it execute:

gcloud kms keys create github-token --location global --keyring my-secrets --purpose encryption
gcloud kms keys list --location global --keyring my-secrets

Now the actual value of the GitHub token can be added, or actually, the encrypted value. First the GitHub token must be put in a plaintext file, simply execute:

echo "[GITHUB TOKEN]" > github_token.txt

Then the actual encryption can be done with:

gcloud kms encrypt \
  --plaintext-file=github_token.txt \
  --ciphertext-file=github_token.enc.txt \
  --location=global \
  --keyring=my-secrets \
  --key=github-token

For the Cloud Build, a base64 encoded version of the encrypted token is needed, which then can be printed in the terminal to copy-paste and use:

base64 github_token.enc.txt -w 0 > github_token.enc.64.txt
cat github_token.enc.64.txt

You can copy-paste the output of the previous command for later use. Then it is wise to delete the three text files which were created, since they will persist for a while.

Configuring Cloud Build

Good, all is done to have our GitHub token stored safely, to be used during the Cloud Build. The Cloud Build process might need permission to access your CryptoKey during building, but that can be done easily. Furthermore, the "cloudbuild.yaml" file needed several modifications.

The bottom part, after secrets contains the part where the GitHub token, stored as encrypted key in Cloud KMS, is read so it can be used as environment variable. Here, the base64 encoded version of the encrypted GitHub token is used, per the documentation.

Furthermore, the GitHub token can now be considered a secret environment variable which can be passed to the "Dockerfile" as build argument. The build stage of this Docker image is the place where Maven pulls in dependencies and needs to have access to GitHub packages. So the "Dockerfile" changes in the following manner:

It receives the GITHUB_TOKEN as argument from the Cloud Build file. It then publishes this as environment variable. Finally it copies the "settings.xml" file to the correct location, which can now use the GitHub token environment variable to fetch the GitHub package we published earlier.

So that’s that, Cloud Build is now also set up to fetch GitHub packages during building!

Conclusion

What have we found out? Publishing to GitHub packages is relatively easy. Creating a GitHub token and changing some configuration is all it takes.

Unfortunately, using a GitHub package as a dependency is harder. Unlike other dependencies (e.g. from Maven central), authentication is required to download a GitHub package during the build of a program. Quite some configuration is required to get this to work. This needs to be done on every build environment where this dependency is used.

The fact that this makes it so much more harder, makes GitHub Packages less attractive to use.

But hey, who knows, maybe I’m using GitHub packages completely the wrong way? Maybe it is mostly intended to be used in GitHub Actions, where you don’t have to worry about a GitHub token? I’d love to hear your thoughts!