Blog

Why you should ditch YAML and use Jsonnet instead

17 Jan, 2023

I don’t know exactly when the realisation came, but looking at the YAML format and how it is used: it must have been one of the devil’s masterpieces. Similar to flies attracted to a huge pile of cow excrement, developers, me included, are pushed by an invisible force towards this format.

It could have been so easy. I mean, there are many alternatives, typed and untyped, to tame the config mess. But NOOOOOOOOO, let’s use YAML. I think the realisation came when I first saw a Helm chart. It took me a while to get it:

  1. Helm is the package manager for Kubernetes
  2. I write Helm charts as “packages”, which can be reused by others
  3. Nobody ever reuses the chart and if one does, it becomes this type of extremely tight coupling every clean-code blog is warning about
  4. Trial and error till it works
  5. You are nearly there, you just need to get the indentation right

Or it is a so called “Arbeitsbeschaffungsmaßnahme“, a way to expand the work load to get more developers off the streets.

I don’t know.

YAML has an (overly) complex spec, suffers from unintended parsing effects, such as the Norway-Problem, might be unsafe and is often abused to solve problems that shouldn’t be solved in YAML in the first place.

What are the alternatives?

I started to play with Jsonnet and HOCON. HOCON is not much known outside of the Java/Scala bubble and the support in other languages is therefore “meh“.

And Jsonnet? I personally think it has the right balance between power and simplicity.

Introduction Jsonnet

What is Jsonnet? In a nutshell:

A data templating language for app and tool developers

Think of enhanced JSON. Technically it is a superset of JSON with some programming language constructs mixed in. That means Jsonnet needs to be “compiled” or transformed. Usually the output is JSON.

Jsonnet = JSON + Variables, Conditionals, Arithmetic, Functions, Imports and Error Propagation

Example:

local person3 = import 'person3.libsonnet';
{
  person1: {
    name: "Alice",
    welcome: "Hello " + self.name + "!",
  },
  person2: self.person1 { name: "Bob" },
  person3: person3
}

The flow is simple, you start with a Jsonnet file (the main file), where you can import other Jsonnet files (Figure 1). The main file you can compile to JSON.

Jsonnet to JSON Figure 1: Jsonnet to JSON flow (a) Jsonnet has import functionality, you can import other files, often named with .libsonnet as extension, via the import directive, e.g. local martinis = import 'martinis.libsonnet';. (b) The importing configuration file (the main file) is merging all the imports into one configuration object and (c) writes it to JSON output.

More info in the official documentation.

Power vs. Simplicity

A power vacuum will be filled, if you want it or not. In case of YAML that would be e.g. Jinja. But what happens in the developer’s brain if you can use Jinja? The steps are simple:

  1. I have Jinja
  2. I can create my own Python functions and
  3. integrate them in Jina.
  4. Now I can use them in the YAML template
  5. I also skip unit tests, because, hey, it’s configuration, right?

Alternatively I can try to do everything in Jinja and use YAML as “glue”-code.

Result: Spaghetti, snake pieces and chop sticks.

Jsonnet has a natural ceiling: if you cannot solve it in Jsonnet, your problem might be too complex for a configuration. Better switch to a real programming language and abstract away the complexity into a separate module with tests. That keeps the configuration lean. In my opinion Jsonnet has a good “correlation” between looks clean and is clean. Why is this important, you might ask? Similar to code, configuration is a document. Developers read it, extend it, improve it or make a mess out of it. They need to maintain it. Therefore it is better to make wrong code look wrong.

Tooling

However, I have to admit, Jsonnet is a niche config language and the tooling is a bit, let’s say, “quirky”. There is a natural “bump”, keeping others from easy adoption.

First you have the package jsonnet, coming from jsonnet.org/. Unfortunately there are no real installation instructions. github.com/google/jsonnet is more illuminating:

brew install jsonnet
apt install jsonnet
<your favourite pkg manager> install jsonnet

should work.

Additionally there is go-jsonnet, the official go implementation (GitHub repo). It is a drop-in replacement for the C++ implementation. It comes with a formatter (jsonnetfmt) and linter (jsonnet-lint). Handy! (if you know it)

brew install go-jsonnet

The go repo has some installation instructions.

vscode extensions are available, too (I did not use them, though).

Python

I make it very short (because there is not much to tell):

pip install jsonnet
# or you can also use the go version
# pip install gojsonnet

Call it like this:

import _jsonnet
# in case of the go-based pkg use
# import _gojsonnet
import json

res = json.loads(_jsonnet.evaluate_file("info.jsonnet"))

or the official example:

import json
import _jsonnet

jsonnet_str = '''
{
  person1: {
    name: "Alice",
    welcome: "Hello " + self.name + "!",
  },
  person2: self.person1 {
    name: std.extVar("OTHER_NAME"),
  },
}
'''

json_str = _jsonnet.evaluate_snippet(
    "snippet", jsonnet_str,
    ext_vars={'OTHER_NAME': 'Bob'})

json_obj = json.loads(json_str)
for person_id, person in json_obj.items():
  print('%s is %s, greeted by "%s"' % (
      person_id,
      person['name'],
      person['welcome']))

There is more documentation on the bindings page, scroll down to “Python API” and the tests of the Python module. The tests reveal that there are quite some arguments to _jsonnet.evaulate_* functions. A quick glance at the c-code reveals:

static PyObject* evaluate_snippet(PyObject* self, PyObject* args, PyObject *keywds)
{
  ...
  static char *kwlist[] = {
    "filename", "src", "jpathdir",
    "max_stack", "gc_min_objects", "gc_growth_trigger", "ext_vars",
    "ext_codes", "tla_vars", "tla_codes", "max_trace", "import_callback",
    "native_callbacks",
    NULL
  };
  ...
}

Undocumented, unfortunately. The useful ones are probably ext_vars, ext_codes, tla_vars, tla_codes, because they are also mentioned in the official doc. If you want to understand how the import_callback argument is working, have a look at the thread in the Github issue.

As general remark: environment variables need to be passed via the function arguments. That is an advantage: no side effects.

Syntax

Here I’m lazy, just check the official docs (Tutorial, Getting Started, …) . They are quite good if it comes to syntax. Help you can find on StackOverflow.

Workflow and Pipelines

Preparing configuration for multiple environments in a CI/CD pipeline is one of the standard tasks for every developer nowadays. I try to move as much code from the YAML pipeline description into a Makefile or bash script. I do that because it gives me the possibility to run parts of the pipeline locally for debugging purposes. To get an idea, here is an example Makefile:

.PHONY: build fmt clean

SHELL=bash

ENV?=acc

build:
    mkdir -p out
    # Emit multiple files
    # If you use multi-file output, the keys are the file names,
    # `app-config-file-1.json` in this case.
    jsonnet --ext-str env=$(ENV) -m out app-config-files.jsonnet
    # Emit single file to stdout
    jsonnet --ext-str env=$(ENV) app-config.jsonnet >out/app.json
    <out/app.json jq -r '.server' >out/server.txt

fmt:
    jsonnetfmt -i *.jsonnet

clean:
    rm -rf out/

(If needed, I sometimes extract parts of the results into a different file using jq.)

The Makefile uses these jsonnet files:

// jsonnet/app/app-config-files.jsonnet
local env = std.extVar('env');

local config = {
  dev: import 'dev.jsonnet',
  acc: import 'acc.jsonnet',
  prd: import 'prd.jsonnet',
}[env];


{
  'app-config-file-1.json': config,
}

// jsonnet/app/app-config.jsonnet
local env = std.extVar('env');

{
  dev: import 'dev.jsonnet',
  acc: import 'acc.jsonnet',
  prd: import 'prd.jsonnet',
}[env]

// jsonnet/app/prd.jsonnet
{
  server: 'prd.bargsten.org',
}

// jsonnet/app/dev.jsonnet
{
  server: 'dev.bargsten.org',
}

// jsonnet/app/acc.jsonnet
{
  server: 'acc.bargsten.org',
}

You can of course use one file per env and import the required config settings there, flipping around the approach above:

jsonnet $ENV.jsonnet >out/app.json

Conclusions

Have fun!

Logo

(You can find this post also in my personal blog)

Explore related posts