Blog

Package management with Nix

19 Dec, 2019
Xebia Background Header Wave

As software engineers we use package managers on a daily basis. We use them to install dependencies we need to run and build software we write. Probably every software engineer can relate to the frustration that will eventually arise from using these package managers. Sometimes packages that seem to work on your colleagues machine just fine, are broken on yours. Even though package managers have improved substantially over time, issues like these still arise. Maybe there is some fundamental design flaw in the way we approach package management. There is a package manager that tries to do things different and it is called Nix. Let’s take a look at what Nix is and how you can use it on your machine today.

Enter the Nix

Nix is many things, a language, an operating system (nixos),  a deployment system, and a continuous integration system (Hydra). But at its core it is a package manager, specifically a package manager that is designed from the ground up to be pure and immutable. Pure in the sense that building a package can not perform any uncontrolled side-effects, and immutable in the sense that no global package state is mutated in place. This is a fundamental shift from the current approach to installing packages where we install all binaries to a global and mutable directory e.g. /usr/local/bin.This shift allows us to support different versions of the same executable or library on our operating system.

Nix is pure, which means it can distinguish the same package build with different input parameters. It captures all input parameters, which could be anything from the version of GCC, to the library dependencies (e.g. glibc). So in Nix a bundled package is not just referenced by it’s name and version, but is prefixed with the aggregation of all parameters used to build that package.  This aggregation is represented as a hash, so when any of the input parameters changes this hash will also change.

Referencing binaries by these hashes would be really unproductive, we want to be able to type git in our terminal and not 86b4f4cd27e63c8e05c745807dd281ec04c38521-git. Nix solves this problem by creating a symlink under the name git inside a directory exposed on your PATH to the actual location of the git binary in the nix store. A collection of such symlinks is called a profile.

Symlinks enable immutability. Because we no longer have to mutate the binaries directly inside directories exposed on our PATH (e.g. /usr/local/bin), but instead rely on the /nix/store directory. And atomic operations, because the creation of symlinks happen as a final step in our installation procedure. So an installation that fails half-way through will not have any noticeable side-effects.

Tutorial

In this tutorial I will explain how you can use nix to install binaries and create isolated and reproducible development environments. It is, by no stretch of the imagination, a complete guide on how to use Nix. But hopefully it will capture your imagination of Nix’s possibilities.Install the nix package manager:$ curl https://nixos.org/nix/install | shAfter installation we have some new commands available:

  • nix-env used to install dependencies and alter your current profile
  • nix-shell used to start shells based on nix expressions
  • nix-store used to query or manipulate the nix store

And a directory has been created for the nix store at /nix, this is directory containing all installed packages, profiles and configuration. Your global environment will only be mutated by symlinking into the nix-store.We can now execute several actions to search, install, and uninstall packages:

Search for packages starting with nodejs in the remote repository

$ nix-env -qaP 'nodejs.*'

Install Node 12 in your profile

$ nix-env -iA nixpkgs.nodejs-12_x

Running this command will show that node’s binary is reachable from a directory called .nix-profile. If you follow the symlinks in the directory you will end up at a binary in the nix store somewhere in /nix/store.

$ which node

You can inspect the /nix/store directory to see all the dependencies that are currently installed on your machine. You will see that all packages are prefixed by a hash, which as mentioned in the first section represent all the input parameters used to build this package.

When we install a second version of Node it would override the symlink in our nix-profile.

$ nix-env -iA nixpkgs.nodejs-11_x

When you run node –version it will tell you node is bound to version 11.

$ node --version

Since the binary of node is a symlink to the actual binary stored within the nix store (at /nix), we can also load an isolated shell which is points node to a specific version of node in the nix store.

Start a new shell in which nodejs12 is available, once you exit the shell nodejs-12 will no longer be on your path.

$ nix-shell -p nodejs-12_x

With the nix language it is also possible to write a configuration file that we can reuse. These configuration files are called derivations and are the most important building block of nix. They describe how package are to be build and with what parameters. They are written in the nix language. Take for example this derivation which will load nodejs v12 in the environment:

with import <nixpkgs> {};
stdenv.mkDerivation {
  name = "node-environment";
  buildInputs = [
    pkgs.nodejs-12_x
  ];
}

If you want to learn more about writing nix derivations see the nix-pill on writing derivations

If we store this file as default.nix we can load it inside a nix shell with the following command:

$ nix-shell default.nix

We can now create reproducible development environments by writing a nix derivation, to enter the environment we still require an invocation of nix-shell, we could also leverage a tool called direnv to automatically load the nix derivation while entering the directory containing our default.nix derivation.

Bonus: in tandem with direnv

direnv is an environment management tool developed by zimbatm. It essentially sets and removes environment variables in the shell depending on the current directory and the presence of a .envrc file.

Before each prompt, direnv checks for the existence of a .envrc file in the current and parent directories. If the file exists (and is authorized), it is loaded into a bash sub-shell and all exported variables are then captured by direnv and then made available to the current shell.

Since direnv supports nix out of the box we only need to create a .envrc and enable the nix integration, to do this run the following in the directory containing your default.nix file:

$ direnv edit .

Add the following line to the file to enable nix integration, and save:

use nix

Once the file is saved direnv will automatically reload the environment we should see something along these lines:

direnv: loading .envrc
direnv: using nix
direnv: export +AR +AS +CC +CMAKE_OSX_ARCHITECTURES +CONFIG_SHELL +CXX +HOST_PATH +IN_NIX_SHELL +LD +LD_DYLD_PATH +MACOSX_DEPLOYMENT_TARGET +NIX_BINTOOLS +NIX_BINTOOLS_WRAPPER_x86_64_apple_darwin_TARGET_HOST +NIX_BUILD_CORES +NIX_BUILD_DONT_SET_RPATH +NIX_BUILD_TOP +NIX_CC +NIX_CC_WRAPPER_x86_64_apple_darwin_TARGET_HOST +NIX_CFLAGS_COMPILE +NIX_COREFOUNDATION_RPATH +NIX_CXXSTDLIB_COMPILE +NIX_CXXSTDLIB_LINK +NIX_DONT_SET_RPATH +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_IGNORE_LD_THROUGH_GCC +NIX_INDENT_MAKE +NIX_LDFLAGS +NIX_NO_SELF_RPATH +NIX_STORE +NM +NODE_PATH +PATH_LOCALE +RANLIB +SDKROOT +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP +TEMP +TEMPDIR +TMP +__darwinAllowLocalNetworking +__impureHostDeps +__propagatedImpureHostDeps +__propagatedSandboxProfile +__sandboxProfile +buildInputs +builder +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +gl_cv_func_getcwd_abort_bug +name +nativeBuildInputs +out +outputs +patches +propagatedBuildInputs +propagatedNativeBuildInputs +shell +stdenv +strictDeps +system ~PATH

If we run which node we should see that node is now loaded from the nix store:

/nix/store/9pfsxwkr43907dp7spwidpr3hjbz0v5w-nodejs-12.5.0/bin/node

Conclusion

Managing multiple binary versions or reproducible build environments is just the start of nix’s capabilities. It can be used to manage entirely reproducible operating systems or to create reproducible container images. Nix is in potential a great tool for managing reproducible development environments. While the tooling is still a bit rough from a UX perspective and the package registry support could be better on OS X. The philosophy of nix shows great potential for the future iterations of it’s operating system and package manager.

Caveats

  • Nix is only supported on Linux and Mac OS, and currently the derivation support for Linux is superior to that of Mac OS.
  • Since the Nix store is immutable it is not possible to install global dependencies of your programming language package manager inside the nix store. As this is exactly what npm will try to do when you run npm install -g this operation will fail. The following blog post describe a way in which you can circumvent this issue: https://nicknovitski.com/nix-npm-install

Additional material

Matthisk Heimensen
A hands-on engineering consultant developing software across the stack. Helping teams deliver more predictably with more fun.
Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts