Blog

Persistence with Docker containers – Team 1: GlusterFS

17 Aug, 2015

This is a follow-up blog from KLM innovation day
The goal of Team 1 was to have GlusterFS cluster running in Docker containers and to expose the distributed file system to a container by ‘mounting’ it through a so called data container.
Setting up GlusterFS was not that hard, the installation steps are explained here [installing-glusterfs-a-quick-start-guide].

The Dockerfiles we eventually created and used can be found here [docker-glusterfs]
Note: the Dockerfiles still containe some manual steps because you need to tell GlusterFS about the other node so they can find each other. In an real environment this could be done by for example Consul.
Although setting up GlusterFS cluster was not hard, mounting it on CoreOS proved much more complicated. We wanted to mount the folder through a container using the GlusterFS client but to achieve that the container needs to run in privileged mode or with ‘SYS_ADMIN’ capabilities. This has nothing to do with GlusterFS it self, Docker doesn’t allow mounts without these options. Eventually mounting of the remote folder can be achieved but exposing this mounted folder as Docker volume is not possible. This is an Docker shortcoming, see docker issue here
Our second – not so prefered – method was mounting the folder in CoreOS itself and then using it in a container. The problem here is that CoreOS does not have support for GlusterFS client but does have NFS support. So to make this work we exposed GlusterFS through NFS, the step to enable it can be found here [Using_NFS_with_Gluster].  After enabling NFS on GlusterFS we mounted the exposed folder in CoreOS and used it in a container which worked fine.
Mounting GlusterFS through NFS was not what we wanted, but luckily Docker released their experimental volume plugin support. And our luck did not end there, because it turned out David Calavera had already created a volume plugin for GlusterFS. So to test this out we used the experimental Docker version 1.8 and run the plugin with the necessary settings. This all worked fine but this is where our luck ran out. When using the experimental Docker daemon, in combination with this plugin, we can see in debug mode the plugin connects to the GlusterFS and saying it is mounting the folder. But unfortunately it receives an error, it seems from the server and then unmounts the folder.
The volume plugin above is basically a wrapper around the GlusterFS client. We also found a Go API for GlusterFS. This could be used to create a pure Go implementation of the volume plugin, but unfortunately we ran out of time to actually try this.
Conclusion:
Using distributed files system like GlusterFS or CEPH sounds very promising especially combined with the Docker volume plugin which hides the implementation and allows to decouple the container from the storage of the host.
Update:

Between the innovation day and this blog post the world evolved and Docker 1.8 came out and with it a CEPH docker volume plugin.

guest
3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Milo van der Linden
Milo van der Linden
6 years ago

The folks at rancher.com have a excellent solution, they have paired glusterfs with convoy. I am running glusterfs inside my containers without any privileges on the hosts without problems.

Josh
Josh
5 years ago

How have you found performance for this? I’m using GlusterFS and Convoy on 3 VPS hosts and I’ve found the performance to be very slow. I’m not sure if it’s the Overlay network or the lack of computational power of the VPSs. What are your hosts and how fast is your network?

Jaime Garza
Jaime Garza
6 years ago

Hi, I managed to create an scenario that could mount gluster servers inside containers without too many hacks and that works also on top of kubernetes by following the sidecar pattern, could you tell me if you see any disadvantages of the work I’m doing?
Here is the tutorial I made: https://medium.com/@jaime_ecom/glusterfs-cluster-with-kubernetes-c09725d69900#.18m72xv4k

Explore related posts