Blog

Bootstrapping and monitoring multiple processes in Docker using monit

11 Jul, 2014

If you have every tried to start a docker container and keep it running, you must have encountered the problem that this is no easy task. Most stuff I like to start in container are things like http servers, application servers and various other middleware components which tend to have start scripts that daemonize the program. Starting a single process is a pain, starting multiple processes becomes nasty. My advise is to use monit to start all but the most simple Docker application containers! When I found monit while delving through the inner works Cloud Foundry, I was ecstatic about it! It was so elegant, small, fast, with a beautiful DSL that I thought it was the hottest thing since sliced bread! I was determined to blog it off the roof tops. Until…. I discovered that the first release dated from somewhere in 2002. So it was not hot and new; Clearly I had been lying under a UNIX rock for quite a while. This time, the time was right to write about it! Most of the middleware components I want to start in a docker container, have a habit to start the process, daemonize it and exit immediately, with the docker container on its tail. My first attempt to circumvent this while starting a tomcat server in Docker looked something like this:

/bin/bash -c "service tomcat7 start;while service tomcat7 status;do sleep 1;done

Quite horrific. Imaging the ugliness when you have to start multiple processes. A better solution is needed:  With the zabbix docker container  the problem was solved using simplevisor. As you can read in this post that was not a pleasant experience either. As I knew little about simplevisor and could not solve the problem, I put in an issue and  resorted to a plain installation. But a voice in my head started nagging: “Why don’t you fix it and send a pull request?”  (actually, it was the voice of my colleague Arjan Molenaar). Then, I remembered from my earlier explorations to the inner workings of Cloud Foundry, a tool that would be really suitable for the job: monit. Why? It will:

  1. Give you a beautiful,readable specification file stating which processes to start
  2. Make sure that your processes will keep on running
  3. Deliver you a clean and crisp monitoring application
  4. Reduce all your Docker starts to a single command!

In the case of the Zabbix server there were seven processes to start: the zabbix server, agent, java agent, apache, mysql and sshd. In monit this looks as follows:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
        start program = "/sbin/service mysqld start"
        stop program = "/sbin/service mysqld stop"
check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid
        start program = "/sbin/service zabbix-server start"
        stop program = "/sbin/service zabbix-server stop"
        depends on mysqld
check process zabbix-agent with pidfile /var/run/zabbix/zabbix_agentd.pid
        start program = "/sbin/service zabbix-agent start"
        stop program = "/sbin/service zabbix-agent stop"
check process zabbix-java-gateway with pidfile /var/run/zabbix/zabbix_java.pid
        start program = "/sbin/service zabbix-java-gateway start"
        stop program = "/sbin/service zabbix-java-gateway stop"
check process httpd with pidfile /var/run/httpd/httpd.pid
        start program = "/sbin/service httpd start"
        stop program = "/sbin/service httpd stop"
        depends on zabbix-server
check process sshd with pidfile /var/run/sshd.pid
        start program = "/sbin/service sshd start"
        stop program = "/sbin/service sshd stop"

Normally when you start monit it will start as a daemon. But fortunately, you can prevent this with the following configuration.

set init

Your Dockerfile CMD can now always look the same:


    monit -d 10 -Ic /etc/monitrc

Finally, by adding the following statement to the configuration you get an application to view the status of your container processes,

set httpd
     port 2812
     allow myuser:mypassword

After starting the container, surf to port 2812 and you will get a beautiful page showing the state of your processes and the ability to stop and restart them.

monit overview monit process control

Just delve into the documentation of monit and you will find much more features that will allow you to monitor network ports and files, start corrective actions and send out alerts.
Monit is true to its UNIX heritage: it is elegant and promotes an autonomous monitoring system. Monit is cool!

guest
14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Nico
Nico
8 years ago

Very nice, I’ve tried this today and it works fine. However, when I stop a running container, and then start it again, monit doesn’t start again and the container immediately quits. Am I missing something?

Kees de Kooter
Kees de Kooter
8 years ago

Just what I was looking for, thanks! Would it be possible to share the Dockerfile with us?

jnc
jnc
7 years ago

Hi Mark,
Thanks for this article, really nice.
I’m facing the following issue when I stop my container seems like that process started by monit are getting killed as when I restart it I can only see monit up.
Then monit is unable to restart these process as the pid files haven’t been deleted.
Is there a smart way to have these pid files removed or to force monit to restart the process ?
Cheers
JN

jnc
jnc
7 years ago

Thank Mark for the quick reply.
Maybe it will be easier to explain with the container status when it starts for the first time.
My entry point is a shell script which starts monit and bash.
Monit starts correctly a wildfly server and a h2 DB nice.
root 1 0 0 15:55 ? 00:00:00 /bin/bash /opt/startup.sh
root 9 1 0 15:55 ? 00:00:00 monit -d 10 -c /etc/monit/monitrc
root 10 1 0 15:55 ? 00:00:00 bash
root 42 1 0 15:55 ? 00:00:00 /bin/sh /opt/wildfly
root 74 42 99 15:55 ? 00:00:14 /usr/lib/jvm/java-7-orac
root 93 1 4 15:55 ? 00:00:00 java -Xms256m -Xmx1024m
If I stop my container and restart it, here is what I got.
root 1 0 0 16:00 ? 00:00:00 /bin/bash /opt/startup.sh
root 9 1 0 16:00 ? 00:00:00 monit -d 10 -c /etc/monit/monitrc
root 10 1 0 16:00 ? 00:00:00 bash
Monit then tries to restart wildfly and h2 but the former pid files (ids 42 / 93..) are still there and it fails.
What I don’t get is why they haven’t been deleted when monit received the stop status during container stop (I guess it received it).
If I use monit stop all in command line or use the web interface, I’m able to restart the services properly.
Thanks
JN

david
david
7 years ago

In my case, monit wrote more havoc than helped.
I’m monitoring 2 processes: nginx and mysql, and after a few start/restarts/crashes, mysql’s pid got reused by another process, and monit was happily reporting that everything is 5×5, while in reality mysql was not running.
I believe monitoring by pid is inherently wrong, but not sure if there’s any software that is smarter than monit – perhaps automatically recording and matching command line would be a way to go..

David
David
7 years ago

Thanks Mark,
To add some details, this is my monit.conf:
check process nginx with pidfile /run/nginx.pid
start program = “/usr/sbin/nginx -c /etc/nginx/nginx.conf” ..
stop program = “/usr/sbin/nginx -s stop”
check process mysqld with pidfile /run/mysqld.pid
start program = “/usr/sbin/mysqld –user=root” ..
stop program = “/usr/bin/bash -c ‘kill `cat /run/mysqld.pid`'”
Should this work?
* The example on this page has `/sbin/service mysqld start`, but in my container /sbin/service does not work: (/bin/systemctl: No such file or directory)
I believe docker containers do not expect services, so something extra needs to be installed for /sbin/service to work.
* Also, looking at Docker’s official mysql image, starting mysql with simple `mysqld` is correct way to go, not sure if this should work with monit though.
* I tried adding ‘&’ at the end of mysql command – that didn’t work.
* Executing `mysqld` outside of monit works fine.
both nginx and mysqld start fine, but then mysqld dies for no apparent reason, and shows as “defunct” in `ps` output.
and then monit fails to restart it, it seems confused., probably because the pid file is still there.
Thanks.

Gary Wisniewski
Gary Wisniewski
7 years ago

We’ve just released a process manager that runs as PID#1 in containers, designed specifically for docker. It’s just one process, but handles syslog redirection, cron jobs, dependency-based startup, and even emulates systemd notify, but it’s very very lean. I’d love any feedback, I believe it can do all the above quite well. Best place to give a quick look is here: https://github.com/garywiz/chaperone-docker

Explore related posts