The ultimate continuous delivery deploy(it) toolkit

Putting software in production can be a challenge, often the frequency of going to production is low and the amount of changes/features involved is high. This usually results in a long and painfull deployment process. The release probably contains as many changes as possible because if you do not get your change/feature in this release the next release may very well be in 6 – 12 months.
To create the illusion of being able to prevent this and have more control the deployment department is often confronted with huge, 100+ pages, deployment guides decribing the deployment process in numerous, usual manual, steps. Incidents occur in the newly deployed software. Since there were so many changes in the new release, how are we going to find out which part of the newly deployed software is causing the problem.

If it hurts, do it more often.

Recently a phenomenon called continuous delivery arrived which promises to work on those problem areas described in the introduction. A frequently heard statement in the continuous delivery scene is “If it hurts, do it more often”. So, in other words continuous delivery can be achived in the following way:
# Update apache website every 10 minutes */10 * * * * cd /wwwroot/www.myFancyWebSite.com && git pull && service apache restart
Done! That was easy, right?
Well, probably not entirely what you were looking for. The statement refers to getting the amount of changes involved in a release down to a bare minimum and release as automated as possible and as often as possible. In other words: “shipping code as soon as it is ready“. Next question, of course, is: When is software ready to go to production?
Key here is: short feedback loops and only test if necessary.

Short feedback loops
- Provide the developer with feedback on the code he checked-in as soon as possible, for example: run a regular build which can run from anywhere between every x hours all the way up to every commit.
Only test if needed
- If the code that was committed does not comply to standard X (for example standard code checking or the code does not even compile) you do not need to do test Y.
Organize your test in such a way that the cheap (in terms of amount of time spent doing the test or in amount of involved resources) tests come first and are followed by more expensive tests. For example:
- If the code does not compile, there’s no need to run the unit tests
- If the unit tests fail, there’s no need to run the functional tests
- If the functional tests fail, there’s no need to run the integration tests
- If the integration tests fail, there’s no need to run the load tests

This brings us to an important question: When is software ready to go live? Organize your continuous delivery process in such a way that when the software passes all the tests, is ready to go to production.
This results in a lot of go-live moments with advantages such as:

Only a small set of changes time go to production at one time. In case of problems, this narrows down your troubleshooting scope.
Going live many time with a small change set results in a better understanding by your customer of the progress being made.

There are a few companies who have molded their software delivery proces into a continuous delivery, one in particular i would like to point out is flickr. They even have a website showing their recent deployment activities

Tools of the trade

At this moment my preferred toolset to put together a continuous delivery setup are:

git
jenkins
maven
sonar
nexus
Deployit

Cool, i want this too!

For the setup above i have compile the following cookbook which will, after completion, result in a 64 bit CentOS 5.8 vm running the following services:

git
jenkins
maven
sonar
jbossas

The components will be wired accoring to this schema

Prerequisites

Create a vm with the following specs:

2 GB of memory
5 GB harddisk

NOTE: This blogpost assumes the vm has internet access.

Getting the VM up and running

As i prefer to keep things lean and mean, for this blogpost a minimal 64 bit CentOS 5.8 will be installed using the net-installer. The following nine steps provides you with vm ready to be used in this cookbook.

Start your empty vm booting from the attached CentOS net installer iso.
For language and/or keyboard-type: Accept the defaults or change it to whatever suits your environment
The installation-method is, of course, http.
tcp/ip configuration: whatever suits your local network needs for internet access.
Select a mirror service from the CentOS website.
- Provide the web site name: my.fast.mirror.com
- CentOS directory: path/to/5.8/os/x86_64
Click next on the welcome screen, choose to do a fresh install of CentOS.
Partition your disk to suit your needs. This cookbook assumes a totel harddisk size of 5GB divided up as 1GB for swap and 4GB for storage
Network setup: make sure you set a hostname and configure as needed for your vm to fit in your network and to have internet connection.
Finalize the installation by selecting your timezone, entering your root password and unselecting all installation tasks including the default selected “Desktop – Gnome”.

Let the installer do it’s job, once the system is rebooted you have a fresh base 64 bit centos 5.8 vm available.

Post install actions

Update your system: Install pending CentOS updates by issueing: “yum update”
Hosts file configuration: make sure your /etc/hosts file has a separate entry for you hostname and current ip address (don’t forget to remove the current hostname from the entry of the local loopback adapter (localhost)). Basically, your /etc/hosts file should contain the following two lines
- 127.0.0.1 localhost
- [yourLocalIp] [yourHostname]
reboot

Installation procedure

Due to a known compatibility issue with the jpackage-utils package between the default centos repository and the jpackage repository it is important to do the following steps in the order as they are described.
yum install jpackage-utils java-devel
To be able to install the rest of the packages we have to add the following repositories:

jpackage (needed for jboss)
epel (needed for git and git-daemon)
jenkins
sonar

Adding the repositories mentioned above can be done by creating a file named /etc/yum.repos.d/cicd.repo with the following content:
[jpackage-generic] name=JPackage generic mirrorlist=https://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0 gpgcheck=0 [jpackage-generic-updates] name=JPackage generic - updates mirrorlist=https://www.jpackage.org/mirrorlist.php?dist=generic&type=free&release=5.0-updates gpgcheck=0 [epel] name=Extra Packages for Enterprise Linux 5 - $basearch mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=epel-5&arch=$basearch gpgcheck=0 [jenkins] name=Jenkins baseurl=https://pkg.jenkins-ci.org/redhat gpgcheck=0 [sonar] name=Sonar baseurl=https://downloads.sourceforge.net/project/sonar-pkg/rpm gpgcheck=0
After adding new repositories i have a habit of clean ing up the yum repository cache
yum clean all
Next step is to install jboss using the following command (the disablerepo parameter is used to work around the compatibility issue discussed earlier):
yum install --disablerepo=base,updates,extras jbossas
Now the the application server (jboss) is installed we can install the rest of the toolkit using the followng command:
yum install git git-daemon jenkins sonar subversion.x86_64
To make sure jboss binds to the correct interface (to make to accessable from outside the VM) execute the following
echo "JBOSS_IP=`hostname`" >> /etc/sysconfig/jbossas
To prevent port conflicts later on in this cookbook, change the jenkins network port configuration as follows
Change the listenport:
sed -i 's/JENKINS_PORT\=\"8080\"/JENKINS_PORT\=\"8180\"/g' /etc/sysconfig/jenkins
Change the AJP port:
sed -i 's/JENKINS_AJP_PORT\=\"8009\"/JENKINS_AJP_PORT\=\"8109\"/g' /etc/sysconfig/jenkins
To make sure the git-daemon starts using xinetd we have to disable ipv6 for the git-daemon as follows:
sed -i 's/flags/\#flags/g' /etc/xinetd.d/git
To make sure the git-daemon incoming pushes from the client we have to enable receive-pack as follows:
sed -i 's/export-all/export-all --enable=receive-pack/g' /etc/xinetd.d/git
Now make sure the git-daemon is started using xinetd:
chkconfig git on chkconfig jbossas on
Start the git-daemon, jenkins and sonar
service jbossas start service xinetd start service jenkins start service sonar start
Add firewall rules for the running services
iptables -I INPUT -p tcp --dport 8080 -j ACCEPT iptables -I INPUT -p tcp --dport 8180 -j ACCEPT iptables -I INPUT -p tcp --dport 9000 -j ACCEPT iptables -I INPUT -p tcp --dport 9418 -j ACCEPT

Creating git repository

This cookbook assumes a git repository with the name petclinic, to create this git repository on the git server on out vm do the following
cd /var/lib/git/ git init --bare petclinic chown -R nobody:nobody petclinic
After executing the “git init” command the server should have responded with the following message: “Initialized empty Git repository in /var/lib/git/petclinic/”

Developer setup

In our usecase we have a server part and a developer part. We are now ready to configure the developer workspace. This consists of cloning a git repository and fill it with sourcecode and commit this back to the git server. This cookbook assumes the developer logs on to the vm with a separate developer account (in this blogpost username dev is used). If the developer account (dev) does not exists yet, please create it now.
Log on as user dev and create a workdir from where the developer can do his/her work on the git repository, in this blogpost the workdir is called mywork. From the mywork directory clone the empty git repository as follows:
git clone git://localhost/petclinic
You probably receive a warning like “warning: You appear to have cloned an empty repository.”. In his case this warning can be safely ignored. Next, go to the repository directory, make sure there’s only a .git directory present in de petclinic directory. Our demonstration application is the well known petclinic application, let’s get it from the springframework.org subversion repository.
From the petclinic directory (the one with the .git directory in it) execute the following command:
svn export https://src.springframework.org/svn/spring-samples/petclinic/trunk petclinic-web
Now that the petclinic application is in our repository we need to create a root-pom in order to get the build process running. In this root pom we will refer to the pom.xml in the petclininc-web directory and add the jboss plugin and it’s configuration for direct deployment to our application server.
Go to the mywork/petclinic directory and add a root pom.xml with the following content:
<project> <modelVersion>4.0.0</modelVersion> <groupId>org.springframework.samples</groupId> <artifactId>org.springframework.samples.petclinic-root</artifactId> <name>petclinic-root</name> <version>1</version> <packaging>pom</packaging> <modules> <module>petclinic-web</module> </modules> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>jboss-maven-plugin</artifactId> <version>1.5.0</version> <configuration> <hostName>cicd</hostName> <port>8080</port> <fileNames> <fileName>${project.basedir}/petclinic-web/target/petclinic.war</fileName> </fileNames> </configuration> </plugin> </plugins> </build> </project>
Next, add all the code to the repository, commit the code and push it to the git server
git add . git commit -m "Initial commit" git push origin master

Jenkins configuration

First, install the relevant jenkins plugins
java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s https://xebia.com/blog:8180 install-plugin git java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s https://xebia.com/blog:8180 install-plugin sonar
Restart Jenkins to activate the plugins
service jenkins stop service jenkins start
Open the jenkins console, in this cookbook we assumes the jenkins console listens on port 8180
Go to “Manage Jenkins” -> “Configure System”
At the JDK section:

Click on “Add JDK”
Enter a JDK name, for example: jdk16
Uncheck “Install automatically”
As JAVA_HOME enter: /usr/lib/jvm/java-1.6.0-openjdk.x86_64

At the Maven section:

Click the button “Add Maven”
Enter a Maven name, for example: mvn3
Leave “Install automatically” checked
Select the most recent version from the “Install from Apache” dropdown box

At the Sonar section:

Click on “Add Sonar”
Enter a name, for example: sonar

Press “Save” at the bottom of the screen to commit your changes
Create a new (petclinic) build job

Provide a build job name, for example: petclinic
Select “Build a maven 2/3 project”
Press OK
In the “Source Code Management” section, select Git
Enter the Repository URL, for example: git://localhost/petclinic
In the “Build” section, enter “clean install jboss:redeploy” in the “Goal and options” field
In the “Build” section, click on the “Advanced” button. Add “-Dmaven.test.failure.ignore=false” to the “MAVEN_OPTS” field (this will stop the build process if a unit test fails)
In the “Post-Build Actions” section, select “Sonar” from the “Add post-build action” dropdown box
Click on “Save”
Start the build by pressing “Build Now”
In the “Build History” section the newly scheduled build appears, click on it
Click on “Console Output” to monitor the progress of the build process

Configure the git server to start a jenkins job after a change was pushed to the server
echo "curl -s https://cicd:8180/job/petclinic/build" > /var/lib/git/petclinic/hooks/post-receive chmod +x /var/lib/git/petclinic/hooks/post-receive chown nobody:nobody /var/lib/git/petclinic/hooks/post-receive

Update your sourcecode

Now you can see for yourself how this works. A nice test to start with is to update something that can immediately be seen from the webbrowser. For example the title in the titlebar, to update this, update the file petclinic-web/src/main/webapp/WEB-INF/jsp/header.jsp
vi petclinic-web/src/main/webapp/WEB-INF/jsp/header.jsp git add petclinic-web/src/main/webapp/WEB-INF/jsp/header.jsp git commit -m "Updated version number" git push
The “git push” command should trigger the jenkins build process which in turn starts maven which deploy’s (after succesfull building and testing the application to the development environment. Usually this takes around four minutes. Open the jenkins console (this cookbook assumes Jenkins runs on port 8180) to monitor the build proces. To see what exactly happens, just zoom in on a running job and select “Console Output”
Please don’t hesitate to commit a piece of broken (non compiling code) or even break a junit test on purpose, you will notice the entire process comes to a halt facilitating the shot feedback loops. The developer knows minutes after the commit if he has broken something.

But wait, there’s more!

If you have come so far you probably now have a continuous delivery system running on yout VM. The problem with it is that is only able to do the same deployment over and over again. This means the deployable (in this case a .war file) is always the same accross different environments even though environment specific parameters are not. Another issue is that in this configuration you can only deploy to one predefined part of the environment, it does not have a notion of the rest of your environment, for examples dependencies with other systems. To solve this i prefer to use a tool called Deployit by xebialabs. The figure shows where Deployit fit’s in the landscape

Deployit automatically deploys applications to middleware and cloud environments. Fast. Reliable. Cost Effective deployments – even in complex multi tier environments
The Application Release lifecycle is unnecessarily complex, manual and time consuming and its cumbersome nature is putting businesses in a position of weakness. This one very critical process determines how fast a business can deliver products to market and yet is complex, error-prone and full of bottlenecks. An optimized application release process is proven to drive products to market faster, to strengthen the overall product roadmap and to allow the business to be more reactive to market needs, year over year.
Application Release Automation is transforming how these large Enterprises deploy applications on to middleware and cloud environments. The Application Release process determines how fast a business can deliver products to market and yet, is complex, error-prone and full of bottlenecks. An optimized deployment process is proven to drive products to market faster, to strengthen the overall product roadmap and to allow the business to be more reactive to market needs.
Deployit is a self-service application release automation platform that automates the deployment of applications components (webcontent, EAR and WAR files, configuration files, datasources, topics and queues etcetera) onto middleware and cloud environments, including; application servers, web servers, databases, messaging engines and many more. Deployit reduces deployment errors, increases deployment speeds and ensures a cost-effective, traceable and maintainable deployment process for all of your Enterprise deployments.
Deployit is transforming how the Enterprise deploys applications, removing the need for manual configurations, traditional scripting and unnecessary process bottlenecks. A Deployit user can simply upload pre-built packages into Deployit, map the packages to environments and Deployit’s AutoFlow Engine takes care of the rest.
Value of Deployment Automation for Continuous Delivery:

Deployment automation is central to continuous delivery
Plug into build and continuous integration tooling
Support secured delivery pipelines through security
Integrate with release management for governance
Auditing & traceability
What is deployed where = visualisation of your delivery pipeline

Value of Deployment Automation for DevOps:

Add application tier deployment to environment provisioning using e.g. Puppet or Chef
Automated scaling of applications
Automated remediation/downgrade triggered by monitoring
Applies the “de facto DevOps standard” model-based paradigm to applications for extreme scalability

To be able for Deployit to login, enable login (don’t forget to provide a password) of jboss linux user:
usermod -s /bin/sh jboss
Without going into too much detail about Deployit i basically have to perform the following configuration in Deployit to prepare it for it’s task
Create sshhost

Create jbossinstance

Create environment

Next is configuring Jenkins not to deploy to jboss.
Go to the Jenkins console and open the petclinic project and choose “Configure”
In the “Build” section remove “jboss:redeploy” from the “Goal and options” field. It should only read: “clean install”
As a last step we have to update the maven build, logon as the previously created linux dev user and go to the previously created mywork/petclinic directory. Create a new directory (is project) called petclinic-dar, this is the project which will hold the dar. Create a file named pom.xml with the following content:
<project> <modelVersion>4.0.0</modelVersion> <groupId>org.springframework.samples</groupId> <artifactId>petclinic</artifactId> <name>petclinic-dar</name> <version>1.0-SNAPSHOT</version> <packaging>dar</packaging> <build> <plugins> <plugin> <groupId>com.xebialabs.Deployit</groupId> <artifactId>maven-Deployit-plugin</artifactId> <version>3.7.1</version> <extensions>true</extensions> <executions> <execution> <id>Deployit-plugin-test</id> <phase>install</phase> <goals> <goal>deploy</goal> </goals> </execution> </executions> <configuration> <environmentId>Environments/myDevEnv</environmentId> <username>admin</username> <password>admin</password> <deployables> <deployable> <name>petclinic</name> <type>jee.War</type> <groupId>org.springframework.samples</groupId> <artifactId>org.springframework.samples.petclinic</artifactId> </deployable> </deployables> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>org.springframework.samples</groupId> <artifactId>org.springframework.samples.petclinic</artifactId> <version>1.0.0-SNAPSHOT</version> <type>war</type> <scope>provided</scope> </dependency> </dependencies> </project>
Also the root pom of the project needs to be updated, the jboss plugin has got to go and the dar project has to be added. The new root pom.xml looks like this:
<project> <modelVersion>4.0.0</modelVersion> <groupId>org.springframework.samples</groupId> <artifactId>org.springframework.samples.petclinic-root</artifactId> <name>petclinic-root</name> <version>1.0</version> <packaging>pom</packaging> <modules> <module>petclinic-web</module> <module>petclinic-dar</module> </modules> </project>
Now, please feel free to do the same update tests as done proviously and see how it works.
For those reader who do not have access to a Deployit installation, please have a look at the links in this blogpost and hava a look at this screenshot which shows the Deployit console after a successful deployment.