Integrating deployment automation and configuration management

My colleague Robert van Loghem and I have been blogging about Java EE deployment the last few weeks. And that is not without a reason; we have built Deployit, a product to automate Java EE deployments. We’ve already mentioned it before in some of our podcasts and blogs, so now might be a good time to explain some of the concepts behind our product.

The problem

The problem our product is addressing is that Java EE deployments are tedious and unpredictable affairs that can also be very dependent on specific knowledge. The guys at ZeroTurnaround conducted a survey on Java EE redeployment and restart times. The results are interesting, especially when you consider that these times are for a redeployment in a development environment. Consider the challenges of doing this in a production environment with an application server cluster, multiple web servers, and databases and so forth that all need to be configured and restarted using a full redeployment scenario. I believe that the only way to do deploy applicaitons in a reliable and efficient manner is to automate this task. Something that IT analysts such as Gartner and Forrester also see as an emerging and hot topic.

In our experiences dealing with this issue, we’ve found that middleware environments can be so complex that nobody has the complete picture. It’s like the story of the blind men and the elephant where a number of blind men touch and elephant to find out what an elephant is like. One touches the trunk, the other the tail and another one a tusk. Each one of them knows a small part of the elephant but none of them has the complex picture of the elephant. For middleware environments this means that the OS guys can tell you about all the boxen in the netwerk, the Java EE guys about the application servers and the applications running on them, the DBAs about all the databases and the schemas that are around, while the network guys know what is connected to what. Without proper configuration management nobody can get the complete picture without consulting with all of these groups. Because knowing what is already there and then figuring out what to change to get an application into production is such a big part of deployment, I believe that you cannot effectively automate deployments without having proper configuration management in place. That is why our deployment automation product Deployit combines these two aspects.
So how does Deployit do that? Let me explain by going over each of the three components involved:

the configuration management database,
the change resolution engine and
the execution engine.

The configuration management database

The first component of Deployit you will meet when you start up its Flex user interface is its configuration management database (CMDB). The CMDB holds all the information about the middleware environment. In the picture to the right (you can click it to enlarge it) you can see a Deployit CMDB for a very simple demo environment. Each line represents a configuration item (CI) and the CMDB can be summarized as follows:

The two hosts, “Apache 2.2 host (VMWare)” and “WAS 6.1 host (VMWare)”, are the hosts running the middleware. As the name implies these hosts are running on VMWare.
The CI “Apache 2.2 test server” represents an Apache HTTP Server instance running on “Apache 2.2 host (VMware)”
The CI’s whose name begins with “WAS 6.1” represent the IBM WebSphere Application Server 6.1 installation on “WAS 6.1 host (VMware)”
The CI’s whose name begins with “PetClinic” represent v1.0 of the PetClinic demo application; an EAR file (PetClinic-1.0.ear) and some static content (petclinic.zip).
The CI called “Test environment” is a group of all the CI’s that make up the test environment. In this case that would be the “Apache 2.2 test server” and the “WAS 6.1 test cluster” CI’s.
Finally the CI “Deployment of PetClinic on the test environment” represent the fact that the PetClinic application has been deployed on our test environment. In Deployit a deployment is not only an action, it is a state of the environment. Just like a wedding is the happy day and married is the state you are in from that moment on.

The change resolution engine

The CMDB in Deployit gives you an overview of the actual state of your environment. But how does that help you deploy anything? Well, for that you switch to the Design tab. As an example let’s pick the scenario where we want to upgrade the deployed application. As mentioned before, this is actually a scenario that is way more common than the initial deployment.
So we switch to the Design tab and import v2.0 of our application into the CMDB. And then we modify the deployment CI to reflect that we want v2.0 to be deployed in our test environment. This declaratively tells Deployit that we want to perform an upgrade and Deployit’s change resolution engine will define the steps that need to be executed. The nice thing about this is that anybody can perform a deployment because it is as simple as modifying the deployment in the user interface. The same thing goes for middleware configuration actions such as adding a cluster member, modifying a datasource, etc.
To find out what steps the change resolution engine has determined need to be taken, we switch to the Changeplan tab in the user interface, where we will see a list of changes (to summarize the changes made) and the list of steps to be taken:

First v1.0 of the PetClinic application will be stopped and undeployed.
Then v2.0 of the application will be deployed and started.
But that is not all! The new version of our PetClinic application should be made available through the WebSphere plugin in Apache so Deployit generates the WebSphere plugin configuration and copies the generated file over to Apache.
The old static content will be deleted from the Apache host and the new static content will be copied to it.
Finally Deployit will restart Apache so that it picks up the updated WebSphere plugin information.

The execution engine

When we have made all the changes we want to make and when we have reviewed the steps to be to our liking we press the “Deploy” button in the top button bar to execute the steps. Deployit’s execution engine will log in to the target machines and execute the steps. Once all the steps have been executed, Deployit will update the CMDB to reflect the changes made: v2.0 of the PetClinic application will now be visible in the actual tab and the “Deployment of PetClinic on the test environment” CI will show that PetClinic v2.0 has been deployed on the test environment.
Deployit’s execution engine does not require any agents to be installed on the target system. It uses SSH to log in to the target machines and there it will use the native interfaces of the target middleware (e.g. wsadmin or WLST) to perform the necessary work.

Advantages

There are quite a number of advantages to integrating configuration management with deployment automation:

There is no more need to update text files, Excel sheets or Wikis with the current state of your environment.
The actual environment and the configuration mangement database will stay in sync when all changes are automated. And that is what we want, right?
The steps needed to make a change happen are determined from the information in the CMDB and the changes made to it. The system can automatically find out whether to do an initial deployment or an upgrade. Or whether to generate WebSphere plugin configuration or not.
There is no more need for deployment manuals, not even those describing which scripts to run!
Anybody can perform a deployment; there is no more dependency on specific resources.

This is a very brief summary of how Deployit works. In later blogs I will explain the concepts behind our product.