How do you setup a environment that support the continuous deliver of enterprise Java applications? How do you manage the large number of machines that are involved? How do you enable self-service, continuous delivery of applications onto the platform?
In this blog post we will give a description of an open source Java Application Platform as a Service that we created for our customer, using VMware, Redhat Enterprise Linux, Apache WebServer, JBoss Enterprise Application Platform, JBoss Operations Network, Puppet, Deployit,  F5 Load Balancer and  a Layer7 SecureSpan gateway.

Data Center Quality Platform

The customer wanted a data center quality Java Application Platform with the following features:

• Standard configuration
• Standardized provisioning
• Standardized deployment
• Centralized monitoring
• Centralized access control
• Virtual environment
• Proving technology

Current situation

As the current Java application platform was based on HP-UX on Itanium, the customer was facing high cost for hardware, software licenses and fading support from software vendors.  As all applications ran on a HP Superdome, it was very difficult to add resources to individual applications. In addition, development teams spend too much time taking their software through the development, test and acceptance environments, resulting in slow delivery of software into production. Finally, it was difficult to provide 24×7 availability because all applications are running on a single machine.

Java Application Platform

The following figure illustrate the solution architecture of the java application platform.

In the following paragraphs we will describe the purpose of the most important components.

Dual Data Center – HP

Not shown in the figure, is the hardware setup of the platform. It consists of HP blades setup in two data centers on two different locations. This provides the basic infrastructure for 24×7 availability and fault tolerance.

VMware ESX

VMware ESX is deployed on top of the hardware in the dual data center. This provides us with the ability to create virtual machines and provide high availability in case of single server of single site failures. It also allows us the quickly scale up virtual machines and increase the resources assigned to individual virtual machines.
For all machines in the platform we use a single VMware template image. This image is installed with RedHat Enterprise Linux and a puppet client.

Puppet

Puppet fully automates system management. It is used for the installation of software packages, conformity tests and day to day system administration tasks.  For every type of node, we have a puppet plan. When the machines boots, the puppet agent provisions the machine with all the necessary software and configuration according to the plan for that machine.
The use of Puppet completely automates and standardizes the configuration, ensures 100% reproducibility of the configuration and is fast. Provisioning of a new machine from the template to full operational mode is done in a matter of minutes.

JBoss EAP

JBoss Enterprise Application Platform is the Enterprise Java applications server for all java applications.  The installation and configuration is done by Puppet and uses the official RedHat RPMs.  Puppet configures JBoss to ensure that :

• JBoss management applications authenticate users against Active Directory, providing a single point of authorization for operations.
• A JBoss Oracle database schema is automatically provisioned for that specific instance of JBoss, providing persistence for the JBoss server system state.
• All Business Applications can authenticate users using SAML against the Layer7 Identity provider, providing a single point of authentication and authorization for their customers.
• The JBoss instance is added to the pool in the F5 Load balancer
• The application server is added to the Deployit infrastructure inventory, providing the tenants of the platform with the ability to deploy applications to the server.

JBoss application servers are always deployed in multiples of two, where each server of a pair is assigned to a physically different data center location by VMware.
The use of puppet provides us with a fast and reproducible way of provisioning JBoss application servers, allowing for a fast and reliable scale out mechanism for the applications.

JBoss Operations Network

JBoss Operations Network (JON) is used for monitoring all the resources in the platform.   By default, Puppet installs a JON agent on every machine. This agent scans the inventory of the machine and reports it to the JON server.
JON has a very good support for high availability and fail over. By simply adding a JON server machine, agents will automatically distribute themselves across the servers and failover if necessary. Each JON server also runs a JON agent, making sure that unavailability of a JON server is also covered.
In JON we created a number of alert templates for different resource types (os, apache, jboss, jon, puppet, etc.)  that will monitor and report critical conditions on the system.  All error messages from the JBoss servers logs are reported as incidents.
All alerts and clearing conditions from JBoss Operations network are reported via SNMP to TNG Unicenter.
Through the  use of JBoss Operations Network all machines, servers and resources in the platform are automatically added to the centralize monitoring system.

Deployit

Deployit is used for the automated deployment of applications onto the platform. It automatically deploys all the application components in a stack to the appropriate containers.  Deployit :

• deploys static content and proxy configuration to the apache webservers,
• deploys enterprise java application components to all individual JBoss servers in the farm,
• executes SQL scripts to the database,
• configures the F5 loadbalancers to add or remove servers or applications to the pool,
• applies environment specific changes to the application configuration.

The deployment plan for a specific application is prepared in close cooperation between the application developer and platform management staff. When the deployment plan is finished, developers can deploy new versions of the application themselves, directly from a build tool or manually.This ensures solving any installation or configuration problem isn’t postponed until the application is installed for production use, but rather is solved at the early stage of any development.
The same deployment plan is used for all environments. Authorization can be configured per enviroment and per application. LDAP is used to authorize software developers to deploy and configure an application for development and testing purposes, while integration specialist can deploy the application in production.
The use of Deployit provides the platform with a fully automated and standardized deployment mechanism, improving the speed of deployment of applications through the development, test and acceptance environments while reducing the number of staff involved and lowering the number of configuration errors.

F5 Load Balancer

The F5 Load balancers is used to support scalability and fail over for the JBoss Application Server farm.
The pools are configured to use a sticky session protocol based upon the JSESSIONID session cookie. If the cookie is not present, round-robin load balancing of the HTTP requests is performed.
Puppet adds the JBoss servers to the  pool in the F5 Load Balancer.
When a server is scheduled for a restart, the server is taken out of the pool. This ensures that this server does not get any new request, but will still be servicing existing sessions. When the session count in JBoss drops to zero, the server is restarted and restored to the pool.
The use of the F5 Load Balancer provides us with the ability to increase and decrease the number of servers in the farm, provide load balancing, fail over and graceful decommissioning of servers in the farm.

Layer7 XML Gateway / Identity Provider

The Layer7 SecureSpan Gateway is used as centralized security policy enforcement point and SAML identity provider.
Layer7 supports multiple authentication methods, Kerberos, digital certificates, username+password and is able to use multiple identity stores.
Puppet configures all JBoss application servers with SAML support and configures Layer7 as the identity provider: JBoss receives authentication (identity) and authorization (roles) information as a SAML-token. The information contained in the token is translated to a standard JEE-principal user (using a tiny layer of custom code), so all JEE applications can access the authentication and authorization information in a standard way.  Whether the JEE application is a web application or provides webservices, from a security there’s no distinction. All application designers have to do is declare the application security roles conform the JEE standard.
The use of Layer7 standardizes the authentication and authorization for all business applications and centralizes access control.

Conclusion

The customer wanted a modern data center quality Java Application Platform to ensure that java applications could be deployed with lower cost and with high availability and easy scalability.
VMware, the dual data center, Layer7, the F5 Load balancer and JBoss provide the infrastructure for a  high availability and scalability for any java application. The combination of VMware, Puppet and Deployit are the fabric to enable continuous delivery of enterprise java applications.
Through virtualization and automated provisioning and deployment it has become possible to add a completely new, correctly configured machine to a cluster in a matter of minutes, completely secure and under full monitoring.