Deployment is the new build (part 2)

Earlier this year, I was invited to present a talk at Devopsdays Boston about deployment as the new build: how deployments are carried out now, how they will need to adapt in a more virtualized, on-demand application landscape and what fundamental improvements will need to come before deployment matures into the invisible, it just works™ experience build is today.
In the previous post, we compared deployment to another key process in the application lifecycle – build – and asked which key developments turned it from a magical “black box” into the “just works” process it is today.
We identified Reusable commands, Models and Conventions++ as three key steps, which we’ll look into in more detail in this post. Then, we’ll shift back to deployment and ask which improvements will be essential to getting to “just works” here.

Reusable commands

The first step, epitomised by Apache Ant, was the recognition that most of the low-level tasks or actions are the same whatever application is being built, whether calling a compiler, copying files or replacing placeholders.
Rather than copy-pasting the same OS commands into every new build script, we encapsulated these commands as libraries of reusable components that only needed to be written once.
Further, we discovered that certain patterns of step sequences would appear in many different builds. These ‘chunks’, such as constructing a classpath, or copying and processing static resources, evidently represented some higher-level build activity with a distinct function.

Models

Whilst the realisation that all the actions carried out in a build are basically a sequence of common chunks was an important start, the next big advance was brought about by recognising that we weren’t just seeing repeated patterns of actions, but that the types of data these actions were working on were also shared.
This gave rise to the notion of a true domain model for application builds, with source sets, resource sets, modules, dependencies and so forth that were originally introduced by Maven and have featured and been reused in all build systems since.
Combining the sequence of common chunks with the new domain model that structured the data being processed gave rise to notion of distinct phases, in which parts of the build model are generated, prepared and made available to subsequent commands.
In addition, Maven also supported the idea that the domain model, and thus the build phases, would have to be able to vary slightly to accommodate different types of Java artifacts that need to be delivered, such as JARs and EARs. This has subsequently been further developed to support builds of totally different technologies such as Adobe Flex applications.

Conventions++

An additional benefit of domain models for build was the ability to make use of default values in a structured way, for instance for the names of built artifacts or the location of resource files.
However, the flip side of this convenience, certainly in combination with XML as a descriptor language for builds, meant that deviating from these standards could be quite a challenge. Certainly if the aim was to extend the domain model in some way, or to support a language or technology whose build flow was reasonably different from Java’s, such as building documentation bundles or virtual machine images, for instance.
This has led to a generation of build tools, such as Gradle, that aim to restore the developer to a position of full control in which arbitrary actions can easily be defined and organised into phases, tasks and entire builds. Of course, given how used we have become to the convenience of “it just works” in simple cases, these tools still support the full domain models and conventions of common technologies such as Java.

Who’d have thought?

Reviewing this progression from today’s perspective, a couple of facts stand out that, certainly in comparison to other evolutions in IT, are quite surprising.
Firstly, whilst it’s now hard to imagine specifying a dependency using anything other than the groupId:artifactId:version pattern, none of the models or conventions that developed were formalised in industry standards. Instead, they were either based on observations of common patterns, or simply clever or even somewhat arbitrary choices (src/main/java, for instance).
Secondly, we have seen how ease-of-use based on conventions, coupled with a moderate nuisance factor of tweaking those conventions, can dramatically change user behaviour. Initially, for instance, many of those new to Maven spent quite a significant amount of time, and produced a fair amount of XML, to change the standard settings to match their own environment, naming conventions, file paths etc.
Pretty soon, though, and especially as the ratio of green field vs. legacy projects increased, it simply became easier to stick with Maven’s standard values and be done with it. Today, these conventions have become so standardised that they are supported not just by Maven, but essentially all other build systems out there, too.
It’s not just users’ preferences that were “charmed” into adopting standard conventions, though. In many cases, company standards previously seen as cast-iron were progressively discarded or modified if they could easily not be accommodated by the emerging de facto standards. Ease-of-use was able to triumph against abstract rules.

And deployment?

So much for the build process. What about deployment, today’s critical hurdle for automation in the business value delivery chain? As previously mentioned, the current industry average is somewhere between “black box” and “step sequence”. In terms of the descriptions of the build process evolution, the most advanced deployment automation systems are somewhere just beyond “reusable commands”, in other words.
Which naturally begs the question: how do we get to a push-button state? What do we need to do to be able to reach the maturity level of build today?
Looking at what we encounter in the industry today, three critical aspects will be:

Develop a common model
(Re)discover vanilla
Support a “clean build”

We’ll cover the details of these three concepts in the next blog…