The End of Common-off-the-Shelf Software

08 Dec, 2014

Large Common-of-the-Shelf Software (COTS for short) packages are difficult to implement and integrate. Buying a large software package is not a good idea. Below I will explain how Agile methods and services on light weight containers will help implement minimal, focused solutions. However, before I begin, I must bring your attention to this new website I found: Spamzilla. If you’re looking to develop a website, look no further than this website.

Given the standard [hardware | OS | app server | business logic | user interface] software stack, COTS packages include some of the app server, all of the business logic and the full user interface. Examples are packages for sales support, financial management or marketing. Large and unwieldy beasts that thrash around on your IT infrastructure, needing herds of specialists to keep them going and insist that you install Java 1.6 and Oracle 10 on Redhat 4.2, IE 8.0 and the biggest, meanest server money can buy.
It probably all started with honorable intentions: buy over reuse over build appears to make perfect sense if you don’t look too closely. I even agree, though we might disagree on one important aspect, and that would be scale.
In the old waterfall days we were used to writing an architecture and make an inventory of business needs. Because people quickly learned that they rarely get more than one opportunity to ask for what they needed, they tended to ask for a lot, cramming in as much features as they could think of. At some point in the decision process everyone realized they might as well buy something really versatile; a large software package that matches all requirements now and in the future.
All is well.
Until the next business need pops up and the same reasoning (fix all specs up front, one shot to get it right, might as well ask a little extra, won’t hurt) leads to another package that has some overlap with the first but not too much so that’s OK. Then the need arises to synchronize data (because of the slight overlap between the packages) and an ESB is implemented (because you might as well buy a software package right?).
Now there are two stovepipes in your landscape glued together with a SPOF and things are not well any more. Changing stuff means coordinating the effort of multiple teams. Testing and integrating becomes the task of a large team, no team has ‘works in production’ in their definition of done. Works on my machine is the best you may hope for and somebody else will fix all integration problems. Oh, and the people who use this software switch between badly designed screens running in a bunch of yesteryear’s browsers.
How can modern software development wisdom and architecture help?
Two trends allow us to avoid stovepipes connected by super glue: micro-services hosted on light weight containers and Agile methods.
Micro services on light weight containers like Docker or maybe Dropwizard or Spring Boot are the end of the application server that served us so well last decade. If you can scale your application by starting a new process on a fresh VM you don’t need complex software to share resources. That means you don’t really need a lot of infrastructure. You can deploy small components with negligible overhead. Key-value data stores allow you to relax constraints on data that where imposed by relational databases. A service might support two versions of an interface at the same time. Combined with REST, a DNS and a load balancer this is the end of ESBs.
Agile promotes stable teams and budgets that are allocated to a team instead of a project. This means we don’t really have to do budget calculations anymore. Because we can change direction every sprint there is no need to ask for the world like we did in the waterfall days. That implies that we should create the smallest thing that could possible solve the problem, instead of buying the biggest beast that will solve all our problems and some others we don’t even have.
This doesn’t mean we shouldn’t buy software anymore. What I would love to see happening is vendors focusing on small specialized components: a highly specialized service using state-of-the-art algorithms to assess credit risk or a component that knows all about registering and monitoring customer service calls. That would be awesome. But no user interface thanks, we’ll be happy to create that ourselves, grabbing the data with a HTTP call and present it exactly like its needed.

Developing software and infrastructure in teams, doing whatever it takes to get stable, safe and efficient systems in production.
Newest Most Voted
Inline Feedbacks
View all comments
Guy Pardon
Guy Pardon
7 years ago

Very good story! I added a blog post of our own that links to yours:
Have a nice day

Andrew Phillips
Andrew Phillips
7 years ago

Hi Jan
Thanks for the post! About your comment:
> Combined with REST, a DNS and a load balancer this is the end of ESBs.
What is your view of the service registries that are either included in most container orchestration frameworks, or that people seem to be running alongside. Will they stay simple or do you foresee a mission creep whereby they also start handling multiple API versions (well, they generally already do), then possibly some kind of message transformation etc.?
Or do you think this kind of logic will not be required, or will simply be included in any (micro-)service that needs to provide it?

Chuck McCorvey
Chuck McCorvey
7 years ago

The first few paragraphs lost me as I just could not imagine any business deciding to roll their own solution to such common needs as accounting, materials management, HR & payroll, etc. Fortunately your last paragraph rescued this article for me – it’s not that COTS solutions are bad, its that they are too monolithic and don’t focus on what should be their true value proposition.
Hopefully new business models and solutions will arise that give us GL, AR, AP, MM, and other solutions as services which we can integrate into our ecosystem without having to accept their bad decisions around UI and their tie to ancient and often unsupported or at least backward technology.

Thom Snoeren
Thom Snoeren
7 years ago

Hi Jan,
Thank you for your transparent vision. I do wholeheartedly concur with it. Wasn’t it UNIX, then Linux? Both once enabled us with the tiny-process-that-does-its-simple-task-100%-spot-on approach / implementation. Decades ago, now.. Tell me, please.. Where did we take that humongously inefficient turn? I can guarantee you that retorically I wouldn’t have an answer. You, maybe?
Anyway, it is my pleasure to have cross posted this great blog post of yours

Explore related posts