Developing a SOA-based integration layer framework: introduction

A few years ago I was asked by one of our customers to help them make better use of their integration layer. At that time it consisted of 2 or 3 integration platforms, some home-made software running on top of those, and a few minor rules about how to use it all. None of the benefits of a ‘real’ ESB were reached, even though that was always the idea.
Ever since then me and my team have been working on a framework that could fulfill the ‘business needs’:

• Efficiency in business processes, such as automatically coupling sub processes into a whole.
• Consistency in data representation, such as the representation of a customer in a CRM system and in a financial system.
• Flexibility and time-to-market accelerated by the IT department.

It was decided that this framework should be build on top of the already existing platform layer:

Normal
0

false
false
false

EN-US
X-NONE
X-NONE

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}

Figure 1: Integration Layer layers.

Normal
0

false
false
false

EN-US
X-NONE
X-NONE

/* Style Definitions */
table.MsoNormalTable
{mso-style-name:”Table Normal”;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:””;
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:”Calibri”,”sans-serif”;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;}

The framework was designed in such a way that it could function as a lightweight ESB, although we tried avoiding the dreaded ‘E-word’ for as long as possible, as people tend to get a bit nervous when you use it.

Below are several subjects we needed to address while developing the framework. In future blogs I’ll elaborate on these; whenever such a more detailed blog is posted I’ll add a link here.

Goals & challenges

Partly based on the aforementioned business needs, several goals in areas like business centricity, federation and the appliance of SOA design principles were set by the team.
Of course no task is complete without a set of accompanying challenges. We’ve encountered our share of those as well, such as lack of knowledge on key topics, and resistance from within the organization. In a future blog I’ll dive deeper into those, and the ways we managed to handle them.

Building blocks

Like any decent framework ours consists of components & patterns, both conceptual as well as physical. For me as the architect the conceptual stuff was the most important; as I could not always find what I wanted ‘in the real world’ I had to come up with some of these building blocks myself.
Basis for both the other building blocks, as well as some of the products we deliver to the projects, are our so called framework services.  Most of these are designed in such a way that they can be reused time and time again. A few are not, merely functioning as proxies for the application services they call.
One of the most important aspects of the framework is that it provides generic features. Systems using the framework to transport messages to counter parties benefit from out of the box features in the areas of routing, robustness, security, persistence and transformation.
Another big part of our framework is the connection patterns, each providing a unique combination of a message exchange pattern and a feature-set.

Integration products

Having a nicely composed framework is fine and dandy, but in the end it’s all about working software (and the documentation that goes with it), or as we like to call it: our ‘integration products’.
Whenever we get a new assignment for an application to be exposed through our framework, we start with use case descriptions. After all, no connection exists without a business reason. For that we had to come up with our own type of architectural view model.
Probably most important are the integration layer connection points. The descriptions of these make clear how messages should look like in order to be transported, and what features a consumer can expect. Once they’re implemented and deployed in a specific environment, the application can be reached through the integration layer framework.

Conclusion

The framework has been running for quite some time in production now, with no real problems whatsoever. More and more applications are reachable through the federated endpoint layer we provide, meaning that using the framework has become the default way of connecting systems from different business domains, as well as exposing our product services to the outside world.
The goals have been met for the biggest part, although there is always room for improvement. Which is a good thing, because we’ve got a lot more up our sleeve, especially in the feature department. And implementation-wise we’ve learned a lot, so some of the older software solutions are eligible for refactoring.
Most important right now though is getting the framework better manageable: a lot of the configuration is still done by hand, and it being the center of the enterprise when it comes to the transportation of messages, there’s a need to get a better grasp on what’s going on.
That’s it for this intro; in the next blog I’ll go into more detail about the aforementioned goals.