Developing a SOA-based integration layer framework: challenges

A few years ago I was asked by one of our customers to help them make better use of their integration layer. Ever since then me and my team have been working on a framework in support of that. This is the third in a series of blogs on the development of our framework, and discusses the challenges we had to meet.
In the previous blog of this series I mentioned the goals we had to reach. Succeeding in doing so of course meant we had to overcome a lot of challenges. In order to keep this blog from reaching the size of one of the books of the ‘Lords of the Rings’ trilogy, I’ll keep it limited to the five below, which together form a pretty good picture of what we had to deal with.

Challenges

1. Resistance in the organization
There is something an integration layer has in common with SOA, and one of the reasons why both of them fail so often: as soon as you’re getting the word out of what you want to do, people start to dig their heels in, based on what they think you’re doing (as opposed to what you actually are doing). Also, all of a sudden they need to follow certain rules to get their data from other systems, which to them is just static. Plus in their mind performance is always an issue (even if it isn’t, not to mention the fact that we’ve never had any problems in that area). The worst about this is having to always defend yourself. When two systems communicate with each other using the integration layer, and something goes wrong, it’s the fault of the integration layer by default (shoot the messenger!). Guilty until proven innocent, indeed.
The answer is of course to keep people informed at all times, and working very closely together with the project teams (even becoming part of such teams for a longer time when necessary, as per the Scrum methodology). It does help if there are architectural policies in place, like the one we currently have that prescribes the use of the integration layer when invoking services in other domains.
2.Keeping generic features generic
This is one that was especially present when developing the bulk of the framework. We were part of a big project trying to tie two systems together that were not really built for that kind of thing. As the deadline was getting closer, the pressure to make a few shortcuts in order to get specific requirements implemented was rising. Fortunately we had a project manager who was capable of keeping his eye on the big picture. Still, I do recall meetings with lots and lots of screaming.
3. Dealing with a lack of SOA knowledge
If there’s one thing I found out the past few years, it’s that lots of people have no clue what SOA is about, but still pretend they do (or even worse: are convinced they do). That made it particular hard to sell some of the more SOA-influenced decisions (like the usage of certain WS-* standards, and yes, they do work in our case, thankyouverymuch). Worst (or best, depending on how you look at it) example was when a software developer told me that SOAP and queues don’t mix, making a part of our framework defunct in his eyes. The only thing to do in such a case is of course educating people through presentations, workshops, written documentation and discussions.
4. Dealing with a lack of platform knowledge
In the meantime, not all was well on the implementation side. We had this appliance that we had to use, and that had lots of options we actually could benefit from, but not too many experts available. That meant that we really had to get our hands dirty, and also that we had to make a few rookie mistakes before we got things running. By then we had already managed to alienate some people, as in their eyes we could not deliver. Remember what I wrote about concept and implementation earlier on? This is where it hurt.
5. Working without a Canonical Data Model
This may seem like a strange one but think about it: all these different applications in these different domains, trying to communicate with each other, but speaking their own tongue. The integration layer in the middle, transforming messages to and fro. Contracts for the consumer and from the provider being just different enough to get you into a versioning hell if you’re not careful. Suddenly this ‘intrinsic interoperability’ goal of Service-oriented Computing all the cool kids are talking about starts to make a lot of sense.

Conclusion

As mentioned above, this is just the tip of the iceberg. In the end it’s all about ‘can you deliver?’ which fortunately we could. And up until that time, having an elephant skin did help.
That’s it for this time, next one will be about the building blocks of the framework.