Integrating systems is a social skill

Summary
Systems integration within limited timeframes is not mainly a technical art. It is a matter of social and organizational skills. Work on the touchy feely side, and tech will follow. “If you don’t manage the above factors well, you will be in for it anyway, no matter how cool, flexible, state of the art and gorgeous your technology.”

Introduction
In almost every software project that is done in a substantial company we need to integrate with other systems. When a team takes on such a software development project, there is often a tendency for developers and project leads to focus on the tasks that need to be done to create the software on their own side, and write scoping statements and non liability remarks to make sure they are not bothered with the integration risks. Especially when The Other System doesn’t have appropriate interfaces available that you can use, and “they” have to build new ones, then the relationship with the other system is kept at arm’s length in contracts.
Understandable, because in our projects we evidently cannot take on the responsibility for work outside our sphere of influence. However. If we want our project to really succeed, we cannot keep it at “they have to do their thing, and they have to deliver on time. If they don’t, then our project may fail but it will not be our fault!” Nope, not good. You will not deliver value to the customer like that, the only thing you do is cover your ass. Comfortable, but not outstanding craftsmanship.
Estimates
Now, suppose you really want to make sure that the system you are writing will be able to interact correctly with all of the necessary parties involved. Many projects fail because the temporal impact of interacting is seriously underestimated. Interacting can have a tremendous effect on the estimated total project time. Consider what has to be arranged:

• You need to communicate with the other party and define an interface between the systems.
• They have to build their end of the interface.
• You will have to stub out their system in order to be able to do your tests for as long as they don’t have anything you can talk to yet
• There will be unexpected changes to the interface. (No, don’t say that you can just finalize the design and be done with it: there WILL ALWAYS be unexpected changes to the interface)
• You will have to share an integration test environment. Stubbing is nice, but only up to a point.
• In the integration environment you will need “conditioned” test data that is meaningful to both systems. When your “other system” has already been around for a while they will probably have a predefined set of test case-data in their database, which of course is not compliant to your own test case-data. So you will have to arrange for new, shared, test data that is meaningful both to you AND the Other System.
• You will need access to their test system. If you are really lucky then the customer will have an integral acceptance test environment in place, with production-alike network infrastructure, many systems joining in, and with conditioned datasets. Not likely. Highly unlikely. So you will need to arrange for access to their test environment, which may be several hops away. Some companies shield their environments through subnetting: arrange access through the routers or try to get a machine of your own inside their test domain: often procedure-heavy activities.
• At integration time they will have interpreted your agreed specs differently (case sensitive, long instead of int, slightly different encryption or compression algorithm, what have you) and you both need to re-implement parts of what you had written.

So, all of this goes for one single system. Think of what happens when you need to integrate three or four, especially when the data goes to and fro between these systems. For a valid end- to-end roundtrip you will have to have all of the above in place for all of the systems in question.

Multiplication, sooner than addition
When you extrapolate this, it is not unrealistic to calculate with an almost exponential factor to the complete duration of your project. And I’m not talking “raw development time” here, no, I mean the time needed to get from project start to actual production. I have often seen estimates like “xxx weeks, plus an extra iteration for each system”. That will only be true if these systems are unrelated and if all conditioned data and integration environments are in place, if you can reach all systems from where you are developing AND everyone is at peace with the world. Again: unlikely.
Make sure you start requesting DTAP (Dev,Test, Accept, Production) environments immediately at the start of the project, the same goes for (network) access, etc. etc.
So the advice here is: TAKE HEED! Multiply, rather than add! They may not be as flexible as you are!
Agile approach
Now, if we were living in an agile world, then the odds would be a bit more optimistic. You will insist on early integration, they will not object to that, better yet, they will be enthusiastic to do so. You will gradually work together toward a cool interface, which will keep changing as needed to get to the most efficient, open solution. Test data will be created in unison, you will learn from each other, if one party has a problem and starts lagging a bit you offer to lend him one of your experts in exactly their problem field, who happens to be on the project anyway. Sigh. Love and understanding. Success. Profit. Happy Customers. Bliss.
Why not
Hm. If the company where you’re doing your project is large enough, this Arcadia is a rare find. The atmosphere among developers in traditional companies can sometimes be one of fear and loathing

• Fear of not being as good a developer as your competitor
• Fear of being punished for making mistakes
• Loathing of not being taken seriously as a craftsman
• Loathing of not being allowed to shine and make a difference

This is obviously not a great basis for collaboration. But as it is, sadly, not uncommon, you had better find a way to deal with it. As difficult as it is to change structurally, you can still do something to improve your chances of a successful integration.
Social calls
Make sure you have at least met once, face to face, with the Other System’s developers, project leads and with whomever else comes into play. If you are on opposite sides of the ocean, use video conferencing with even the crappiest of webcams if needs be. (No excuses like “we don’t have a video conferencing room”). Use direct messaging or irc to communicate. Of course, if the company doesn’t allow a streaming protocol through the firewall there’s not a lot you can do. But in that case send your photograph with your email-messages. Believe me, even knowing what the other party looks like can make a tremendous difference.

We’re in this together
Try to create an atmosphere of mutual interest. There will probably be a sense of togetherness around a system, a “three musketeers” feeling in the developer group. That’s great, because it improves team gelling and with that fun and productivity. You, as someone of the outside world, want something from them, you want to invade Their System and you will have an impact on their original plans. Evil Outsider! So what you want to do is overcome that and try to become accomplices, get them to realize that we work for a business process that needs you to integrate. “Yo, WE are going to make this business process work, guys, great isn’t it?” Even in traditional development environments you can gain a huge headstart by being sensitive to this sort of relationship management.
When there are more systems than one that you need to integrate with, try to initiate this same feeling of togetherness between all systems. Gather the teams, or at least the team members you will be working with, and have an integral meet and greet session somewhere. If politics prevail, hold the session on “neutral ground”, say in a café. Don’t hesitate for a second to pay for all drinks and bar food, the cost of that is absolutely neglectable relative to the enormous profit you will have in terms of risk mitigation.
Infrastructure
A special aside about infrastructure. There can be a huge difference in how slick the infra groups are organized. In some organizations a simple application form for an infra resource will be sufficient to get a nicely humming [machine/router/databaseaccess/…] way before you desperately need it. Whereas in other circumstances you will have to fight fierce battles for every inch of UTP. If the latter is the case, approach the infra groups like you would approach The Other System: try to make them your friends and allies.
Don’t squeeze it
It’s no big deal if you don’t agree on the definitive interface. Give them the feeling they are in the lead. Even if you might think the developers in The Other Systems are not the greatest, they will know a lot more about their system than you do. You don’t need to squeeze all possible brilliance out of the design. If it works well then it’s ok, as long as you can manage to give them a feeling of ownership and authority. Remember the fear and loathing: if you can give them the feeling they are in the lead and are taken seriously, they will be grateful for the opportunity.
Technology
“Hey, when will you start to talk about the Technology? Soap, Web services, Corba, RMI, OSF/DCE, vanilla XML over http, what should we use, synchronous, asynchronous, what?” you will ask. My answer is: All Of That Is Trivial. If you don’t manage the above factors well, you will be in for it anyway, no matter how cool, flexible, state of the art and gorgeous your technology.