New Design for Integration

One of the hottest technologies that is critical to every organization is the use of an Integration Platform. The traditional integrations that have been built up within an enterprise are all point to point. When you draw a picture of the simplest set of systems, it often looks like a spider web. A more complex system like ERP will be unreadable if every integration is shown. A properly designed integration platform will simplify the complexity and give the business flexibility in their design.

Integration platforms should be jointly sponsored both by the business and technology leaders. It offers real value to both organizations. Integration platforms provide the following business drivers for both the business and IT.

Business Leaders can expect to see the following benefits in their business from a well architected integration platform

• Minimize duplicate data entry into systems

• Reduce users from swiveling between applications

• Rapid onboarding of partners

IT departments can see the following from the platform

• Ability to discover existing integrations via a catalog

• Simplified management of the platform

• Consistent and well defined deployments

• Solution monitoring for availability and performance

To understand why integration platforms can achieve all these benefits, it is important to understand how the problem is traditionally solved. Integrations have historically been done with point-to-point integrations. The concept is simple. For every transaction between two systems, an integration is built. For a simple example, a customer service site accepting a payment may need to write 3 different systems. When there is only a single source sending application, the picture is pretty simple.

When you add additional sources of the same transaction, it gets really messy. Now imagine supporting a change of address transaction or a refund. The whole solution gets even messier. If you switch from a point-to-point integration to an event driven integration, the solution simplifies. There is a single integration from the application to the integration platform for each transaction. The diagram stays simple.

The integration platform manages the logic of what needs to be updated. It is taken out of each application and centralized so that everyone can use it. This creates the following simplification:

• Adding a new system that initiates the transaction only requires integration to the integration platform. In the example above, the number of new integrations is reduced from 3 to 1.

• Adding a new system that gets updated only requires a single new integration and none of initiating systems have to be touched.

But there are other benefits that both IT and the business will see. A mature integration platform will allow:

• Integration platforms will provide a catalog of services that encourage developers to reuse existing services rather than creating their own.

• Logic around how systems interact is defined in a single spot rather than being spread across multiple applications and processes.

• Operational management of integrations is in a single console. The operations team have a consistent way to see problems and fix them.

The great news is that a point-to-point integrations patterns can easily be transformed to an event driven pattern. If you have a bunch of point-to-point integrations, you have the start on the toughest part of building event driven integration platform. Building out integrations for older systems is a real pain because the system either doesn’t have an API or the system has been modified in such a way that the APIs are broken.

After years of building integration platforms, I have developed a 5-step pattern for event driven architecture that allows for old point to point solutions to be incorporated. Actually, it works equally as well for new integrations as well.

Authenticate, Authorize, and Accept

Every transaction must be accepted into the system. However, a transaction must be authenticated (is their identity valid) and authorized (are they allowed to execute this transaction) before being accepted. Accepting the transaction consists of validating the input, acknowledging the receipt, and logging the transaction for tracking and security.

Transform to Canonical

A critical step in the process to transform incoming requests to a common standard. By transforming to a common format, you have the ability to accept incoming transactions from multiple senders who may have different data formats. This allows you to build a single process to process all requests regardless of the sender. It also allows you to grow the transaction as business evolves and new fields are added.

Augment with Data

The next part of processing is augmenting the transaction with additional data. There may be data needed from other systems or databases that contain information needed for the destination. Some transactions may have lots of data. Others will not have any need for this.

Transaction Delivery

This part of the transaction is the actual execution of the transaction. It is invoking each system that needs to be updated and ensuring all updates are completed successfully. While invoking each system, it is likely that you will need to transform the canonical to a format for suitable for the system receiving the update. The common standard from step 2 is essential at this point since it is likely that you may either have multiple systems that may need updating with different formats or an update must be made to one of multiple systems based upon where the record resides. It also allows these end systems to evolve without break the other stages of the transaction.

Completion

The last step of processing the transaction is completing the transaction. It involves sending a status back to the sender of the success or failure of the transaction. It could be a reply to the original message for synchronous transactions or delivered asynchronously based on the service agreement. It can also involve writing logs and transaction data containing status of the completed transaction.

The benefits of an integration platform combined with an event-driven architecture can give your business a competitive advantage. Integration technology has been around forever. Each evolution of the technology has solved different problems. The integration platforms of today have combined previous generations and added cloud technologies to give IT agility and flexibility to meet the demands of the business.