Agile Integration Architecture

October 19, 2018

The centralized deployment of integration hub or enterprise services bus (ESB) patterns where all integrations were deployed to a single heavily nurtured (HA) pair of integration servers used to introduce a bottleneck for projects. Any deployment to the shared servers had the risk of destabilizing existing critical interfaces. No individual project could choose to upgrade the version of the integration middleware to gain access to new features.

A significant challenge faced by service-oriented architecture was the way that it tended to force the creation of central integration teams, and infrastructure to create the service layer.

This created ongoing friction in the pace at which projects could run since they always had the central integration team as a dependency. The central team knew their integration technology well, but often didn’t understand the applications they were integrating, so translating requirements could be slow and error-prone.

Many organizations would have preferred the application teams own the creation of their own services, but the technology and infrastructure of the time didn’t enable that.

Agile Integration Architecture

“A container-based, decentralized and microservices-aligned architecture for integration solutions”.

Three Aspects of Agile Integration Architecture

1: Fine-grained integration deployment

What might we gain by breaking out the integration in the siloed ESB into separate runtimes?

2: Decentralized integration ownership

How should we adjust the organizational structure to better leverage a more fine-grained approach?

3: Cloud-native integration infrastructure

What further benefits could we gain by a fully cloud-native approach to integration?

We could split-up the enterprise-wide ESB component into smaller more manageable and dedicated pieces.

These “fine-grained integration deployment” patterns offer specialized, right-sized containers, providing better agility, scalability and resilience, and look quite different from the centralized ESB patterns of the past. Figure below demonstrates in simple terms how a centralized ESB differs from fine-grained integration deployment.

Fast lightweight runtime:: They run in containers such as Docker and are sufficiently lightweight that they can be started and halted in seconds and can be easily administered by orchestration frameworks such as Kubernetes.
Dependency free:: They no longer require databases or message queues, although obviously, they are very adept at connecting to them if they need to.
File system based installation:: They can be installed simply by laying their binaries out on a file system and starting them up-ideal for the layered file systems of Docker images.
API-first:: The primary communication protocol should be RESTful APIs. Exposing integrations as RESTful APIs should be trivial and based upon common conventions such as the Open API specification. Calling downstream RESTful APIs should be equally trivial, including discovery via definition files.
Digital connectivity:: In addition to the rich enterprise connectivity that has always been provided by integration runtimes, they must also connect to modern resources.