Integration as a Microservice

Blog Series | Agile Integration | #3

With a Distributed Integration Platform & Integration Frameworks to more Agility

Author: Oliver Faust

Integrations are classically implemented through centralized, monolithic Enterprise Service Bus (ESB), which can often become a bottleneck (see Rethink your integration strategy)). Today, more agility is needed. Agile integration solutions offer a remedy (see Forms of agile integration - an outline). These used cloud paradigms to have integrations built and operated decentrally by DevOps teams. In this article, we will look at how such a decentralized integration can be implemented with cloud-native microservices.

Distributed Integration Platform as a technical foundation

For distributed development to work optimally in an enterprise environment, it needs a provided integration platform. In this blog, we analyze the implementation using a Distributed Integration Platform (DIP). This DIP provides the foundation and working tools so that integrations can be implemented quickly. Such a platform must provide technical as well as organizational services:

  • Microservice templates for rapid bootstrapping
  • Infrastructure for integration patterns (event streaming, messaging, API mgmt, etc.)
  • Libraries for cross-project problems
  • Unified tracing, monitoring & logging
  • Governance controls
  • Provision of guidelines, best practices as well as coaching and support services

A provided container platform serves as the technical foundation of the platform.

Taking advantage of microservices and integration frameworks

Based on this platform, DevOps teams can develop their integrations independently for the provided container platform (e.g. OpenShift/Kubernetes). The integrations themselves are then developed based on a specific integration framework. There are a variety of such frameworks which are suitable for this purpose. In this blog, we will look at a combination of Quarkus as well as Apache Camel as an example:

  • Quarkus is a Kubernetes-native Java framework that is optimized specifically for containers. It promises a smaller memory footprint which leads to lower runtime costs and faster startup times. In addition, it is also possible to compile the application natively with the help of GraalVM, thus massively reducing the memory footprint and startup time once again, since a JVM is no longer required.
  • Apache Camel is a widely used integration framework, which has over 300 connectors for all possible integration scenarios. Camel is very well suited for use in microservices. Furthermore, it is possible to map even complex integration patterns in a uniform and declarative notation in just a few lines of Java code. This allows integrations to be written "as code" and rolled out using common DevOps methods, which is hardly possible with iPaaS solutions with their low/no-code approaches. Accordingly, many developers prefer integration frameworks instead of graphical solutions. Nevertheless, integration frameworks can lead to complex codes, which in turn requires some expertise.


Comparing a traditional ESB setup with a microservice-based approach reveals several changes. Figure 1 shows an example of what such a difference might look like. On the one hand, the integration logic is moved to where the integration is needed: in or to the actual applications. On the other hand, the integration responsibility is also shifted. This is no longer centralized with an integration team, but shifts to the applications with integration needs. These organizational changes are also discussed in the blog post 5 Learnings for Agile Integration .

  • Quarkus ist ein Kubernetes-natives Java Framework, welches speziell für Container optimiert ist. Es verspricht einen kleineren Memory Footprint was zu tieferen Laufzeitkosten sowie schnelleren Startup-Zeiten führt. Zudem ist es möglich, mit Hilfe der GraalVM die Applikation auch native zu kompilieren und somit den Memory Footprint und die Startup-Zeit noch einmal massiv zu verkleinern, da keine JVM mehr benötigt wird.
  • Apache Camel ist ein weit verbreitetes Integrations-Framework, welches über 300 Konnektoren für alle möglichen Integrationsszenarien verfügt. Camel eignet sich sehr gut für den Einsatz in Microservices. Ausserdem ist es möglich, in wenigen Zeilen Java Code auch komplexe Integrationsmuster in einer einheitlichen und deklarativen Schreibweise abzubilden. Damit werden Integrationen “as Code” geschrieben und mit gängigen DevOps Methoden ausgerollt, was bei iPaaS Lösungen mit ihren Low-/No-Code Ansätzen kaum möglich ist. Entsprechend bevorzugen viele Entwickler Integrationsframeworks anstelle von grafischen Lösungen. Trotzdem können Integrationsframeworks zu komplexen Codes führen, was wiederum einiges an Expertise benötigt.

Vergleicht man ein traditionelles ESB Setup mit einem Microservice-basierten Ansatz, so zeigen sich einige Veränderungen. Abbildung 1 zeigt exemplarisch, wie ein solcher Unterschied aussehen kann. Einerseits wird die Integrationslogik dorthin verschoben, wo die Integration nötig ist: in respektive zu den eigentlichen Anwendungen. Andererseits verschiebt sich auch die Integrationsverantwortung. Diese ist nicht mehr zentral bei einem Integrations-Team, sondern verlagert sich zu den Anwendungen mit Integrationsbedarf. Diese organisatorischen Änderungen werden auch im Blogbeitrag 5 Learnings for Agile Integration diskutiert.


 

202108_Blogserie Agile Integration_Integration as a Microservice_Nr3_DE.png
Figure 1: Traditional integration middleware vs integration platform with microservices

This distributed and cloud-native architectural approach also allows integrations as a microservice ...

  • scale & deploy individually
  • be portable (multi- and hybrid-cloud capability)
  • to be tested automatically (CICD)
  • to develop faster and with existing know-how (e.g. Java or agile methods)

But what is also true for integration microservices (as well as for microservices in general): the appropriate "cut" is often not so easy to make. The Domain-Driven Design (DDD) approach can help here, in which the modeling is carried out from a functional point of view and thus supports the division of the system into suitable units.

Conclusion

Integration as a microservice is already a reality and thus the integration layer can also benefit from the above-mentioned advantages of a cloud-native microservice architecture. Of course, as with so many architectural decisions, trade-offs must be made. In this case, a trade-off must be made between flexibility and complexity. The application landscape becomes more heterogeneous and resources are needed to build up expertise. Nevertheless, a liberalization of the integration layer takes place, which can contribute to the agility of a company.

 

Blog Series:

#1

#2 | 5 Learnings for agile integration

#3 | Integration as a Microservice (coming soon)

#4 | Controlled Risks: Agile integration requires automated testing (comming soon)

#5 | Agile Integration made easy with Red Hat (comming soon)

Your ipt expert

I look forward to hearing from you