Agile Integration made easy with Red Hat

Blog Series | Agile Integration | #5

In our blog series on Agile Integration, we introduced you to possible forms of agile integration, you learned five learnings , saw how it interacts with microservices, and received recommendations for optimal testing.

Now, in this blog post, we will use a fictional example to show how the Red Hat Integration Bundle can be used to efficiently build and effectively run an agile integration platform.
By applying the best practices described in the blogs published so far, integrations can be developed within a very short time. These bring all the required data to and from the business applications in a high-performance manner and thus improve the user experience.

What defines an agile integration platform?

In the past, an integration platform usually consisted of a central infrastructure component on which all integrations ran, regardless of which patterns or concepts were used. This often involved specialized development tools with unique processes that had to be used by a dedicated integration team.
In the modern agile world, the central infrastructure component is distributed into different components that can be operated as containers based on their functionalities. In addition, instead of specialized development tools, frameworks are used that integrate with the most popular programming languages. This means that developers can continue to use their familiar tools and processes without much effort.
To support DevOps processes, all components will be automatable via APIs, CLIs or other interfaces.

How does Red Hat implement an agile integration platform?

The Red Hat Integration Bundle consists of multiple integration frameworks and specialized infrastructure components. In short, a classic Distributed Integration Platform (DIP).

Figure 1: Red Hat Integration Bundle (excerpt)

The integration frameworks Fuse for Enterprise Integration and Debezium for Data Integration are based on Java. Thus, they offer easy integration into runtimes such as Spring Boot or Kafka Connect. Java support and the wide range of plug-ins for IDEs and build tools allow established DevOps processes to continue without any problems.

The following infrastructure components rely on the OpenShift Container Platform and can be easily installed and configured in a distributed manner via operators within minutes:

AMQ Broker for messaging
AMQ Streams for events
Apicurito for API design
3scale for API Management
Apicurio as schema registry

Self-service functionalities can be implemented via APIs and CLIs and are then available to developers. Thus, the Integration Bundle supports DevOps processes as well as hybrid and multi-cloud models.

PS: With Fuse Online, there is another infrastructure component that is intended for Citizen Integrators. In principle, this is an iPaaS with a browser-based front end for very simple integrations. However, we won't go into it further in this blog.

How does this work in practice?

In the past, you knew the phrase 'We integrate application X with system Y'. Now we say 'Application X integrates system Y'. This is to show the difference between classical and modern forms of integration.

Today, integration is directly part of the application implementation and should no longer be seen as a separate development discipline. This means that integration microservices are implemented by development teams in the same way as business microservices.

As a prerequisite for this type of agile integration, a corresponding organization is required, which is supported both by the management and, above all, by the teams. Only if the organization is set up accordingly, i.e. development teams are empowered (e.g. through training, consulting or blueprints), integration can successfully happen directly in the development teams.

Is there an example?

As a fictional example, let's use a classic microservice application with:

Business application
Backends for core systems
Classical queues for decoupling a database

Figure 2: Implementation Enterprise Integration with Camel

Distributed DevOps teams could now implement such an application using the Red Hat Integration Platform and integrate it with other systems as follows:

Apicurito is used to create the API definitions for the backend services.
Fuse projects for the backends are generated directly in Apicurito from the API definitions.
In addition, the API definitions are published to the Apicurio Schema Registry, where the definitions for the asynchronous messages are already stored.
With this contract-first approach, the distributed teams independently develop the different components of the application and integrate them using CICD pipelines.
Automated testing at all stages ensures quality and compatibility.
The individual components are deployed as containers on the OpenShift platform and further developed in independent releases
The 3scale gateways and AMQ brokers are deployed, configured and operated by the DevOps team itself in self-service mode
Accesses to backends, databases and queues are integrated into the Fuse components using Camel technology adapters, which are available for most systems and protocols

Now we extend the fictitious example with data integration capabilities. That is, events on the database are written to an event streaming system, to which other applications can respond.

Figure 3: Implementation of Data Integration with Debezium

A (different) DevOps team could extend the existing application as follows:

A Kafka Connect component for the database is generated with Debezium
Like the previous components, this component is also deployed as a separate container without dependency on a Kafka Connect cluster via CICD pipeline and individually developed further
The schema for the events is published in the schema registry for potential clients
For the central AMQ Streams platform, topics and users are created via self-service

The whole example can be implemented exclusively with Camel or Debezium DSL code.

202107_Blogserie Agile Integration_Blog Nr5_Abbildung04.jpg

Abbildung 4: Code Snippet mit Camel DSL 'Rest zu Queue zu DB'

For special cases, custom extensions can be created with reasonable effort thanks to many entry points. More complex integrations also follow the same pattern, as technology and processes can be easily scaled.

Is an investment worthwhile in the long run?

All components are based on well-known open source projects. Fuse on Apache Camel, AMQ Broker on Apache Artemis, AMQ Streams on Apache Kafka / Strimzi, etc.
The question, why one should acquire a subscription from Red Hat, if one can assemble the complete integration platform for free, is thus quite justified.

The question of direct costs is only one side. On the other side are the indirect costs, if one would do without a subscription.

The most obvious is the lack of support from Red Hat, which should be provided promptly (e.g. via analyses or hot fixes), especially for productive problems. Due to the pure open source community, this is not guaranteed. Expenditures for independent troubleshooting and subsequent correction of these can quickly increase, and potentially long downtimes can also occur.
A second source of indirect costs is the interaction of the individual components, predominantly with the integration frameworks. Like any software, these frameworks also have dependencies on other libraries. Here the version management is central, whereby with a Subscription over extensive tests by Red Hat is ensured that all used libraries co-operate problem-free. Here, own efforts for version management can also grow quickly and are also recurring with new releases.

What are the advantages in summary?

In this blog we have shown how easy it is to build an agile integration platform with the Red Hat Integration Bundle.
By supporting standard technologies, microservice architectures, and DevOps processes, teams are empowered to easily develop and run distributed integrations independently without depending on a central integration team.
With consistent use of cloud-native technologies, the platform is ready for use in private, public, hybrid and multi-cloud scenarios.
As a result, this modern integration platform supports the development of business solutions by flexibly making data available where it is needed. Moreover, this data is available in the desired form. This increases the satisfaction of developers and users, i.e. of customers.