Blog Series | Agile Integration | #1
Author: Thomas Stuber
Traditional, centralized integration solutions lead to organizational bottlenecks and miss the advantages of modern architectural concepts such as microservices or cloud-native developments. What is needed, therefore, are innovative approaches that enable decentralized integration by DevOps teams (see blog post Rethink your integration strategy). But which forms are available? And which ones are suitable for your needs? In this blog, we provide an overview of which forms can be used for agile integration and which ones fit your needs.
Simplified, the technologies and concepts that can be used for agile integration can be broken down as follows (based on Gartner ):
Typically cloud-hosted integration platforms that offer mostly low- or no-code applications. These offer ease of use, which also appeals to "citizen integrators".
A DIP is provided by a platform team itself. The users of the DIP develop integration logic, which is packaged as microservices and deployed as containers.
An integration framework allows integrations to be implemented directly in one's own application without deploying an additional container. Often, these frameworks are also used with DIPs for implementation.
For each of the aforementioned forms, there are a variety of products on the market. A clear demarcation between them is not always given. Often the functionalities merge into each other. It is of course also possible to use these forms in a hybrid mode - completely individually and depending on the needs and strategy of the company.
If the forms of agile integration are considered in terms of complexity for implementation and operation, as well as deployment flexibility, there is the following order of preference:
An iPaaS offers the advantage that it does not have to be operated by the user. It also enables users to use it quickly and easily via GUI. Citizen integrators are also served, i.e. people who are not software developers. Accordingly, no-code or low-code approaches are used for integration. Prefabricated adapters enable the connection of a large number of applications, systems or data types. In the cloud environment in particular, there is a wide selection, well-known examples being Dell Boomi or Microsoft Azure Integration Services.
In practice, however, such iPaaS solutions often quickly have limitations. Often, the integration problems are so diverse and specific that the simple adapters of iPaaS solutions cannot cover all cases (for example, the connection to event streaming systems). If extensions can be added by oneself, again a development team is needed to program the extensions. Therefore, in our opinion, iPaaS solutions are more suitable for companies with a manageable integration complexity or with strongly standardized (cloud) interfaces. At best, also as a supplement to an existing DIP, provided that dual operation is economically worthwhile.
On a DIP, on the other hand, integrations are typically developed as code. This gives much more control and flexibility in the implementation compared to an iPaaS. It also allows DevOps teams to build and operate integrations in a manner analogous to the DevOps philosophy and agile methodologies. The DIP also provides the necessary foundations that the teams can use for implementation (e.g. libraries, monitoring, documentation) or also supporting infrastructure for the common integration patterns (e.g. streaming, messaging or API management). Preferably, integrations are developed in a DIP using an integration framework. This achieves a higher level of standardization and also benefits from the framework's broad and specialized range of functions. In our experience, developers who already work with microservices have a natural and very fast access to a DIP. Red Hat Fuse on OpenShift or MuleSoft Anypoint Platform (Private Cloud Edition) are corresponding examples of such a platform.
The downside of a DIP is that the development teams have to build up some know-how for developing integration microservices and should understand modern ways of working according to agile methods. In addition, the effort required to operate a DIP is significantly higher than using an iPaaS.
Integration frameworks are libraries that provide essential integration patterns out of the box. These frameworks can either be embedded within an application, or used to build dedicated integration applications (microservices) (DIP). Integration frameworks offer their own Domain Specific Language (DSL), which can be used to efficiently implement the integration tasks. Furthermore, they hold a variety of adapters ready to integrate common application interfaces (REST, SOAP) and data types (JSON, XML). Well-known examples of integration frameworks are Spring Integration or Apache Camel.
From the developer team's point of view, integration frameworks are exciting tools that offer a lot of flexibility and are also a perfect fit for common agile development methods. If integration frameworks are used without a DIP, aspects such as governance and operations (e.g., monitoring) are typically missing. There is a risk of a certain "wild growth". Consequently, further effort is required to address this issue.
In this blog, we have created an overview of which technologies and concepts can be used to implement agile integration and have highlighted the advantages and disadvantages. In summary, we recommend the following deployment scenarios based on our experience:
However, "the" solution does not exist here either. Ultimately, it depends on the concrete problems and the needs of a company.
Gartner (10 March 2021): Choosing Application Integration Platform Technology
#1 | Forms of agile integration - an overview