The rise of enterprise integration - our business landscape calls for an architectural rethink

by Shawn McAllister, CTO and Chief Product Officer at Solace

A large multinational enterprise typically comprises thousands of applications, with data inputs spanning hybrid/multicloud environments, Internet of Things (IoT)/mobile devices, and distributed operations.  

These data events, by their very nature, happen in real-time – a customer places an order online; a supplier updates available inventory; a passenger scans a boarding pass; a sensor detects a sudden temperature change. These events are not synchronous, and they all trigger follow-on actions that ripple throughout different departments and operations across an enterprise.

The vast volumes of information in play, all containing data critical to an enterprise’s day-to-day operations, is underscoring a dramatic shift in the way increasingly globalised business systems are integrated.

The old integration ways cannot match the needs of today’s real-time business world

Traditional methods such as integration platform as-a-service (iPaaS), and enterprise service bus (ESB) are still being used to knit together data across this complex web of disparate systems. However, they struggle to keep pace with the demands of today's data-driven landscape. Synchronous, point-to-point solutions simply aren't built to handle the ever-growing volumes of real-time data coursing through modern enterprises to an ever-increasing number of applications. 

Traditional integration methods result in a complex web of connectivity that is difficult to understand, fragile, can’t deal with bursts and is generally not robust to failures and outages. This creates longer and longer application runtimes which, in turn, result in poor user experience.

To simplify integration, become more real-time, and more connected, organisations need to adopt an “event-driven” approach.

Event-driven data needs event-driven integration

Event-driven integration offers a new and innovative approach to connecting systems through the instantaneous sharing of real-time events. At its core, whenever an event occurs within a system, a message is published to a central hub called an event broker. Other systems subscribe to this broker, or a network of event brokers, called an event mesh, receiving messages in real-time and reacting accordingly.

Concept artwork on connected devices generated by Blue Willow.

As we say in the real world – they are all on the same page! This on-demand, “data as a service” approach unlocks a level of flexibility and scalability far more dynamic, responsive, and real-time than traditional methods.

Leading analysts now recognise event-driven integration as a key component to optimise real-time movement of business-critical data. Gartner identified this trend in the evolution of IT systems architecture, heralding an event-native mindset that moved from looking at IT systems as being “data (centric) custodians” to one that looks at IT systems as comprising the “nervous system” of an enterprise. More to the point – viewing “data in motion” as opposed to “data at rest” as the real source of effective decision making.

It should be noted that this event-driven integration approach works perfectly well with an organisation’s existing iPaaS, essentially augmenting the iPaaS with an event-driven platform. In fact, with iPaaS and event-driven integration working in tandem, IT teams can migrate the right information flows incrementally to move towards becoming event-driven, allowing a phased implementation of key business processes over time.

This is reflected by IDC recently, noting in IDC Market Glance - Connectivity Automation, 2Q24*: “Another trend that is helping fuel the growth of event-driven architecture (EDA) is the potential to utilise EDA in conjunction with iPaaS to split queuing, avoid bottlenecks, and manage workload and data traffic asynchronously. With an event broker layer, an organisation can also ensure that they have visibility into the state of queued messages even if there is a sending error.”

An event-driven approach means re-thinking and re-architecting integration.

Integration of the past 20 years has been fairly centralised, with monolithic applications with many dependencies. Current integration approaches have placed integration components in the data path – i.e., at the core. Integration, whether real-time or batch, requires connectivity and transformation. The challenge with centralised integration approaches is that they couple connectors, transformations, mappings, and potentially transactional context into one deployable runtime, through which all data must flow in a synchronous manner. As such, the integration component can become a bottleneck and suffer from many of the same problems as monolithic applications.

Event-driven integration turns this approach inside out. It moves integrations and connectors to the edge, with decentralised and real-time data flow and events in the middle. The result is an application and integration architecture that is more agile, scalable, robust and real-time – much like event-driven microservices.

Today’s use cases demand an integration architecture that can handle both traffic bursts and slow/offline consumers without impairing performance; can scale to handle an increase in consumers, producers and in data volume; and can guarantee delivery of data even to temporarily unavailable consumers. It also demands an architecture that lets you easily plug in processing components and distribution mechanisms without impairing the overall design and even to easily adopt new technologies you had not even imagined before – such as generative AI (gen AI) agents and large language models (LLMs).

Arriving at an optimal event-driven integration implementation does not happen overnight. It’s an evolutionary process that can be measured in four transformative milestones:

Breaking down data silos

Traditional systems often create data silos, hindering accessibility and informed decision-making. Event-driven integration fosters information democratisation and availability, allowing businesses to readily access the data they need, when they need it.

Dealing with unexpected bursts in traffic

The decoupling of components serves as a “shock absorber” that gracefully deals with unexpected bursts of traffic, such as sudden spikes in demand, leading to vastly increased order volumes. This decoupling also provides a more robust infrastructure that is tolerant of failures and outages and prevents cascading failures.

Nurturing innovation at warp speed

Event-driven integration streamlines the process of integrating new applications and services, empowering businesses to innovate faster and stay ahead of the curve. If you used event-driven integration today, I guarantee it would be faster and easier to give your gen AI applications real-time enterprise context to work from, because you would just add these new components to your existing event distribution architecture.

Keeping the user’s finger on the pulse

Enabling real-time data integration provides users with a more consistent and up-to-date view of information, leading to a smoother and more efficient experience.

Word is getting out as organisations across industries such as financial services, manufacturing, retail, and more are beginning to embrace event-driven use cases.

PSA Singapore has adopted an event-driven strategy as part of its journey to bring to life the world’s largest automated container terminal. Event-driven integration is a critical pillar of Tuas Port’s automated operations and maintenance, acting as a resilient shock absorber that is transforming port operations, setting new standards for efficiency, adaptability, and customer satisfaction.

Singapore’s national debit system, NETS, also leverages an event-driven approach to securely route real-time information across large numbers of transactions at more than 102,000 acceptance points. With a continuous event-driven flow of information, NETS can quickly develop and deploy new digital payment services, have visibility into customer behaviour, and tailor new services and loyalty programmes in tandem with the shift to cashless commerce.

Testament to this, a recent IDC Infobrief found that 82% of survey respondents have either begun or are planning to add between two to three event-driven use cases in the near future.

Increasing data volumes and connectivity levels amid shifting consumption models and customer expectations are changing the way large organisations need to architect the flow of critical information across their business. Traditional integration approaches are no longer able to support businesses that have to match customer, employee, and supplier requirements in real-time.

Event-driven integration offers a path for multinational businesses to modernise their integration strategy and empower their business to be more adaptable, scalable and robust – in a landscape that is only going to continue to become more digitised and real-time. 

*Shari Lava, Andrew Gens, June 2024.