Your event-driven apps are only as good as your event broker

By Shawn McAllister, Solace’s CTO and Chief Product Officer

In the first of a two-part series, McAllister discusses the critical role event brokers play in event-driven architecture (EDA).

Concept art representing business complexity generated
by Blue Willow.

EDA, a software design pattern that can enable real-time, event-driven communication among diverse applications via an event broker, is becoming pervasive among enterprises that want to power time-sensitive applications, business processes, and insights (advanced analytics, machine learning [ML], AI) at scale.

Implementations of EDA platforms are now on the fast track, with 82% of IT leaders saying their company plans to apply EDA to 2-3 new use cases within the next 24 months. A new IDC Infobrief charts this rise. EDA adoption goes hand in hand with digital maturity, with 47% of respondents describing their EDA journey as either maturing (“centralised”) or “advanced”.

And the rise of EDA is industry-agnostic. Research shows more and more businesses across all industries – retail, financial services, aviation, manufacturing, transportation and logistics and more – are seeing the need to adopt this pattern of application connectivity, to effectively enable modern use cases like kerbside pickup, preventative maintenance of machinery, digital twinning etc.

At the very heart of EDA transformation is the event broker, middleware software that routes events and other data between various applications, systems and devices. As EDA has risen in appreciation and use, so has the market for event brokers grown, to the point today that there are a wide variety of brokers to choose from. Event broker capabilities can vary drastically, so it is important for architects and application teams to consider their evaluation criteria carefully.

An event broker is message-oriented middleware that enables the reliable transmission of events between different components of a system, acting as a mediator between publishers and subscribers. It is the cornerstone of event-driven architecture, and all event-driven applications use some form of an event broker to send and receive information.

As a first and critical step, the choice for IT leaders is now not whether to embrace EDA, but rather which event broker to choose to underpin their EDA, or which ones to use for which use cases, as often an enterprise might find that they need more than one type of broker, because some are better suited to particular use cases.

Analysts are jumping into the debate, notably, David Mooter of Forrester who recently outlined the choice between a “log stream broker” vs a “smart broker”. A log stream broker can support high data throughput and tolerate some degree of complexity, as well as message replay when combined with real-time analytics and event sourcing. A smart broker, on the other hand, can support complex message routing, granular controls on message filtering, global order guarantees, and transactional commits as well as many other capabilities.

Use cases signpost the choice – analytical vs operational

The fundamentals of event brokers, Mooter explains, allow organisations to build their applications as a collection of composable services that are decoupled in nature. This provides the benefits of agility, scalability and resilience but, depending on the use cases, he argues, choosing between a log stream broker or a smart broker can make all the difference.

Log stream brokers are best epitomised by Apache Kafka. Over the last few years Apache Kafka has taken the world of data streaming by storm, because it’s very good at its intended purpose of aggregating massive amounts of “log” data and streaming it to analytics engines and big data repositories.

Not all event brokers are equal to the job in hand

Unfortunately, the popularity and prevalence of Kafka has led many developers to use Kafka for use cases for which it is not ideally suited – namely operational, “run-the-business” scenarios which often involve a mix of applications, systems and devices that need to tap into specific event streams to run effectively.

Log-based brokers use rigid, flat topic structures to describe the data they transmit, which puts the onus on the applications to do the work to filter through all the data that’s being streamed. This can be like drinking from a firehose, as applications need to consume, filter and throw away events they don’t need which leads to increased cost and complexity, not to mention security concerns.

As businesses get more connected, the broker must get smarter

On the flip side, ‘smart’ brokers do a lot of the thinking, filtering, routing – especially as enterprises work to become more integrated, connected and real-time across diverse IT and operational technology (OT).

These brokers feature rich and flexible topic hierarchies which enable applications to easily publish and subscribe to the very specific subsets of data they are interested in. From an event streaming perspective, smart brokers support a wide range of message exchange patterns beyond publish/ subscribe, including request/ reply, streaming and replay, as well as different qualities of service, such as best effort and guaranteed delivery.

This makes smart brokers ideal for operational use cases, acting as the “digital nervous system” of a distributed enterprise. Smart brokers go beyond simply analytics – imagine a global bank streaming over 150 billion events a day between low-latency trading platforms and market data centres in multiple locations across the globe.

The choice becomes clear – if an organisation is looking to address operational “run-the-business” applications and use cases across a distributed enterprise, they need a smart broker.

This is the first of a two-part series. In part 2, out next week, McAllister details how readers can choose the right smart broker to run their business.