New Frontiers for the Enterprise Service Bus

Important advances in Business Intelligence (BI) in conjunction with advances in technology  continually increase the need for integration of systems within all enterprises today. Enterprise resource planning (ERP) application, to name one example, now consist of numerous component technologies such as analytics performance tools, IoT sensors and event streaming to machine learning applications, as well as databases and visualization tools, programming tools, and much more. Ideally, these components are integrated smoothly to communicate with each other and provide resource sharing ubiquitously across enterprises such as: 

• Customer Relationship Management (CRM)
• Human Resource Management (HRM)
• Supply Chain Management (SCM)
• Accounting and Finance (AF)
• Sales and Marketing (SM)

The unique needs of these enterprise applications motivated the development of diverse integration tools over the years. But the unique feature requirements of one enterprise integration yielded solutions which were not fully suitable for another enterprise, due to differences in  business logic. In other words, the same integration tool which provided freedom and scalability for one enterprise resulted in limitations and the necessity of ad hoc coding for another enterprise. The most familiar of those integration solutions include:

• Service-oriented architecture (SOA)
• Event-driven architecture (EDA)
• Business-to-business (B2B)
• Services provided through Web (SPW) 
• Enterprise application integration (EAI)

Rather than listing the advantages and limitations of each, we will devote our attention to the  one integration strategy which has now evolved to unify all of the above. To address the challenges of improving performance across all enterprises and their component technologies, the Enterprise Service Bus (ESB) now emerges as the most effective and comprehensive integration strategy. Among the brightest technologies emerging to supply the functionality needs of ESB is Apache Pulsar.  

Popular Apps and ESB 

While big stakeholders such as financial and investment enterprises exert the most influence on enterprise software evolution, we can also see the benefit of ESB in everyday applications. Today you can pop up a live flight tracker application on your phone and see every airplane in the sky over the US or China in real-time! Applications like this depend on the inter-connectivity of many and varied systems which themselves depend on the spontaneous translation of numerous data formats and transport protocols. The speed at which the data from flight transponders and sensors is encoded and relayed through databases, analytics engines, and ultimately to the user, is even more impressive. Live flight tracking, as well as rideshare applications that connect millions of passengers and drivers via real-time event streaming, are all ultimately driven by a promethean workhorse: the Enterprise Service Bus. 

In fact, it is none other than this dry sounding abstraction straight out of a computer science textbook – the Enterprise Service Bus (ESB) – which integrates a plethora of otherwise disparate frameworks and platforms into a harmonious and cooperative orchestration. Indeed, these applications and microservices, each of which contains genius of its own, all speak in languages, protocols, and data formats which would be mutually unintelligible without the ESB, or, as in previous years, require ad hoc coding to interface. What is an ESB and how does it do these miracles?

The Enterprise Service Bus Brings it All Together

In the 1990s, engineers who coded ad hoc interfaces between ERP systems, factory sensors, GPS receivers and many others, realized that standard, recyclable interfaces should be built to abstract and automate this coding, and that was the origin of Service Oriented Architecture (SOA), one of the architectures from which ESB evolved. Still another important evolutionary step soon followed, that of the Event Driven Architecture, which we will elucidate shortly. SOA and EDA were initial steps toward the integration of large enterprise systems such as:

• Enterprise Resource planning (ERP)  
• Customer Relationship Management (CRM) 
• Enterprise Financial Services (EFS)

Only a few years ago, the system integration required to operate the commercial flight network or ridesharing applications described above could be achieved only by the giants of the time such as IBM, or by laborious in-house coding. Today, however, these feats of engineering prowess can be accomplished for a fraction of the price by open source ESBware including Apache Pulsar. Our purpose here is to explore today’s ESB architecture, look at some of its groundbreaking use cases, and identify leaders who are pioneering this orchestration.

Increasing Relevance of ESB Today

ESB is absolutely the most promising strategy for integrating business applications across distributed and diverse frameworks and platforms today. Abundant research demonstrates this successful trend. With a fundamental vision of ESB providing elastic interfaces among numerous applications, the material outcome of ESB is a middleware layer which is a superset of both SOA and EDA. As such, ESB provides support for transforming messages, live streaming data, smart routing, as well as protocol transformation. ESB is therefore also an important component of emerging machine learning applications, as well as a critical part of the recent Edge Compute trend.

ESB Upper and Lower Level Tech Today

Comprehensive ESB functionality nearly defies description because it seeks to enable communication among all applications, data formats, and protocols. Let’s look first at an upper level set of the general connectivity required to meet standards, and thereby begin to open the view toward defining ESB:

• Technologies including MQTT, SOAP, REST, JMS
• Data formats such as Apache Avro, JSON, XML
• Open frameworks including Nginx or Kubernetes 
• Proprietary interfaces such as EDIFACT, SAP BAPI
• Programming languages including Java, .NET, Go, Python 
• Application architectures: Client Server, Monolith
• Communication paradigms: batch processing, real time event streaming, as well as Pub-Sub 

As our view expands, hints of the complexity of ESB functionality applicationear. The functional benefits of ESB are compelling, and the complexity leads many enterprises to seek third party solutions instead of developing in-house.  To lend further support for that trend, let’s have a brief look at a finer granularity in the lower level specifications of ESB architecture:

Messaging	Send-receive, message processing, message transformation and enhancement
Event Processing	Real-time event stream processing
Sync/Async	Synchronous and asynchronous transport protocols
Routing	Content-based routing, addressable, static/deterministic routing, rule-enforcement, policy-enforcement
Mediation	protocol transformation
SOA	Service Oriented Architecture support
EDA	Event Driven Architecture support
Business Process	Business Intelligence BI workflows and applications connectivity
Functions	event-interpretation, correlation, pattern-matching
Security	Encryption, TLS, delivery guarantees, transaction management
Analytics Management	monitoring, audit, logging, metering, admin console, BAM (BAM is not a management capability in other words the ESB doesn’t react to a specific threshold. It is a business service capability surfaced to end users.)
Platform agnostic	general agnosticism to operating-systems and programming-languages; for example, it should enable interoperability between Java and .NET application
Protocol Conversion	comprehensive support for communication protocols
Adapters	Integrate legacy systems
Multi-tenancy	Security-model for authorization and authentication of ESB users
Governance & Compliance	Enforcement of business rules and specs
Abstraction	Capability to extend reusable and recyclable code and services throughout  enterprise application

ESB Use Cases in Real-Time

Financial services enterprises represent a vast collection of software application which require real-time messaging, transaction, translation, process orchestration, and protocol transformation. For example, international stock markets presents a use case challenge for ESB, which is to implement a fully scalable and flexible middleware layer to integrate of a

set of systems to handle trade and price information during trading, clearing, settlement, and  deposit services. An ESB solution integrates multiple transports and protocols, using transformed XML payloads. A key requirement is the guarantee of message delivery and zero downtime, with monitoring and logging for performance analytics. One innovative solution uses the Publisher-Subscriber model, incorporating Apache Pulsar via a Pulsar host such as Pandio to integrate upstream systems and downstream systems. Providers of ESB middleware solutions include:

• IBM App Connect
• Open ESB 
• Mule ESB 
• Software AG webMethods
• Peregrine Connect.
• webMethods Integration Server
• Pandio

Bringing ESB to Fruition

Important to choosing an ESB solution today is the cost – benefit relationship. We are no longer dependent on the giants. Fortunately, leaner and more assertive companies offer ESB solutions today which are 2.5 times faster than traditional competitors at half the cost! For example,  Pandio is the only ESB designed for next generation workloads (AI/ML). Based on the purely Cloud native PubSub solution Apache Pulsar, Pandio hosts a full range of ESB functionality to connect SOA and EDA based architectures in diverse applications including:

• Real-time driver & passenger locations
• AI-driven traffic and ETA forecasting
• Emails to BI members
• Application performance metrics logs feeding analytics
• Code logs like Heroku
• License tag lookup API and other microservice interfaces
• Event streaming locations from GPS microservices

A growing number of enterprises today recognize the increasing complexity of interfacing software applications. While ESB is an elastic interfacing solution, outsourcing the  implementation of ESB substantially reduces in-house development overhead. Furthermore, as  Pulsar is the rising star among ESB solutions, more enterprises engaged in AI-based analytics, Edge Compute applications, event streaming, and machine learning, are engaging Pulsar’s speed and reliability by way of hosted Apache Pulsar as a service. Pandio is an emerging leader of ESB built on the proven advantages of Apache Pulsar by a diverse engineering team of interfacing experts dedicated to ESB implementation.