Distributed Messaging’s Crucial Role in Artificial Intelligence

The current 10-year cycle in Enterprise IT is dedicated to the implementation of Artificial Intelligence (AI) solutions based in Machine Learning (ML) on “Big Data” platforms. Distributed messaging in data streams currently provides the event-driven architecture for most of the world’s largest ecommerce, social media, and web publishing applications operated by enterprise corporations online. The DevOps requirements for managing these software applications in cloud data centers for billions of users at a time with companies like Google, Apple, Facebook, Microsoft, Amazon, and Ali Baba show how Artificial Intelligence is becoming more critical to automation and innovation in platform services. Distributed messaging permits the object-oriented processing of events in data stream architecture for coding machine learning solutions into custom software applications to add new features and functionality.

The diversity of developer solutions using AI/ML is increasing at high rates of growth on the basis of cloud TPU platforms like Google’s TensorFlow, Azure Machine Learning, or Amazon’s Sagemaker. Many companies are currently using AI/ML in production for image, video, text, and voice recognition services. For example, a factory may use image recognition and AI to sort products for defects via assembly line robotics or a website may use voice recognition with AI for translating conversations in real-time for customer service. Ecommerce companies use AI/ML for content and product recommendations that are based on a user’s browsing history, likes, or purchases. Distributed messaging services built around coded events in a software platform provide the raw data packets for AI to use for processing and pattern recognition at scale for websites and mobile applications with hundreds of millions of registered users.

In social networking, billions of posts from user profile pages are recorded each day, along with the user’s likes, visited websites, browsing history, purchase patterns, etc. for the purposes of display advertising, content screening, and contact recommendation services. Services like Spotify can use the events on their platform, i.e. music that a user listens to or likes, including historical data to recommend featured content that other users with similar browsing patterns also liked for recommendations. The cloud AI/ML platforms use TPU-hardware for data science based on algorithmic learning patterns from huge data lakes of customer information to make their recommendations. Amazon, Google, Microsoft, and other major enterprise companies use AI/ML to offer better product recommendations to their customers in ecommerce, social networking, or SaaS products like CRMs with B2B and supply-chain modeling capabilities. 

With millions of simultaneous users on a website, DevOps technicians need to guarantee that AI/ML solutions operate with security, reliability, and advanced functionality. Distributed messaging systems offer a tested and proven means to operate AI/ML functionality at the scale of LinkedIn, Amazon, or Google at maximum capabilities. Two of the major development companies for autonomous driving vehicles, Tesla and Uber, are also implementing distributed messaging services to create the “events” needed to create AI-driven software for self-navigating vehicles. When architecting the requirements for a national or international network of autonomous, self-navigating vehicles using AI/ML for image/video depth processing, the number of events per second becomes much larger than any of the available software services on the market today. Distributed messaging technology evolving out of the Apache Hadoop project with Kafka and Pulsar are providing the platform base for developing these services integrating self-navigating vehicles with cloud TPU hardware for data processing in combination with onboard IoT and edge servers.

In summary, the world’s largest websites and mobile applications use distributed messaging services in data stream architecture to support billions of events recorded from their users every day in simultaneous online activity. The data center requirements to support event-driven processing at this scale have led researchers in computer science over the last 20 years to continually build new solutions. The Apache Hadoop ecosystem includes Kafka and Pulsar which are the premier open source solutions for enterprise. Apache Kafka and Pulsar offer two unique and interrelated approaches to distributed messaging in event-driven architecture that is being implemented for AI/ML solutions by programming teams worldwide for social networking, ecommerce, streaming media, and autonomous vehicle navigation systems. Consider new serverless architecture for quick uptake and parallel administration of these services.

Leave a Reply