Event Streaming, Messaging, and Queuing for the Software & Technology Industry
Pandio leads in event streaming, messaging and queuing for software and technology companies by hosting Pulsar with an integrated AI neural network. Our resulting increase in performance is unmatched in performance, reliability and security – the KPIs of Software Tech.
Use Cases
- Content update notification
- Pushing content to files
- Logging and log analytics
- Data warehouse indexing
- Event streaming to AI and ML apps
- Queue as a service
- Data center operations
- Manage state for running tasks
- Build interactive apps
- Reconstruct state from event history
- Real time monitoring and notifications
- IoT analytics
- Streaming data transformations
- Real time analytics
- Data distribution
- Event driven workflows
- Pulsar as a Service, billions messages per day
- Enterprise Message Bus (i.e. Splunk DSP)
Content Update Notification
Yahoo! Japan is a massive scale software development enterprise which uses the Apache Pulsar pub-sub messaging platform to manage and deliver its content about news, weather, maps and other topics.1 News updates represent a critically important use case in which Apache Pulsar truly excels.
Content update notification
Yahoo! Japan uses Pulsar cleverly for alerts about page content updates. Partner companies send notifications to topics which triggers pushing content file updates and notifying services to fetch all types of content files, such as weather and news.
Yahoo! Japan also makes abundant use of Apache Pulsar for more specifically technical objectives behind the scenes. This includes the increasing important area of logging, in which AI based analytics now improves site performance and user experience. Mail indexing is also handled by Pulsar:
Log collection
Logging and log analytics have become critical performance components in app development and maintenance. Yahoo! Japan implements Pulsar to collect logs from PaaS, CaaS, and other development platforms. Pulsar Functions are important to filtering and processing log content into data pipelines bound for Prometheus, HBase, and twilio.
Mail service job queues
Another innovative Pulsar use case at Yahoo! Japan handles the heavy loads incurred by mail indexing. Pulsar efficiently manages mail indexing asynchronously so that Producers can send jobs quickly and consumers can fetch jobs as needed.
Yahoo! Japan chose Pulsar because it scales to handle a practically unlimited number of users and services with zero message loss in clusters which scale across data centers. Here is what Joe Francis, Director of Storage and Messaging at Yahoo! said about Pulsar:
“Apache Pulsar is integral to the streaming data pipelines supporting Oath’s core products including Yahoo Mail, Yahoo Finance, Yahoo Sports and Oath Ad Platforms. It handles hundreds of billions of data events each day and is an integral part of our hybrid cloud strategy. It enables us to stream data between our public and private clouds and allows data pipelines to connect across the clouds.“
Data Center Operations
A global cloud provider, OVHcloud built messaging with Apache Pulsar into its network of 30 data centers in order to provide the highest standard of data integrity and security. OVHcloud recently migrated its “Queue as a Service” product to Apache Pulsar to leverage the many improved features evolving in Pulsar. OVHcloud listed the following feature benefits of Pulsar which were not supported in Kafka as reasons for the shift:
- Built-in Multi tenancy and Geo-replication
- Cost free topic creation
- Infinite retention
- Tiered storage, decoupled from processing
- unlimited scalability
Among the outcomes achieved by OVHcloud was the use of Pulsar Bookkeeper in the creation of their “Z Garbage Collector” and Prometheus Exporter, both feature additions to their platform. They also listed multiple journalDirectory to better exploit SSD throughput, and one ledgerDirectory per HDD. In his presentation at the Pulsar Summit, Pierre Zemb expounded the technical advantages of Pulsar in significant detail, including OVHcloud’s own internal benchmarking outcomes showing the clear superiority of Pulsar.3
Build Interactive Applications
Iterables is a growth marketing platform with an impressive list of clients, and their mission is to “maximize lifetime value by enabling you to create better experiences for—and deeper relationships with your customers.” Clients including AT&T, ASICS, and Zillow rely on Iterables, which in turn relies on Apache Pulsar as its core messaging platform. Let’s explore some of the important use cases in which Iterables leverages Pulsar’s superior architecture.
On the business side, Iterables provides its clients with B2B tools for customer engagement across a wide range of markets. To achieve this on the technical side, Iterables develops interactive products which communicate with each other via the Apache Pulsar messaging platform. And Iterables’ clients likewise interact through diverse event types through billions of events, workflow states, and services every day.
Although Iterables ideally uses stateless services as much as possible, developers use queues to manage state for long running tasks. Here, Pulsar excels. This diagram shows how Iterables uses Pulsar queues:
Furthermore, along the lines of Iterables’ technical requirements suited perfectly to Pulsar’s Broker architecture:
Ideal Message Broker Requirements
- Supports ordered streams of messages
- Supports work queues
- Highly available
- Delayed work
- initiate some action at a specified time
- retry this message after N minutes
- Many queues
- segmenting queues by customer and further by type
- 10k+ for each service
- different rate limits
In Greg Methvin’s Pulsar Summit presentation, he pointed out that RabbitMQ is not adequate for these requirements. Furthermore, he listed the failure of Kafka in work queue use cases, as well as its lack of support for large numbers of topics. He went on to emphasize that SQS also fails to support delays and inadequate maximum payloads. In highlighting these shortcomings, Methvin demonstrated that Apache Pulsar was therefore the clear choice to fulfill Iterable’s requirements for messaging and queuing. On the positive side of the ledger, Iterables’ reasons for choosing Pulsar included:
Why Iterable chose Pulsar
- Supports our scale:
- a million or more topics
- easy to add storage capacity
- easy to scale throughput
- Key features:
- both streaming and queuing
- negative acknowledgments
- built-in delay functionality
Abundant Use Cases
In the course of fulfilling the Big Data analytics functions for its more than 10,000 clients, Splunk developed a vast set of use cases for Apache Pulsar. Splunk is a comprehensive Cloud analytics application which captures and analyzes diverse data from Internet businesses and generates summary graphs in dashboards. Splunk Pulsar use cases include:
- Real time monitoring and notifications
- Interactive applications
- Log processing and analytics
- IoT analytics
- Streaming data transformations
- Real time analytics
- Data distribution
- Event driven workflows
Top Software Technology Companies Utilizing Pulsar
Why Pandio?
Pandio hosts the perfect distributed messaging by integrating a neural network into Apache Pulsar, creating unparalleled performance and efficiencies as a fully-managed service.
- Scale with confidence
- Pay only for what you use
- Ready-to-go in minutes
- One Click Deploy
- Nothing beats Pandio
- < 10ms latency at any scale
- 2 million messages per second per partition
- Optimized with Artificial Intelligence
- Proven capability
- 99.99% uptime guarantee SLA
- Mission critical applications
- Enterprise grade
- Your trusted partner
- Zero operational burden
- Teams of messaging experts
- There when you need it most
- Pandio has it all
- Streams, Queues, & PubSub
- Integrate Quickly With Stateful Functions
- Distributed SQL Query Engine
- Powerful API and CLI
- Cloud and/or on premise
- Flexible deployments
- Bridge clouds to/from on premise
- Retain control