Kafka vs Pulsar: Why Pulsar Outperforms Kafka Every Time

Apache Kafka and Pulsar both are used within the Hadoop ecosystem, as well as hundreds of other ecosystems, for event processing in real-time data streams for “Big Data” applications operating at the highest levels of web/mobile traffic requirements. In benchmark testing conducted by independent researchers using tools from the OpenMessaging Project, Pulsar outperforms Kafka every time by large margins on the same hardware when processing the same code. The key element assessed by these tests is the latency in the messaging system measured in the amount of time it takes to integrate the individual packet data with the database, storage, and code of the main software application.

Benchmarking Kafka & Pulsar With OpenMessaging Tools

Developers cite the underlying architecture as the biggest reason for the performance differences between Apache Kafka and Pulsar in production at scale. When benchmarking I/O processes on datacenter hardware, the software sets parameters for the number of topics served, the average size of the messages, the number of subscriptions, the rate of production, and the number of messages served per second. In this manner, the support of the hardware for the required software can be ascertained by benchmark testing on Apache Kafka and Pulsar equipment to review the performance for code, database, and storage integration under various rates of simulated or real web traffic.

These results have been confirmed across repeated benchmark testing with many enterprise software packages and database technologies. Apache Pulsar always beats Kafka in benchmark testing because of the unique benefits of the architecture for stream processing of events and messages. On a fundamental level, Pulsar uses an index for message queues that outperforms the log system implemented by Kafka. This includes a dedicated data layer for messaging in Pulsar that is based on Apache BookKeeper.

In summarizing the extensive benchmarking for Apache Kafka and Pulsar completed by third-party researchers using the OpenMessaging framework:

• Both platforms will support 2 million writes per second, however, Kafka requires additional resources/hardware to make this happen.
• Apache Pulsar provides up to 150% higher maximum throughput on the same partitions.
• Events queues run on Apache Pulsar show up to 40% lower message latency results.

A detailed report on these benchmark tests can be found at the Kafkaesque and GigaOm websites. While Apache Kafka currently enjoys widespread market adoption in enterprise production environments for “Big Data” requirements, this market position is being challenged and supplemented by new Apache Pulsar solutions which offer better performance.

Other factors that contribute to the performance differences between Apache Kafka and Pulsar are related to the cloud use of the software. DevOps technicians report greater problems changing the configurations for partitions using Kafka and rebalancing clusters already in production. Kafka systems require the use of Apache Storm, Heron, or Spark for analytics. In contrast, Pulsar works with the Apache BookKeeper and Zookeeper for better processing of message queues. Pulsar developers can integrate with NoSQL database solutions like Cassandra or running Kafka services already installed to improve services. AWS Kinesis, Elastic search, Redis, Mongo DB, Influx DB are other popular services for metrics, analytics, and automation that developers build around to improve performance speeds with Pulsar.

Compared to Kafka, Apache Pulsar Excels

Apache Pulsar processes data streams for events and messages at rates that are 250% faster than Kafka in benchmark tests. Pulsar displays 40% less latency than Kafka in running the same services for websites and mobile applications. Pulsar works well with containerized systems like Kubernetes and AWS EC2 for high-traffic ecommerce websites and social networks. DevOps technicians prefer Apache Pulsar for better geo-location in data center management, multi-tenancy in operations, TLS-authorization in encryption support, and service mesh oriented architecture for Infrastructure-as-Code solutions. Apache Pulsar can also be run through serverless and PaaS products to permit quick enterprise uptake to web/mobile applications already in production to improve the speed and performance of cloud services.