Confluent has unveiled Kora, a cloud-native data engine powering its Apache Kafka® service. This cutting-edge engine is not available for download or to run outside of Confluent’s control, but understanding its inner workings reveals the advantages it offers and how it enhances Kafka service at scale.
While Kora does not replace open-source Kafka for those who prefer managing their own system, it enables Confluent to offer a cloud service at a phenomenal scale. Below are the reasons behind its creation, the benefits it provides, and a glimpse into how it operates.
Why Did We Build Kora?
When Confluent launched its cloud service in 2017, the vision was to offer Kafka in the cloud in full. However, initially, this service was limited to a version of open-source Kafka on a Kubernetes-based control platform, with basic billing, observability, and operational controls. Although it was the best Kafka offering at the time, it still fell short of the original vision.
It became clear that to fulfill that vision, merely running Kafka on AWS servers would not suffice. Successful cloud systems are vastly different in their architecture, operation, and customer experience. Kora was designed to overcome these limitations.
Differences Between Cloud and Self-Managed Systems
Cloud data systems face different challenges than self-managed open-source systems. Apache Kafka is designed to be easy to set up and run with various DevOps tools, providing high performance in any environment, annual updates, and operated by generalist staff.
In contrast, a cloud system must serve thousands of customers, be multi-tenant from inception, be primarily operated by software, offer robust isolation and security, and enable rapid innovations by a large team.
Capabilities of Kora
To meet these requirements, Confluent developed Kora, a data engine that is 100% compatible with all supported versions of the Kafka protocol, but designed as a truly managed service. Some of its capabilities include:
– Multitenancy: Supports thousands of tenants with strong isolation.
– Global Deployment: Operates in over 85 regions across three clouds.
– Operational Efficiency: Operated at scale by a small team, being approximately 1000 times more efficient than an average Kafka team.
– Component Disaggregation: Handles networking, computing, metadata, and storage separately.
– Intelligent Data Management: Optimizes data location between memory, SSDs, and cloud storage to maximize performance.
– Cost Optimization: Reduces infrastructure costs through automation and efficient resource management.
Benefits for Customers
Kora not only enhances Confluent’s operations but also offers significant benefits to its customers, such as:
– Elasticity: Enables scaling up 30 times faster.
– Reliability: Over 10 times higher availability, ensuring a 99.99% SLA.
– Performance: Lower latency compared to self-managed Kafka on similar hardware.
– Cost: Optimized cost structure saving money for customers.
– Compatibility: Fully compatible with the Kafka protocol.
Automated Operations and Continuous Updates
One of the key features of Kora is its ability to be primarily operated by software, reducing the need for human intervention. This includes continuous performance checks and data correction, as well as automatic detection and mitigation of infrastructure issues.
Kora is also continuously updated, meaning customers always have access to the latest improvements and security patches without the need for manual updates.
In summary, Kora represents a significant advancement in how Confluent offers Apache Kafka in the cloud, providing a robust, efficient, and secure platform for its customers’ critical workloads.
For more information, visit the Confluent blog.