Kafka’s move away from ZooKeeper was driven by more than a desire to simplify deployment. As Kafka evolved from a message queue into a large-scale streaming platform, its dependence on an external coordination system began to constrain metadata performance, cluster growth, and operational efficiency. KRaft, short for Kafka Raft Metadata Quorum, is Kafka’s answer: manage metadata through Kafka’s own Raft-based controller quorum instead of relying on ZooKeeper.

This change does not mean that ZooKeeper was a poor choice in Kafka’s early years. It provided a practical way to handle metadata, broker coordination, and controller elections. The problem is that Kafka’s scale and workload eventually outgrew the assumptions behind that design.

What ZooKeeper originally did for Kafka

In Kafka’s earlier architecture, ZooKeeper was responsible for several essential parts of cluster management:

• Metadata storage: It recorded information such as topic partitions, replica placement, and the broker list.
• Cluster coordination: It tracked broker arrivals and departures and helped coordinate partition replica rebalancing.
• Controller election: When the Kafka controller failed, ZooKeeper was used to select a replacement. The controller is the broker responsible for coordinating tasks such as partition leader elections.

This arrangement worked well enough for smaller clusters. However, a Kafka deployment with hundreds of brokers and tens of thousands—or even more—partitions creates a much heavier metadata workload. Under those conditions, ZooKeeper increasingly became part of the bottleneck.

Where the old architecture began to struggle

Metadata operations were not a natural fit for ZooKeeper

ZooKeeper is fundamentally a distributed coordination service with strong consistency, not a metadata store designed for the high-throughput, rapidly changing state of a large Kafka cluster.

Every write must pass through the ZooKeeper leader and be replicated to followers. In a typical cluster, write capacity is often in the range of roughly 1,000 to 2,000 operations per second. That may be sufficient for coordination tasks, but Kafka can generate far more metadata activity in large environments—for example, when topics are created frequently, partition counts are changed, brokers join or leave, or replica states are updated. As the update rate rises, operations that once completed in milliseconds can take seconds.

The amount of metadata also grows with the number of partitions. Kafka stores many ZooKeeper nodes for each partition, including structures similar to /brokers/topics/[topic]/partitions/[p]/state. With hundreds of thousands or millions of partitions, the memory and disk pressure on ZooKeeper becomes substantial. The node structure itself can become a limitation; a single ZooKeeper node is commonly configured to hold around 100,000 nodes by default, creating another constraint for very large deployments.

There is also communication overhead. Kafka brokers must maintain heartbeats and synchronize metadata with ZooKeeper. As the number of brokers and partitions increases, the latency and bandwidth consumed by this cross-system communication become harder to ignore.

Operating two systems increased complexity

A ZooKeeper-based Kafka deployment required administrators to operate two independent distributed systems rather than one.

ZooKeeper normally needed its own cluster, typically with at least three nodes. Its network connectivity, quorum configuration, data directories, ports, and performance characteristics all had to be planned alongside the Kafka brokers.

Troubleshooting also involved a longer chain of dependencies. A problem such as unavailable partitions or a failed controller election might originate in Kafka, in ZooKeeper, or in the connection between them. Engineers often had to correlate Kafka logs with ZooKeeper logs before they could identify the actual failure.

Version management added another layer of work. Upgrading ZooKeeper required checking its compatibility with the Kafka version in use, which made maintenance more complicated than upgrading a single platform.

Cluster growth exposed hard limits

Kafka’s appeal lies partly in its ability to handle high throughput while scaling across brokers and partitions. ZooKeeper’s design made that expansion increasingly difficult.

The number of ZooKeeper nodes required for partition metadata placed a practical limit on Kafka’s total partition count. Modern use cases such as real-time data warehouses and centralized logging platforms may require hundreds of thousands or even millions of partitions, far beyond the scale for which the original arrangement was comfortable.

Controller failover was another concern. When the controller went down, Kafka relied on ZooKeeper’s ephemeral-node competition mechanism to elect a new one. The process involved multiple ZooKeeper reads and writes and could take around 10 to 30 seconds in a large cluster. During that interval, partition leaders could not be elected normally, affecting availability.

Rebalancing could be similarly slow. When a broker was removed or a new broker was added, Kafka had to synchronize replica information through ZooKeeper and then coordinate the rebalance. With a large number of partitions, ZooKeeper read and write latency could stretch the process from seconds to minutes.

KRaft brings metadata management into Kafka

Kafka introduced KRaft in version 2.8 as a way to remove this external dependency. Instead of using ZooKeeper, Kafka uses its own implementation of the Raft consensus protocol to store and replicate cluster metadata.

The architecture separates responsibilities between two types of nodes:

• Controller nodes: Usually deployed as a Raft quorum of three to five nodes, they store metadata, maintain consistency, and manage controller-related decisions.
• Broker nodes: They handle message production and consumption and obtain cluster metadata by synchronizing directly with the controllers.

Metadata is kept in a local log on the controller nodes, using a model similar to Kafka’s own message logs. Raft provides the mechanism for replicating that log and maintaining consistency among controllers.

Higher metadata throughput

Because metadata is managed through Kafka’s own log-based controller layer, it is no longer constrained by ZooKeeper’s coordination-oriented write path. KRaft is designed to support more than 100,000 metadata operations per second in large-scale scenarios, including partition creation and replica-state updates—more than an order of magnitude above the earlier model.

The log-stream approach is also a better fit for high-volume metadata changes. It uses a more compact representation than the large collection of ZooKeeper nodes and can grow with the controller quorum. Brokers communicate with the controllers instead of maintaining a separate metadata relationship with ZooKeeper, reducing cross-system traffic and operational overhead.

A simpler deployment and troubleshooting model

Once ZooKeeper is removed, a Kafka environment only needs Kafka nodes, divided into controllers and brokers. There is no separate ZooKeeper quorum to configure or maintain.

The deployment process becomes shorter because administrators no longer need to prepare ZooKeeper ports, quorum settings, data directories, and related network paths. The original architecture required configuration for both systems; removing one of them is often described as reducing deployment work by about 50 percent.

Troubleshooting is more direct as well. Metadata-related failures can generally be investigated through Kafka’s own logs and controller state instead of requiring a comparison of two separate systems. Upgrades also focus on Kafka itself, reducing cross-product compatibility concerns.

Better support for very large clusters

KRaft removes the old dependency on ZooKeeper’s node-based metadata layout and is designed to support Kafka clusters with millions of partitions in theory. Its more compact binary metadata format reduces memory consumption, while the controller quorum provides a dedicated place to manage cluster state.

Controller election is handled directly through Raft rather than through ZooKeeper’s ephemeral-node mechanism. In the intended design, recovery can drop from seconds to milliseconds, commonly to less than one second under suitable conditions.

Rebalancing also benefits from the simplified path. Controller nodes coordinate the relevant metadata directly, without ZooKeeper acting as an intermediate system. For large clusters, operations that could previously take minutes are expected to move toward the seconds range, depending on the cluster and the amount of state being changed.

A more dependable metadata consistency model

KRaft uses Raft to maintain a strongly consistent metadata log. This makes metadata visibility more predictable and reduces the chance that brokers will act on different views of cluster state. In turn, it helps limit problems associated with inconsistent metadata, such as unavailable partitions or incorrect replica information.

The key improvement is not simply that KRaft uses a different consensus algorithm. It is that Kafka’s metadata model is now designed around Kafka’s own operational needs, rather than being represented indirectly through an external coordination service.

The migration from ZooKeeper was gradual

Kafka did not remove ZooKeeper support overnight. The transition was deliberately staged.

Starting with Kafka 2.8, both the traditional ZooKeeper mode and KRaft mode were available, allowing users to choose based on their operational requirements and the maturity of the features they needed.

Early KRaft releases, particularly versions in the 2.8 to 3.0 range, still lacked some capabilities, including certain security-related features. By the 3.3 series and later, most of the important gaps had been filled, making KRaft suitable for the majority of production scenarios.

Migration tooling was also provided. The kafka-metadata-migrate tool supports moving from ZooKeeper mode to KRaft mode while synchronizing metadata during the transition, making it possible to avoid rebuilding a cluster from scratch.

Kafka’s decision to leave ZooKeeper behind is therefore best understood as an architectural adjustment to scale. ZooKeeper helped Kafka get off the ground by providing a ready-made coordination layer, but the same dependency became increasingly restrictive as Kafka took on larger workloads and more partitions.

KRaft allows Kafka to manage its own metadata through a controller quorum built around Raft. The result is a less fragmented architecture, higher metadata throughput, faster recovery, and a path toward much larger clusters. Kafka did not abandon ZooKeeper because it had no value; it outgrew the role that ZooKeeper had been designed to play.