In my previous post, Kafka Metadata: An Extended Introduction, we discussed Kafka metadata and what it stores. We also discussed how this metadata evolved to be managed by different systems throughout the life of Kafka: first by Zookeeper, then later by KRaft. Later, we explored the underlying structure of each of these systems. If you have not read the blog post, it is recommended (but not required) before moving forward.
In this blog post, we will expand on this knowledge to explore a multi-stage migration from Zookeeper to KRaft. Originally documented to assist with Ansible-based production migrations from Zookeeper to KRaft, these migration steps have been documented and tested with a safety-first mentality to protect Kafka clusters.
Migrating from Zookeeper to KRaft
Migrating metadata from Zookeeper to KRaft can be extremely dangerous for the Kafka cluster, and extremely stressful for Kafka operators. If the migration is performed incorrectly, and without precautions, critical errors may arise within the cluster: topic configurations and partition assignments are vulnerable to corruption, for example.
Luckily, Confluent and the Kafka community have established a set of safety-first metadata migration paradigms. In this blog post, we will discuss “dual-write” mode, which is available through many different Kafka deployment models; in this case, Ansible will be used. During the “dual-write” migration process, the following steps are recommended:
1. Pre-Migration Checklist
This checklist is not officially recommended by Confluent or Kafka; instead, it was created through personal experience when performing several KRaft migrations through Ansible:
- In the cluster, verify there are no in-sync replica errors; this can cause the migration to fail due to healthcheck failures. All other health checks should also pass, but ISR errors are among the most common.
- In each KRaft node, verify the port (often 9093) is open for communication with other KRaft nodes and brokers.
- In each KRaft node, also verify the Kafka log directory is writable. KRaft manages its metadata through an internal topic that is written to this directory.
- Add references to KRaft (also called “kafka controller” or “kraft controller”) in your configuration; the method of adding these configurations will depend on your deployment model For example, these can be added through an Ansible inventory.
- Add a “zookeeper.metadata.migration.enable” or similar boolean flag in your broker and Kraft configurations; the method of adding this configuration will depend on your deployment model. For example, it can be added through an Ansible inventory.
- Configure the KRaft quorum, if required.. This is not required when using Ansible, since the migration job will generate the “controller.quorum.voters” property.
- DO NOT REMOVE ZOOKEEPER CONFIGURATIONS YET.
Ansible is used as an example deployment model, but the most important point (in any case) is that these configurations should be added to the “server.properties” or similar file of Kafka.
2. Migrate to “Dual-Write” Mode
After completing the pre-migration checklist, the next step is the migration to “dual-write” mode. In this mode, Zookeeper and KRaft are both running in tandem.
When new metadata needs to be persisted in “dual-write” mode, brokers will attempt to coordinate with KRaft for its storage. If the attempt is successful, KRaft will also forward the metadata change to Zookeeper. In this respect, Zookeeper acts as a passive synchronized backup of metadata. If the attempt is a failure, this indicates an error during the metadata migration process; Zookeeper is available as a rollback option.
Most deployment models support “dual-write” mode, including Confluent Platform and open source Kafka. In most cases, this will assign a “zookeeper.metadata.migration.enable” variable to your brokers and KRaft controllers until the full migration is complete. As an example, the migration to “dual-write” in Ansible may be performed with the following command:
ansible-playbook -i confluent.platform.ZKtoKraftMigration.yml --tags migrate_to_dual_write
Important: Tag the migration job with “migrate_to_dual_write” or similar. Without this flag, the job may run to completion and without pausing at “dual-write” for safety checks. Rollback to Zookeeper may not be possible in such a case.
3. Verify KRaft Quorum
With the cluster in “dual-write” mode, there is a great opportunity to check the health of KRaft nodes and their communication with the cluster. There are several options for checking the health of the quorum; for production clusters, all health checks should be performed for maximum safety – even if some may seem redundant.
1. Use the kafka-metadata-quorum command to verify the KRaft quorum is healthy. This command will output a list of each quorum member (KRaft node and/or broker), as well as their statuses and quorum roles of leader, follower, or observer. Based on the output, verify there is no consumer lag and there are a correct number hosts for each role. This command is universal to Apache Kafka brokers and KRaft controllers.
Example:
kafka-metadata-quorum --bootstrap-controller kraft-node-host:9093 --command-config /path/to/client.properties --replication
2. Verify the KRaft controller packages are installed on the KRaft nodes. Based on the output of this command, you can verify that your KRaft controllers are running on the same version as the rest of your cluster. The exact command will depend on your hosting environment.
Example (Red Hat + Confluent Platform):
rpm -qa | grep confluent
3. Verify the KRaft controller is running. If this command successfully prints the service as “active”, the controller is running on the system. Again, this exact command will depend on your hosting environment.
Example (Red Hat + Confluent Platform):
systemctl status -l confluent.kcontroller.service
4. Check the logs of the KRaft nodes for errors. Be careful to check for errors relating to metadata, watermarks, in-sync replicas, or the quorum. If there are any errors, they should be resolved before continuing the migration. Again, this exact command will depend on your hosting environment.
Example:
tail -n 1000 /var/log/controller/server.log
5. Check the “ZkMigrationState” metric for a status. There are several ways to do this, but Jolokia should work in most cases. This command will return a key-value object. One of the keys, “value”, needs to have a value of “1”.
curl -k http://localhost:7770/jolokia/read/kafka.controller:type=KafkaController,name=ZkMigrationState
6. Check the logs of the leader KRaft controller to determine if the final “Completed migration” message is printed. If the message cannot be found, be sure to check other KRaft nodes because the leader may have changed since migration. If you use Splunk or another distributed log manager, a search through its interface may be preferred.
Example:
grep “Completed migration” /var/log/controller/server.log
This is not an exhaustive list of safety steps to follow, and should be adjusted depending on your system. Additional checks could be added for verifying metadata is being written to both Zookeeper and KRaft, for example.
4. Remove Zookeeper
After the KRaft quorum and the cluster are verified to be running healthy in dual-write mode, the dependency on Zookeeper may be removed from the Kafka cluster. The migration flag, “zookeeper.metadata.migration.enable”, will also need to be removed or set to false.
If you are using Ansible with cp-ansible, the process is relatively simple. Using the same playbook that was executed for dual-write mode, the second phase of the migration can be performed by including the “migrate_to_kraft” flag. This job will tell your brokers and KRaft controllers to stop sending metadata to Zookeeper, and remove the migration flag.
ansible-playbook -i confluent.platform.ZKtoKraftMigration.yml --tags migrate_to_kraft
Note that the above command will not remove the allocated infrastructure for Zookeeper. For example, Zookeeper references may need to be manually removed from Ansible inventory and playbook files. EC2 or other cloud hosting instances for Zookeeper may need to be manually shut down. Related jobs, like startup and teardown scripts, may also require adjustment.
Before decommissioning Zookeeper infrastructure, which is not performed by the command above, it is important to verify one more time that the cluster is healthy using the options mentioned previously. At this point, a rollback to Zookeeper becomes extremely difficult and risky, potentially requiring data loss or complex recovery procedures. This step should be done as soon as possible.
Operators may want to consider a period of time where Zookeeper runs in an unused state, acting as a safety buffer in the case of a rollback. When the infrastructure is decommissioned, the historic Zookeeper metadata (stored in ZNodes) will be lost forever, making an already-difficult rollback nearly impossible.
After decommissioning Zookeeper infrastructure, it is good practice to check the state of the cluster to verify there were not any lingering dependencies. For example, a broker may not have had its migration flag removed, or missing properties required for KRaft (e.g. quorum, role). In most cases, the cluster should be healthy since dependencies were already removed.
Multi-Cluster & Multi-Region Migrations
The above migration steps can generally be performed with multi-cluster and mult-region systems, but there are issues of scale to consider. Operators will want to know how these clusters can be safely updated in coordination. Luckily, Kafka makes this problem relatively simple to solve.
Each cluster in a multi-cluster or multi-region configuration may be migrated independently to KRaft, and Kafka abstractions between these systems will handle the compatibility differences. Each cluster will only store and propagate its own metadata; only actual data (usually through topic messages) are communicated between clusters.
When migrating multiple clusters, operators may want to take the opportunity to stagger migrations in an attempt to learn (and not repeat) possible issues. This will allow for an incremental migration process with easier planning and maintenance.
Multi-region clusters have a special case of “stretched regional clusters”. These are single clusters that are stretched across several regions. In this case, the KRaft quorum must be maintained across regions at all times. Each region should have at least one KRaft controller for the best fault tolerance. During the migration, all health check steps should be performed in all regions.
Review
In this post, we reviewed an ideal multi-stage metadata migration from Zookeeper to KRaft. For maximum safety, a “dual-write” mode was used to write metadata to both systems. After KRaft was verified to be healthy, “dual-write” mode was disabled and Zookeeper was able to be removed from the system.
If you enjoyed this content, and like the topics that were discussed here, feel free to follow us on LinkedIn!
