LinkedIn has announced the development of a new load balancing tool called Cruise Control. This new development promises to keep Kafka clusters up and running by auto-configuring them using a set of predefined rules.
Announced at the the Kafka Summit in San Francisco, Calif., Cruise Control is a development of a dedicated team at LinkedIn that wanted to recognise the breaking of Kafka clusters and auto configure the broken ones to keep the entire stack running. The team started working on the tool last year.
Microsoft-owned LinkedIn built Kafka as its an open source message streaming engine. As TechCrunch reports, the engine presently handles over 2 trillion transactions per day and uses dedicated 1,800 servers. This is the reason LinkedIn’s tech team has opted the new tool.
Prior to the development of the latest tool, LinkedIn engineers had to reconfigure Kafka clusters manually. The cascading impact across the clusters was a tricky proposition as if one cluster was configured incorrectly. The engineering team, thus, wanted to put a machine in charge of cluster management along with human oversight.
Cruise Control identifies common configurations and applies a set of goals to each one. The tool can assess needs of the cluster, cross-check it with common configurations and set goals to choose the correct ones.
The team at LinkedIn took six months to deliver Cruise Control that can apply load balancing principles in Kafka.
