Kubernetes may be a complex software that is difficult to monitor and manage. But the benefits of running applications may outweigh the disadvantages due to migration of big data apps to Kubernetes is widespread, according to a new survey from Pepperdata.
In a report on the state of big data on Kubernetes, Pepperdata found that more than half of the folks it surveyed say they are “moving big data applications to Kubernetes to reduce overall spending.”
The research firm had surveyed 600 IT and big data professionals from various industries to determine which big data applications enterprises are moving or intending to move to Kubernetes. The study intended to identify the challenges these companies are facing, what applications they are migrating, and how fast the migration is taking place. The vast majority of the participants were employed by companies with between 500 and 5,000 workers.
Kubernetes is the preferred container orchestration technology for its agility, speed and efficiency for scaling and managing apps and infrastructure. Because of this, 77 percent of respondents signaled their intent to migrate 50 percent or more of their big data applications to containers with Kubernetes by the end of the year.
Kubernetes is the preferred container orchestration technology for its agility, speed and efficiency for scaling and managing apps and infrastructure. Because of this, 77 percent of respondents signaled their intent to migrate 50 percent or more of their big data applications to containers with Kubernetes by the end of the year. Applications that topped the list of those moving to Kubernetes include Spark, which was the most popular with 63 percent of the votes. Presto took 2nd with 45 percent, Kafka came in at 42 percent, and Trino scored 34 percent. Flink followed up with 27 percent.
“The velocity of the shift to Kubernetes is surprising. It’s gratifying to see big data move to Kubernetes with this speed. It’s clear that this shift will create optimization challenges as customers understand the nuances of running within the Kubernetes environment,” said Ash Munshi, CEO, Pepperdata.
Applications that topped the list of those moving to Kubernetes include Spark, which was the most popular with 63 percent of the votes. Presto took 2nd with 45 percent, Kafka came in at 42 percent, and Trino scored 34 percent. Flink followed up with 27 percent.
However, despite the lack of visibility most Kubernetes monitoring tools offer on the market, over 42% of our survey respondents rely on solutions offered by their cloud providers. Nearly 15% report that they are still searching for a monitoring solution even after migrating a significant amount of their big data workloads to Kubernetes.
Big data experts predict more Kubernetes in action in the immediate future. The need for big data application monitoring in Kubernetes can’t be understated in the face of the projected increase in usage.
While there are a number of free and off-the-shelf K8s monitoring tools today, very few allow you to step beyond the basic K8s measurements and look at app performance. You need to be able to do both, otherwise, you lack two key things. You either don’t have the full picture—which makes troubleshooting and meeting your SLAs tough, or you have to use a number of different tools—which is a pain. The hard part of K8s is monitoring and tweaking it to get the best performance from your clusters, the report noted.
