Microsoft’s Project Zipline compression algorithm can achieve up to 2x higher compression ratios compared to the commonly used Zlib-L4 64KB model
The amount of data that the big cloud computing providers are processing is growing exponentially, and the growth is seemingly never-ending.
To enable Azure cloud to store and process data efficiently, Microsoft developed a cutting-edge compression algorithm and optimized the hardware implementation for the types of data it sees in their cloud storage workloads.
Microsoft said this enables them to simultaneously achieve higher compression ratios, higher throughput, and lower latency than the other algorithms that are currently available.
Now, the tech giant has open sourced the algorithm, hardware specification and Verilog source code for register transfer language (RTL). Microsoft is sharing all of this through the Open Compute Project (OCP).
Project Zipline compression algorithm
Project Zipline, as Microsoft calls this project, can achieve up to 2X high compression ratios compared to the commonly used Zlib-L4 64KB model.
“Contributing RTL at this level of detail as open source to OCP is industry leading,” Kushagra Vaid, General Manager, Azure Hardware Infrastructure, wrote in a blog.
He continued, “It sets a new precedent for driving frictionless collaboration in the OCP ecosystem for new technologies and opening the doors for hardware innovation at the silicon level. Over time, we anticipate Project Zipline compression technology will make its way into several market segments and usage models such as network data processing, smart SSDs, archival systems, cloud appliances, general purpose microprocessor, IoT, and edge devices.”
According to Vaid, the release of RTL allows hardware vendors to use the reference design to produce hardware chips to allow the highest compression, lowest cost, and lowest power out of the algorithm.
Microsoft is partnering with others in the Open Compute Project on Project Zipline. Among these partners are Intel, AMD, Ampere, Arm, Marvell, SiFive, Broadcom, Fungible, Mellanox, NGD Systems, Pure Storage, Synopsys and Cadence.
