Nvidia introduces KVTC to slash LLM memory by 20x and speed responses, enabling efficient deployment of open models without retraining or architectural changes.
Nvidia has introduced KV Cache Transform Coding (KVTC), a new technique that reduces large language model (LLM) memory usage by up to 20x without modifying model weights or architecture, directly strengthening open-source AI infrastructure.
The method delivers up to 8x faster time-to-first-token (TTFT) while maintaining less than 1% accuracy loss, even sustaining strong performance at extreme compression levels of 32x–64x. KVTC addresses a critical bottleneck in long-context AI systems, where KV cache memory can scale to multiple gigabytes, limiting GPU scalability, increasing latency, and driving infrastructure costs.
By significantly lowering memory demands, KVTC reduces GPU costs, improves prompt reuse, and enables faster multi-turn AI interactions without recomputing full conversation histories.
The approach borrows from JPEG-style media compression, combining PCA-based feature reduction, dynamic precision allocation, and DEFLATE entropy coding accelerated via Nvidia’s nvCOMP library. Compression runs between inference phases, avoiding runtime slowdowns.
“Effective KV cache management becomes critical, as idle caches must be quickly offloaded from GPU memory to accommodate other users, and quickly restored for resumed conversations,” said Adrian Lancucki. He added, “This ‘media compression’ approach is advantageous for enterprise deployment because it is non-intrusive: it requires no changes to model weights or code.”
KVTC outperforms existing methods such as KIVI, GEAR, H2O, and TOVA, which degrade beyond ~5x compression. In tests, a Qwen 2.5 1.5B model reduced memory from 29 KB to 3.2 KB per token with just a 0.3% accuracy drop.
The technique will integrate into Nvidia Dynamo’s KV Block Manager and supports open ecosystems via vLLM, signalling a move towards a standardised compression layer for scalable AI.
