FlexKV: Tencent AI's Approach to Faster Long-Context Cache Hits

Tencent AI ·

Key Info

Tencent AI's FlexKV targets a bottleneck in long-context serving: even when a KV cache hits, loading it from outside GPU memory can leave the GPU idle. It restores the cache layer by layer, overlapping computation with data movement.

Highlights

  • FlexKV restores the KV cache layer by layer: earlier layers compute while later layers load, reducing GPU wait time.
  • Prefetching starts cache transfers early, and asynchronous writeback helps overlap cache I/O with inference.
  • The work focuses on improving the real-world value of cache hits in long-context workloads, where KV cache demand is growing rapidly.
Loading...