vLLM: PagedAttention for 24x Faster LLM Inference

A more efficient way to compute Transformer’s attention during inference

PagedAttention for a prompt “the cat is sleeping in the kitchen and the dog is”. Key-Value pairs of tensors for attention computation are stored in virtual contiguous blocks mapped to non-contiguous blocks in the GPU memory. — Image by the author

Almost all the large language models (LLM) rely on the Transformer neural architecture. While this architectu…