vLLM
This article coincides in timing and pattern with contributions by the same user which have been determined to be AI-generated, and has no significant contributions by other editors. It is presumed to have also been AI-generated, and may be assumed to violate Wikipedia's policies and guidelines. If no editor takes responsibility for it within a five-day grace period, this article may be deleted. To object to proposed deletion, an editor may take responsibility for the content and assume the onus to achieve consensus for its inclusion. Assuming responsibility requires that all content and its sources be closely reviewed and reworked as needed to ensure that they comply with polices and guidelines. This may involve checking that the cited sources support the content and are reliable, determining whether the content is due for inclusion, and rewriting the content to revise any tone issues and resolve possible copyright violations. Once this has been done, this template may be removed. The nominator also gave the following reason for this proposed deletion:
Find sources: "VLLM" – news · newspapers · books · scholar · JSTOR Reviewer tools: guideline project (talk • log) The article may be deleted if this message remains in place for five days, i.e., after 12:27, 30 October 2026 (UTC). |
| vLLM | |
|---|---|
 | |
| Original authors | Sky Computing Lab, University of California, Berkeley |
| Developer | vLLM contributors |
| Release | 2023 |
| Stable release | |
| Written in | Python, CUDA, C++, Rust |
| Type | Large language model inference engine |
| License | Apache License 2.0 |
| Website | vllm |
| Repository | |
vLLM is an open-source software framework for inference and serving of large language models and related multimodal models. Originally developed at the University of California, Berkeley's Sky Computing Lab,[2] the project is centered on PagedAttention, a memory-management method for transformer key–value caches, and supports features such as continuous batching, distributed inference, quantization, and OpenAI-compatible APIs.[3][4][5]
History
[edit]vLLM was introduced in 2023 by researchers affiliated with the Sky Computing Lab at UC Berkeley.[4][3] Its core ideas were described in the 2023 paper Efficient Memory Management for Large Language Model Serving with PagedAttention, which presented the system as a high-throughput and memory-efficient serving engine for large language models.[4]
According to a project maintainer, the "v" in vLLM originally referred to "virtual", inspired by virtual memory.[6]
PyTorch's project page states that the University of California, Berkeley contributed vLLM to the Linux Foundation in July 2024.[7][5] In 2025, the PyTorch Foundation announced that vLLM had become a Foundation-hosted project.
In January 2026, TechCrunch reported that the creators of vLLM had launched the startup Inferact to commercialize the project, raising $150 million in seed funding.[8]
Architecture
[edit]According to its 2023 paper, vLLM was designed to improve the efficiency of large language model serving by reducing memory waste in the key–value cache used during transformer inference.[4] The paper introduced PagedAttention, an algorithm inspired by virtual memory and paging techniques in operating systems, and described vLLM as using block-level memory management and request scheduling to increase throughput while maintaining similar latency.[4]
The project documentation and repository describe support for continuous batching, chunked prefill, speculative decoding, prefix caching, quantization, and multiple forms of distributed inference.[3][5] PyTorch has described vLLM as a high-throughput, memory-efficient inference and serving engine that supports a range of hardware back ends, including NVIDIA and AMD GPUs, Google TPUs, AWS Trainium, and Intel processors.[7][5]
See also
[edit]- llama.cpp
- SGLang
- TensorRT-LLM
- OpenVINO
- Open Neural Network Exchange
- Comparison of deep learning software
- Comparison of machine learning software
- List of software developed at universities
- Lists of open-source artificial intelligence software
- List of large language models
- TurboQuant — online vector quantization algorithm
- 2025–present global memory supply shortage
References
[edit]- ↑ "Release 0.32.0". October 18, 2026. Retrieved October 18, 2026.
- ↑ "vLLM - A High-Throughput and Memory-Efficient Inference and Serving Engine for LLMs". UC Berkeley, Sky Computing Lab.
- 1 2 3 "GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs". GitHub. GitHub, Inc. Retrieved April 22, 2026.
- 1 2 3 4 5 Kwon, Woosuk; Li, Zhuohan; Zhuang, Siyuan; Sheng, Ying; Zheng, Lianmin; Yu, Cody Hao; Gonzalez, Joseph E.; Zhang, Hao; Stoica, Ion (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles. Retrieved April 22, 2026.
- 1 2 3 4 "vLLM". PyTorch. PyTorch Foundation. Retrieved April 22, 2026.
- ↑ "vLLM full name". GitHub. GitHub, Inc. August 23, 2023. Retrieved April 22, 2026.
- 1 2 "PyTorch Foundation Welcomes vLLM as a Hosted Project". PyTorch. PyTorch Foundation. May 7, 2025. Retrieved April 22, 2026.
- ↑ Temkin, Marina (January 22, 2026). "Inference startup Inferact lands $150M to commercialize vLLM". TechCrunch. Retrieved April 22, 2026.
External links
[edit]Large language models (LLMs) | |
|---|---|
| Concepts |
|
| Training, prompting, and alignment | |
| Models | |
| Chatbots and assistants | |
| Agents, coding, and applications | |
| Software | |
| Hardware and infrastructure | |
| Benchmarks, evaluation, and detection | |
| Datasets and data | |
| Organizations | |
| People | |
| Social, economic, and governance | |
