AirLLM Runs 70B Models on a Single 4GB GPU

> **TL;DR:** AirLLM is an open-source inference library that claims to run 70-billion-parameter models on a single 4GB GPU without quantization, by streaming the model one transformer layer at a time so only a slice of the weights sits in VRAM. It reached 175 points on Hacker News while simultaneously taking the No. 1 daily slot on GitHub trending. The trade-off is speed, not quality: memory stops being the hard ceiling and bandwidth becomes the bottleneck instead.

Key Takeaways

- AirLLM claims 70B-parameter inference on a single 4GB GPU — no quantization, original weights at original precision. - It works by layered offloading: load one transformer block into VRAM, compute, release, load the next. - Peak memory drops from 'the whole model' to 'the largest single layer plus activations and KV cache.' - It hit 175 points on Hacker News and No. 1 daily on GitHub trending at the same time — a rare double signal for local-inference tooling. - The cost is throughput. This suits batch and offline work, not low-latency chat.

A 70B model does not fit in 4GB — that is the whole point

Seventy billion parameters at 16-bit precision is roughly 140GB of weights. No amount of driver tuning squeezes that into a 4GB consumer GPU, and [AirLLM](https://github.com/lyogavin/airllm) is not claiming it does. Its claim is narrower and considerably more interesting: at any single instant, a transformer only needs one layer's weights sitting on the GPU. Everything else can wait somewhere cheaper.

That reframing is why the project took the No. 1 daily slot on GitHub trending while simultaneously pulling 175 points on [Hacker News](https://news.ycombinator.com/item?id=49154228) — a combination that local-inference tooling rarely lands on the same day.


How layered offloading works

A large language model is a stack of near-identical transformer blocks executed in strict sequence. Block 12 cannot run until block 11 has produced its output — and once block 11 is finished, its weights are dead weight for the remainder of that forward pass.

Standard practice loads the entire stack into VRAM anyway, because keeping weights resident is dramatically faster than shuttling them in and out. AirLLM takes the opposite bet. It streams the model layer by layer: load a block's weights into VRAM, push the activations through it, release the memory, load the next block. Peak memory collapses from *the whole model* to *the largest single layer, plus activations and KV cache*.

The consequence is structural. VRAM stops being the hard gate on **which models you can run at all**, and becomes a throughput parameter instead. A 4GB card no longer means "7B or bust" — it means "70B, eventually."

Why "without quantization" is the load-bearing phrase

The conventional route onto consumer hardware is quantization: crushing weights from 16-bit down to 4-bit or lower. It works, it is everywhere, and it is lossy in ways that are genuinely hard to predict. Degradation tends to surface in long-context reasoning, instruction-following edge cases and rare-token handling — precisely the places where benchmark averages hide it.

AirLLM's pitch is that layered offloading skips that bargain entirely. You run the original weights at their original precision. You simply run them slowly.

For anyone evaluating whether an open 70B model is actually good enough for a production task, that distinction is worth real money. It eliminates the most annoying ambiguity in local-model evaluation: *was that failure a model limitation, or a quantization artifact?* Test at full precision first, quantize afterwards if the quality budget allows.

The trade-off is bandwidth, not memory

There is no free lunch here, and it is worth sizing the cost honestly before cloning anything.

Streaming weights means every forward pass drags gigabytes across the PCIe bus — and if the model does not fit in system RAM, off the SSD as well. That is a bandwidth-bound workload, and bandwidth-bound workloads are slow in a way no GPU upgrade fixes. Real throughput will swing hard on storage speed, RAM capacity and PCIe generation, so treat any single figure you see quoted as hardware-specific. We are not going to publish numbers we have not measured; the project's own repository is the place to check current benchmarks and supported model families.

The practical read: this is a technique for workloads where latency is negotiable. Overnight evaluation runs, batch document processing, offline synthesis, single-user experimentation, capability testing before you commit to an API bill. It is not a drop-in replacement for a hosted endpoint sitting behind a chat window.

The contrast with where hosted AI is heading is stark. Frontier providers are optimising in the opposite direction — [OpenAI's continuous-voice GPT Live](https://speka.info/blog/openai-ships-gpt-live-continuous-voice-for-devs) is engineered around latency budgets measured in milliseconds. AirLLM trades away exactly that dimension to buy something the cloud will not sell you: unmetered access to full-precision open weights on hardware you already own.


Why the double signal matters

Hacker News points and GitHub trending measure different appetites. Upvotes register *this is interesting* — an argument, a claim, a technique worth debating. A No. 1 daily trending slot registers *I want this on my machine right now*, because stars accrue from people who came to install, not merely to read.

Projects routinely get one without the other. Clever research papers rack up discussion and no stars. Utility libraries accumulate stars quietly with no discourse at all. Hitting both simultaneously usually means a project has landed on a real, widely-felt constraint — and VRAM has been the single most-felt constraint in local AI for years.

It is also a small counterweight to the assumption baked into most long-run AI economics — including [the abundance arguments about AI-driven income](https://speka.info/blog/universal-high-income-musks-2036-ai-money-claim) — that serious inference must concentrate in a handful of datacentres. Tools that push frontier-scale models onto laptops push the other way. Running open weights on your own hardware also quietly sidesteps a set of questions that keep resurfacing around closed frontier models, including [how their training corpora were assembled](https://speka.info/blog/anthropics-project-panama-books-destroyed-to-train-claude).

How to evaluate it before you commit

If you are testing AirLLM, measure the things that determine whether it is usable for your workload rather than the things that make a good headline:

Measure time-to-first-token separately from throughput

Layer streaming front-loads work. The two numbers will diverge much more than they do on a fully-resident model.

Confirm whether you are disk-bound or RAM-bound

If the model fits in system RAM, you are paying PCIe transfer costs. If it does not, you are paying SSD read costs on every pass — a different order of magnitude. This single fact will dominate your experience.

Test at batch size 1, because that is the real use case

The scenarios that justify this approach are overwhelmingly single-user. Benchmarks that batch aggressively will flatter the setup in ways your actual usage will not reproduce.

Verify quality against the claim

The entire premise is no quantization loss. Run your own evaluation set against the same model served conventionally and confirm the outputs match. A claim this specific should be trivially falsifiable — so falsify it before you build on it.

For more coverage of open-source tooling that shifts what runs on local hardware, see our [New AI Tools & Skills](https://speka.info/new-ai-tools/) section.

Frequently Asked Questions

Can AirLLM really run a 70B model on a 4GB GPU?

That is the project's claim, achieved by loading and executing the model one transformer layer at a time so the full weight set never needs to be resident in VRAM. The full model still has to live somewhere — system RAM or disk — and is streamed to the GPU during inference.

Does AirLLM use quantization?

No. Avoiding quantization is the central claim: the model runs at its original precision, so there is no compression-driven quality loss. The saving comes from memory scheduling rather than from shrinking the weights.

Is AirLLM fast?

It trades speed for capability. Because weights are streamed for every forward pass, throughput is bounded by PCIe and storage bandwidth rather than GPU compute, so expect it to be far slower than a fully-resident model.

What hardware do I actually need?

A GPU with enough VRAM for a single layer plus activations and KV cache, and enough system RAM or fast SSD storage to hold the full model. Storage and memory bandwidth will matter more to your experience than the GPU itself.

How is this different from ordinary CPU offloading?

Offloading part of a model to CPU or disk is an established technique. What drew attention here is how far AirLLM pushes the ratio — a 4GB VRAM budget against a 70B model — while keeping full precision rather than quantizing to fit.

Where can I get AirLLM?

The project is open source and published on GitHub at github.com/lyogavin/airllm, where the repository documents supported models, installation and current benchmarks.

Sources

- https://github.com/lyogavin/airllm - https://news.ycombinator.com/item?id=49154228