Open-weight AI models have trailed the best closed models by an average of four months since January 2026, according to Epoch AI's May 29 analysis. A growing number of them now run on hardware many people already own, and two free programs—Ollama and LM Studio—handle the setup: small models need roughly 8GB of VRAM or Mac unified memory, and LM Studio can operate entirely offline.

The largest open models, such as Alibaba's 2.4-trillion-parameter Qwen 3.8-Max, sit far beyond consumer hardware, and closed models won 24 of 31 tests in Alibaba's own comparison. For everyday tasks, though, the gap has narrowed enough that a capable chatbot can run entirely on a personal computer—without the roughly $20-a-month subscription fee and without sending conversations to a company's servers.

"Open-weight" means the company publishes the weights—the files holding everything the model learned—so anyone can download and run them. Gemma 4 downloads in Ollama's library range from about 4.6GB to 20GB. This guide explains how to download and run AI models directly on your computer, what kind of hardware is required, and how to get started with free tools like Ollama and LM Studio—whether the goal is saving money, keeping conversations private, or experimenting with models you can customize yourself.

Why run an AI model on your own machine

A cloud chatbot sends every prompt to a company's servers, and a court ordered OpenAI to preserve ChatGPT logs, including deleted ones, as Decrypt reported in a review of privacy-focused AI tools. A local model keeps the conversation on the user's computer, where nobody can use it without permission.

LM Studio's documentation says the app "can operate entirely offline, just make sure to get some model files first," and that documents attached to a chat can be used entirely offline too.

Subscriptions to paid chatbots run about $20 per month, while the software covered here is free; the cost instead shifts to hardware the user may already own.

Licenses decide what can be done with downloaded weights. DeepSeek's R1 series is under the MIT License, and its model card says the models "support commercial use, allow for any modifications and derivative works," while Google's Gemma 4 and OpenAI's gpt-oss ship under Apache 2.0. Running models under open-source licenses like Apache or MIT means users are free to use, modify, and do whatever they want with a model—and to access improved or modified versions that better fit their needs. Someone who needs a model that is good at medicine, for instance, need not care how well it codes. That level of personalization is only available in customized models.

What your hardware can actually run

Models are sized in parameters—the internal numbers adjusted during training—and the download grows with them. The model has to fit in a graphics card's video memory, known as VRAM, or in a Mac's unified memory, which the processor and graphics chip share.

As a general rule, at least 8GB of VRAM is a sensible baseline, and models compressed to smaller sizes will fit. Serious users should consider a gaming GPU with 24GB of VRAM or more—either by stacking multiple GPUs or by buying dedicated AI racks or unified-memory machines like the Mac Mini or the Nvidia DGX Spark.

Quantization shrinks a model by storing each weight with fewer bits. Hugging Face's documentation describes it as representing weights and activations "with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32)," which cuts memory use at the cost of some accuracy.

LM Studio's system requirements recommend 16GB or more of RAM on Apple Silicon Macs running macOS 14 or newer, though 8GB Macs can handle smaller models. On Windows it recommends at least 16GB of RAM and 4GB of dedicated VRAM, with AVX2 processor support; Intel-based Macs are not supported.

Ollama's library lists Gemma 4 26B at 16 to 19GB and the 31B version at 19 to 20GB. OpenAI's gpt-oss-20b needs only 16GB of memory, and gpt-oss-120b runs within 80GB, according to OpenAI.

How to set up Ollama and LM Studio

LM Studio is probably the go-to choice for most users because it provides a polished graphical user interface. Ollama, by contrast, is aimed at developers and more experienced users who prefer working inside a terminal (CLI).

For LM Studio, download the installer for your operating system from lmstudio.ai, search for a model inside the app, download its weights and open a chat. The app searches Hugging Face, the public hub where open models are published.

For Ollama, macOS and Linux users install with the command curl -fsSL | sh, and Windows users run irm | iex in PowerShell—both listed on Ollama's download page.

Typing ollama pull gemma4:e2b downloads a model, and ollama run gemma4:e2b opens a chat; /bye exits, per Ollama's quickstart. The program also opens a local API at so other apps on the machine can use the model.

Model names ending in ":cloud," such as gemma4:cloud, are cloud models that run in Ollama's cloud and require no download—so they can be skipped if privacy is the point. That said, they are a reasonable second option for those who need more powerful models: calls and prompts handled over the API are not stored on the provider's servers, and are not used to train models.

Generation speed can also be improved locally. Google's Multi-Token Prediction drafters for Gemma 4, available on Ollama under Apache 2.0, speed up generation by up to 3x using speculative decoding, in which a small model proposes several tokens ahead and the large one checks them together. Google said that "if the target model agrees with the draft, it accepts the entire sequence in a single forward pass," and the target model adds one extra token of its own in that same pass.

Local models can also gain web search and file access through MCP servers—short for Model Context Protocol—and LM Studio added support in version 0.3.17, according to the Decrypt guide.

Where local models still lose to the cloud

The four-month average describes the most capable open models, and the largest are far past consumer hardware. Alibaba's Qwen3.8-Max has 2.4 trillion total parameters, with 95 billion active at any given time, compared with the 80GB of memory that gpt-oss-120b runs within.

When a model spills out of VRAM into system RAM, responses may slow considerably, and larger context windows need more memory. Gemma 4's small models offer a 128K-token context window—the amount of text a model can consider at once—while the larger ones offer 256K. Cloud services require none of that planning, and the frontier models behind them still lead on the hardest tests.

The practical choice is therefore a tradeoff between control and convenience: local tools keep conversations on the computer and allow users to choose compatible or customized weights, but users must match the model to available memory and accept that the most capable models may require cloud services or more powerful hardware. Web access is also not automatic when a model is offline; it requires an MCP server or another connected tool.

For those who want to use AI privately, Decrypt's own review recommended "xPrivo with a self-hosted local model for day-to-day drafting that must not leave your machine," and Kagi "when you need web retrieval and accept the API call tradeoff."