Ollama Nearly Doubles LLM Speed on Mac With MLX, But There’s a Catch

Cloud
Share on Pinterest Share on LinkedIn

Ollama has changed a key part of its architecture for running artificial intelligence models on Apple Silicon Macs. Starting with version 0.19, it includes, in preview, an engine based on MLX, Apple’s machine learning framework, capable of raising Qwen3.5-35B-A3B’s generation speed from 58 to 112 tokens per second in benchmarks published by the company. The improvement is close to 93%, though it comes with an important catch: GGUF models that many users already had downloaded still run on llama.cpp and don’t automatically get that speedup.

Ollama and MLX: the key points in 20 seconds

  • Ollama 0.19 introduced an MLX engine for Apple Silicon in March.
  • On Qwen3.5-35B-A3B, generation speed went from 58 to 112 tokens/s in Ollama’s own tests.
  • GGUF models still run on llama.cpp; MLX works with compatible models in safetensors format.
  • In June, new optimizations boosted the MLX engine’s performance by another 20%.
  • Ollama 0.31.1 added Multi-Token Prediction to speed up Gemma 4.

The change is especially relevant for developers who use a Mac as a local workstation to run LLMs, coding agents, or assistants without constantly depending on an external API. But it also shows just how much performance now depends not only on the model and the hardware, but on the inference engine, the weight format, quantization, and platform-specific optimizations.

Ollama officially unveiled the integration on March 30, 2026. The initial version accelerated Qwen3.5-35B-A3B and required more than 32GB of unified memory for the model used in the demo. Since then, both the support and the MLX engine itself have kept evolving.

From 58 to 112 tokens per second by switching engines

The first figures published by Ollama show a considerable difference.

The company compared Qwen3.5-35B-A3B using Ollama 0.18 and its previous Q4_K_M implementation against the MLX-based version with the model quantized in NVFP4. In prompt processing, or prefill, the result went from 1,154 to 1,810 tokens/s, an improvement of roughly 1.57x.

In generation, or decode — the speed the user actually perceives as the answer appears — it went from 58 to 112 tokens/s, around 1.93x faster.

Ollama also noted that an int4 variant could reach 1,851 tokens/s of prefill and 134 tokens/s of generation. These are the company’s own measurements and correspond to models with different quantizations, so they shouldn’t be read as a comparison exclusively between MLX and llama.cpp.

That distinction matters.

The jump doesn’t come from a single optimization. Switching paths also changes the format used by the model and can change its quantization. So simply stating that “MLX is 93% faster than llama.cpp” would overgeneralize a result obtained under specific conditions.

An independent comparison carried out by Ante Kapetanovic just days before the launch helps put the differences in perspective. Testing Qwen3.5-35B-A3B on an M4 Max with 128GB of unified memory, he got 48.1 tokens/s with Ollama running on llama.cpp, 72.4 tokens/s running llama.cpp directly, and 131.8 tokens/s via mlx-lm.

The quantizations weren’t identical either, so those figures don’t perfectly isolate the engine’s influence. What they do show is that the chosen execution layer can significantly change performance even on the same Mac and the same model family.

Updating Ollama doesn’t automatically speed up older models

This is probably the most relevant detail for anyone already using Ollama.

Installing a new version doesn’t mean every model stored on the machine starts running through MLX.

The architecture keeps two inference paths. GGUF models still run on llama.cpp, while compatible models distributed as safetensors can use the MLX engine on macOS on ARM64.

That’s why a user can update Ollama and see essentially the same performance if they keep running the exact same GGUF model.

The switch has to reach the model being used too.

This separation preserves the enormous existing library built around GGUF and llama.cpp while Ollama gradually expands its MLX-compatible catalog. The trade-off is that the improvements aren’t universal.

The first preview version focused on qwen3.5:35b-a3b-coding-nvfp4. Other models and optimizations followed. In June, for example, Ollama documented running gemma4:12b-mlx through the new engine.

The pace of these version updates itself shows that this is still a young architecture. Updates have included fixes for model loading, embedding layers, matrix operations, and new architectures.

Why MLX fits Apple Silicon especially well

MLX isn’t just another interface on top of Metal.

Apple designed it specifically for its processors and their unified memory architecture, where the CPU and GPU can work on the same physical memory. According to Apple, MLX operations can run on CPU or GPU without having to explicitly move data between separate memory spaces.

This doesn’t mean llama.cpp ignores the advantages of Apple Silicon. Its Metal backend is also heavily optimized for Macs.

MLX does add, however, features like lazy graph evaluation and optimizations tightly coupled to Apple’s hardware.

Ollama explained in June that it had made its MLX engine up to 20% faster than its own previous version by fusing several operations into Metal kernels through MLX’s JIT compiler and reworking GPU-side sampling.

The gap takes on another dimension with the M5 generation.

Apple introduced Neural Accelerators inside the GPU cores to speed up matrix operations. In its own MLX tests, the company found improvements of between 3.33x and 4.06x in time-to-first-token compared with an M4 MacBook Pro, depending on the model tested.

Token generation afterward improved much less, between 1.19x and 1.27x.

That’s not a contradiction.

Apple explains that initially processing the prompt is a workload heavily dependent on compute capacity, so the Neural Accelerators can step in directly. Token-by-token generation is far more constrained by memory bandwidth.

On the machines Apple compared, that went from 120GB/s on the M4 to 153GB/s on the M5, a 28% increase.

To use the M5’s improved Neural Accelerator capabilities through MLX, Apple also requires macOS 26.2 or later.

NVFP4 is also part of the equation

The engine switch has come paired with another technical piece: NVFP4.

Ollama is using this low-precision format developed by NVIDIA in some of its MLX-ready models. Compared with storing weights at higher precision, 4-bit quantization reduces the memory required and — especially important during decode — the amount of information that has to be continuously moved from memory.

Ollama argues that NVFP4 preserves quality better than Q4_K_M in its tests with Gemma 4 12B. The company states that it roughly halves the quality loss caused by quantization compared with BF16, and that, with its updated MLX engine, it generates around 20% faster than Q4_K_M. These are vendor-reported results and don’t amount to a universal advantage for every model.

There’s also a practical reason to use the format.

NVFP4 is aimed at inference workloads on NVIDIA infrastructure. Ollama is thus raising the possibility of running locally models prepared with a quantization similar to the one later used in data centers.

But again, the variables need to be separated: when comparing a GGUF Q4_K_M model run through llama.cpp with a safetensors NVFP4 model run through MLX, it’s not only the inference engine that’s changing.

Gemma 4 adds another speedup: generating several tokens at once

The evolution didn’t stop with MLX.

Ollama 0.31.1, released on June 30, added improvements for Multi-Token Prediction (MTP) in Gemma 4 on Apple Silicon.

The idea is to try to anticipate several tokens during generation instead of strictly producing one per step. Ollama dynamically adjusts how many tokens it’s worth proposing.

According to the benchmarks published in the release notes, Gemma 4 12B NVFP4 on an M5 Max went from 50.2 to 95 tokens/s on an Aider Polyglot benchmark, close to 90% faster. The company says the feature is enabled by default, requires no configuration, and doesn’t change the model’s output.

This is a different improvement from the initial jump from llama.cpp to MLX, so the two percentages shouldn’t be added together to produce a supposed total speedup.

What the successive versions do show is where local inference on Mac is heading: increasingly using Apple Silicon-specific features rather than relying solely on a common layer that’s portable across different platforms.

For a developer, this introduces a decision that used to go unnoticed. It’s no longer enough to look at which model Ollama is running. It also matters which variant was downloaded, what quantization it’s stored in, and which engine ends up processing it.

A perfectly functional GGUF can still be the right choice for compatibility, availability, or quality. But updating Ollama and continuing to use the exact same files doesn’t mean the machine is taking advantage of the new MLX speedups.

Frequently asked questions

Does Ollama use MLX automatically on every Mac?

Ollama has an MLX engine for Apple Silicon, but not every model uses that path. Existing GGUF models still run through llama.cpp, while MLX-compatible variants use a different inference path.

Does MLX make Ollama twice as fast?

In Ollama’s initial test with Qwen3.5-35B-A3B, decode speed went from 58 to 112 tokens/s, an improvement of roughly 1.93x. That result can’t be generalized to every model, Mac, and quantization.

What does the M5 add compared with earlier generations?

The M5 GPU’s Neural Accelerators especially speed up the matrix operations used during initial prompt processing. Apple measured improvements of up to 4.06x in time-to-first-token versus the M4, while later token generation improved between 19% and 27% across the models tested.

Do Ollama models need to be downloaded again?

To take advantage of a variant built specifically for MLX, you need to use the corresponding model. Updating Ollama alone doesn’t turn already-downloaded GGUF models into MLX models.

Sources:

Share on Pinterest Share on LinkedIn
Scroll to Top