NightRun takes local LLM execution into territory that’s much closer to firmware than to a traditional desktop. The project boots directly from a USB drive via UEFI, loads the full model into RAM, and runs inference on the CPU with no Linux, no Windows, no conventional kernel, and no network stack. It isn’t trying to replace Ollama or llama.cpp for everyday use; instead, it builds a dedicated machine where nearly all the software that normally sits between the firmware and the LLM disappears.
NightRun’s technical highlights in 30 seconds
- NightRun is a no_std EFI application written in Rust that runs on top of UEFI Boot Services.
- It loads quantized models between 1.3 and 2.4 GB entirely into RAM and then blocks further reads from storage.
- It runs inference exclusively on the CPU, using AVX2 on x86_64 and NEON on the Raspberry Pi 5.
- It implements Llama 3.2, Qwen3, and Granite 4.1 dense models, validated against llama.cpp.
- It includes no conventional kernel, user processes, services, shell, browser, or TCP/IP stack.
The point about UEFI matters. NightRun isn’t bare metal in the strict sense of the term. The project doesn’t call ExitBootServices(), so firmware services stay available throughout the session. UEFI provides the functions needed to handle the display, keyboard, storage, and booting of the different CPU cores.
That’s a deliberate choice. Writing custom drivers for USB HID, xHCI, storage, graphics, and the quirks of thousands of motherboards would turn the project into something much closer to writing an operating system.
NightRun uses the firmware as a platform layer and builds its own model loader, inference engine, memory management, tokenizer, graphical interface, and multi-core execution on top of it.
From BOOTX64.EFI to the First Token Without Booting a Kernel
On an x86_64 PC, the firmware runs BOOTX64.EFI directly. On the Raspberry Pi 5, the entry point is BOOTAA64.EFI.
From there, a sequence begins that’s quite different from any typical local-AI setup:
- NightRun enables the SIMD state the CPU needs.
- It initializes the framebuffer and keyboard.
- It boots the available cores using UEFI’s MP services.
- It detects memory.
- It loads the full model into RAM.
- It checks its integrity while reading it.
- It seals off further access to storage.
- It initializes the tokenizer and conversation template.
- It reserves the KV cache and working memory.
- It starts the chat.
The runtime uses the UEFI Graphics Output Protocol (GOP) to access the framebuffer and draws its own interface. There’s no compositor, Linux terminal, or graphics server hiding underneath.
The same goes for the keyboard. NightRun uses the services provided by the firmware to receive USB input, but text editing, history, and the chat interface all belong to the project itself.
| Layer | Ollama on Linux | NightRun |
|---|---|---|
| Firmware | UEFI | UEFI |
| Kernel | Linux | None |
| OS drivers | Yes | UEFI services |
| Userspace | Yes | No |
| Background services | Yes | No |
| LLM runtime | Ollama/llama.cpp | NightRun |
| Model | RAM + possible I/O | RAM |
| Network | Available | Not implemented |
| Interface | Terminal/web/app | Custom framebuffer |
That architecture explains why describing it simply as “Ollama without internet” falls short — even Ollama’s own recent speed gains on Apple Silicon still depend on a full operating system underneath.
The Model Loads Once, Then Storage Disappears From the Picture
Another difference lies in how NightRun handles the model.
The installer starts from a compatible GGUF file, inspects its architecture, and converts it to NightRun’s own .nrm format. This container is specifically designed so tensors can be used directly from the representation loaded into memory.
During boot, the file is read in blocks, and NightRun computes its CRC-32 checksums at the same time, avoiding a second full pass just to verify integrity.
Once that process finishes, the weights stay in RAM.
The runtime then enforces a rule: any later attempt to read from storage fails. Inference never queries the USB drive or microSD card again.
All the memory it needs is also reserved in advance.
The design includes regions for the weights, KV cache, prefill workspace, decode scratch memory, and graphics buffers. A function precalculates the size of the memory arena, and if the machine doesn’t have enough RAM, the boot fails before the conversation even starts.
The generation loop performs zero heap allocations, according to the project’s documentation.
The practical consequence is that RAM determines which models can be used.
A 1.3 GB Llama 3.2 1B requires a machine with about 4 GB. Llama 3.2 3B and Granite 4.1 3B need around 6 GB, while Qwen3 4B requires 8 GB.
CPU Inference Using Quantized Weights Directly
NightRun currently has no GPU support.
Its engine is built for quantized inference on the CPU and uses the weights without first generating a full decompressed FP32 copy.
On x86_64 it implements specific kernels for AVX2, FMA, and F16C. On ARM it uses NEON and has an sdot-based path for compatible processors such as the Raspberry Pi 5‘s Cortex-A76.
The quantizations implemented are Q8_0, Q4_K, and Q6_K, plus certain F32 tensors.
The project also explains a quirk that tends to get lost when people talk about GGUF files: Q4_K_M doesn’t mean every single tensor is stored as Q4_K. A model can mix Q4_K, Q6_K, and F32 depending on the tensor. NightRun preserves those quantizations during conversion.
Prefill processes up to 64 tokens per pass, while decode works token by token.
That matters because the two phases have different profiles. Initial processing can reuse weights across several tokens, while during interactive generation, performance is much more constrained by memory bandwidth.
NightRun also acknowledges a familiar consequence of transformers: generation slows down as the context grows, because attention has to scan an ever-larger KV cache.
Llama, Qwen, and Granite — But Not Just Any GGUF
NightRun doesn’t try to offer universal compatibility.
It currently implements three families: Llama 3.2, Qwen3, and Granite 4.1 in their conventional transformer architecture.
| Validated model | Quantization | .nrm size | Target RAM |
|---|---|---|---|
| Llama 3.2 1B Instruct | Q8_0 | 1.3 GB | 4 GB |
| Llama 3.2 3B Instruct | Q4_K_M | 1.9 GB | 6 GB |
| Granite 4.1 3B | Q4_K_M | 2.0 GB | 6 GB |
| Qwen3 4B Instruct 2507 | Q4_K_M | 2.3 GB | 8 GB |
Granite’s hybrid SSM/MoE variants are explicitly rejected during conversion. Adding a new architecture means implementing and validating its execution, not just adding its name to the catalog.
There’s a fair amount of technical work behind that restriction.
Qwen3, for instance, uses a different RoPE layout than Llama, a specific RMSNorm for Q/K, and an attention width that differs from the hidden dimension. NightRun implements these differences rather than trying to solve them with a generic layer.
The tokenizer gets similar treatment.
Conversation templates are compared token by token against Hugging Face’s apply_chat_template, and there are specific tests to prevent user-entered text from being accidentally interpreted as control tokens.
llama.cpp Is the Correctness Reference
One of the project’s most interesting technical aspects is how it tries to verify that stripping out so many layers doesn’t change the model’s output.
NightRun uses llama.cpp as the reference for greedy generation.
For the supported families, engine changes must keep token-by-token generation consistent with the reference. Scalar kernels, in turn, are used to check the vectorized AVX2 and NEON implementations.
The project also checks that batched prefill produces the same logits and KV cache as processing tokens sequentially.
That doesn’t prove NightRun is bug-free, but it provides a reproducible method for catching divergences in an inference engine built practically from scratch.
The project itself also avoids a tempting claim: it doesn’t claim to be faster than llama.cpp across the board.
Its published benchmarks show roughly 20 tokens per second during Llama 3.2 1B decode on QEMU/KVM with eight cores and AVX2. Granite 4.1 3B runs at around 13-14 tokens per second, and Qwen3 4B at around 10-11.
On the Raspberry Pi 5, Granite 4.1 3B reached 3 tokens per second in the published test, though that measurement predates the new sdot kernels.
The documentation estimates that decode can be comparable to llama.cpp under certain conditions, while prefill lags behind by a factor of 1.15 to 1.4.
The goal isn’t to win a benchmark. The experiment is about testing where the model can actually run.
No TCP/IP: Isolation Is Built Into the Architecture
A machine running Ollama can also be physically disconnected from the internet. From a practical standpoint, that can offer excellent isolation if configured correctly.
NightRun takes a more radical approach: there’s no network stack inside the runtime at all.
There’s no browser, no daemons, no automatic updates, no telemetry, and no process capable of accidentally opening a TCP connection, because the software needed to do that simply isn’t implemented.
That significantly cuts down the active software surface during a session, though it doesn’t automatically make the machine invulnerable.
UEFI is still part of the chain of trust. The model’s origin, the generated image, the computer used to prepare it, and NightRun’s own code all matter too.
That’s why the project is especially interesting from a systems-architecture standpoint, rather than as an absolute security claim.
An AI Project That Was Also Built With AI
NightRun adds another quirk: its creators say most of the code was written using Claude Code with the Fable 5 model.
That detail stands out especially given the kind of software involved.
This isn’t a matter of generating a typical web app. The repository contains code for UEFI, binary formats, SIMD, memory management, tokenization, quantized inference, a framebuffer, and an installer that writes directly to storage devices.
That’s precisely why the creators emphasize validation through reference implementations, comparisons with llama.cpp, parser tests, and installer checks.
NightRun remains experimental. It requires UEFI, Secure Boot disabled, models that fit entirely in RAM, and a reasonably modern CPU for interactive performance. Real-world x86_64 hardware coverage is still limited, and different firmware implementations can produce incompatibilities.
But technically, it raises an interesting question about the future of local AI.
Until now, the trend has been to make running an LLM inside the operating system easier and easier. NightRun explores the opposite direction: stripping out the conventional operating system and temporarily turning the entire computer into an appliance dedicated to the model.
The result isn’t meant to compete with the convenience of Ollama. It’s a different category: an inference runtime that fits on a USB drive, boots from firmware, loads the weights into RAM, and then runs with no disk and no network.
For certain privacy experiments, isolated systems, and edge computing, that difference can be far more interesting than squeezing out a few extra tokens per second.
Source: Noticias inteligencia artificial
