2026 March
Nemo 3 Family
Another March release here, this time from nvidia. Nvidia releases models often to demo possibilites and to destroy moats so they can sell more cards (maybe, I just guessing based on things that seems completely obvious).
The new Nemo 3 models are a novel type of mixture of experts, where they did some crazy tricks to make more experts fit and run just as fast. The also increased the context window to 1M tokens and trained the model natively at NVFP4. Training at 4 bits instead of the usual 32 or 16 means that the quantization effects are minimized so you get close to the same results at a huge speed, size, and memory savings. In fact the accuracy seems almost identical when compared to running it at 16 bit precision.
Nemo 3 was trained to excel at agentic tasks and at following directions and reasoning its way through structured instructions. The 4 bit quant of the 120B ‘Super’ model can fit completely into 64GB of system ram and runs pretty snappy if you have a GPU to help with prompt processing. There is also a 30B model available for people with less RAM or who have to divide the RAM amongst other computing needs.
Plato’s Opinion
The Nemo series have a reputation for pulling way more weight than other models in their size class, and are revered not only for tasks but for conversations and even role play. The ability to run at 4 bit with minimal loss makes this something that many people should be interested in. Even if you don’t use this model, you might appreciate that the training data, the code used to train it, the process, and the weights are all open and available for anyone to use.
Instructions
Open up a terminal and clone llama.cpp
git clone https://github.com/ggml-org/llama.cpp
Update or install the dev tools and build it
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
cmake llama.cpp -B llama.cpp/build \
-DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
Install uv if you don’t have it
curl -LsSf https://astral.sh/uv/install.sh | sh
Install huggingface hub if you don’t have it
uv tool install huggingface_hub hf_transfer
Run download the 120b model
hf download unsloth/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF \
--local-dir ~/models/gguf/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF \
--include "*UD-Q4_K_XL*"
Or the 30B
hf download unsloth/Nemotron-3-Nano-30B-A3B-GGUF \
--local-dir ~/models/gguf/Nemotron-3-Nano-30B-A3B-GGUF \
--include "*UD-Q4_K_XL*"
Deploy
./llama.cpp/llama-server \
--model ~/models/gguf/NVIDIA-Nemotron-3-Super-120B-A12B-GGUF/NVIDIA-Nemotron-3-Super-120B-A12B-UD-Q4_K_XL-00001-of-00003.gguf \
--alias "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B" \
--prio 3 \
--min_p 0.01 \
--temp 0.6 \
--top-p 0.95 \
--ctx-size 16384 \
--port 8001