A reference run that took 2 hours and 37 minutes. A submission that finished the same job in 24 minutes and 36 seconds. That gap is the headline of an NVIDIA engineering post published September 16, 2026, and it is NVIDIA’s own account of its own results.
The company said its TensorRT Edge-LLM software ran the Qwen3.6-27B language model on a single Jetson AGX Thor Developer Kit and completed all 1,007 turns of the MLPerf Inference v6.1 Edge Agentic performance workload. Output speed was 52.33 tokens per second. NVIDIA put that at 6.4x less time than the llama.cpp reference run published for the same board. The post was written by six NVIDIA engineers, and none of these figures has been independently confirmed here.
Why agents are a harder test
The authors start from a change in where AI runs. Agents are moving out of cloud data centers and into vehicles, robots and other edge devices, they wrote. Unlike a chatbot answering one prompt, an agent works through steps: it picks a tool, reads what comes back and keeps reasoning inside a conversation that keeps growing. That asks a lot of a small device. It has to produce tokens quickly, handle a long shared history and return valid tool calls within limited power and memory.
The MLPerf Edge Agentic benchmark, built by MLCommons, tests an OpenAI-compatible endpoint in two phases. In the performance phase, recorded software-engineering agent sessions are replayed. The model gets a request, writes a tool call, sees the result and carries on in the same conversation. There are 20 conversations and 1,007 generated turns, and the input grows to approximately 23.5K tokens. The accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts, single-turn with reasoning off, which the authors say balances accuracy against evaluation time on edge hardware.
The numbers behind the run
NVIDIA ran the model in SingleStream mode on one board with 128 GB of unified memory at the MAXN power mode. It reported a median time to first token of 247.12 ms, a median time per output token of 14.68 ms and overall BFCL accuracy of 87.94%.
The post credits three main techniques. The first is NVFP4, a 4-bit floating-point format supported by the Blackwell GPU in the Thor board, applied to weights and activations, with an FP8 cache. Smaller data means less memory traffic, which the authors say is the main limit on low-batch decoding at the edge.
The second is cache reuse. Each request repeats most of the earlier conversation, so the runtime restores stored attention data and prefills only the new part. On this workload about 96% of prompt tokens came from a hot cache, and the runtime prefilled roughly 0.5M of 13.6M prompt tokens.
The third is tree-based multi-token prediction. A draft model proposes several likely continuations, organized as a tree, and the main model checks them in one pass. The MLPerf configuration used 8 draft steps, the top 2 candidates at each depth and a 16-node verification tree. NVIDIA said that gave an additional roughly 40% decoding gain over a linear setup with 3 draft steps, because tool names and JSON syntax are often predictable.
NVIDIA published the code on the release/0.9.1-mlpinf branch of TensorRT Edge-LLM, along with a calibrated Qwen3.6-27B NVFP4 checkpoint. The benchmark run uses temperature 0, seed 42, reasoning disabled and concurrency 1.
