SAN JOSE, September 18 — NVIDIA has published a technical blog post describing a TensorRT Edge-LLM submission to the MLPerf Inference v6.1 Edge Agentic benchmark that ran the Qwen3.6-27B large language model on a single NVIDIA Jetson AGX Thor Developer Kit and achieved 52.33 tokens per second. The post, dated September 16 and written by Luxiao Zheng, Palanivel Guruva reddiar, Maximilien Breughe, Zhihan Jiang, Zhijia Liu and Lin Chai, reports that the submission completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds.
The authors state that is 6.4 times faster than the llama.cpp reference run of 2 hours and 37 minutes.
What the benchmark measures
The MLPerf Edge Agentic benchmark, developed by the MLCommons consortium, measures an OpenAI-compatible model endpoint in two phases: performance and accuracy. The performance phase replays recorded software-engineering agent trajectories in which the model receives a user request, generates a tool call, observes the tool result, and continues the same conversation. The workload contains 20 conversations and 1,007 generated turns.
Input length grows across turns and reaches approximately 23.5K tokens, which the authors identify as making long-context processing an important part of the measurement. An Intersection of Union based inline accuracy metric is also measured to ensure the performance phase is running the agent correctly.
The accuracy phase uses Berkeley Function Calling Leaderboard v4 prompts with single-turn only and reasoning off, a choice the post describes as balancing accuracy and evaluation time on edge devices.
Where the speedup comes from
According to the blog post, the speedup comes from several techniques: NVFP4 quantization for weights and activations, an FP8 KV cache, tree-based multi-token prediction, and KV cache reuse across agent turns. KV cache and recurrent-state reuse served approximately 96 percent of prompt tokens from hot cache, reducing repeated prefill computation across the full agent trajectory. Tree-based multi-token prediction with an 8-step, top-2, 16-node verification tree delivered an additional roughly 40 percent decoding performance gain over linear MTP for function-calling workloads.
The post frames the result as part of a broader shift of AI agents from cloud data centers to vehicles, robots, and other edge devices, where inference must generate tokens quickly, process long shared histories efficiently, and produce valid tool calls within limited power and memory envelopes. NVIDIA points readers to the TensorRT Edge-LLM release/0.9.1-mlpinf branch for model export settings, engine build commands, and server configuration, and to a calibrated Qwen3.6-27B NVFP4 checkpoint for deployment.
The company also directs readers to the MLCommons Edge Agentic example for dataset and client configuration details. The blog post was published on NVIDIA’s developer blog and carries an AI-generated summary disclaimer noting that AI-generated content may summarize information incompletely and that important information should be verified. What to watch next is whether the TensorRT Edge-LLM tooling and the calibrated checkpoint draw independent reproductions of the 52.33 tokens-per-second figure on Jetson AGX Thor hardware, and how the approach fares as the MLPerf Edge Agentic workload evolves in future rounds.
