NVIDIA Open-Sources Switchyard to Cut AI Cost and Latency With 2-Model Routing
Updated
Updated · KDnuggets · Sep 4
NVIDIA Open-Sources Switchyard to Cut AI Cost and Latency With 2-Model Routing
3 articles · Updated · KDnuggets · Sep 4
Summary
NVIDIA released NeMo Switchyard as an open-source proxy and library that routes each AI request to a cheaper or stronger model instead of sending all traffic to one frontier model.
Switchyard supports several routing modes, starting with weighted random routing and moving to classifier-based deterministic routing that estimates whether a weak model can solve a task before escalating.
In NVIDIA’s example, a simple math prompt was routed to GPT-4o-mini with 1,428 ms latency, while a harder Redis race-condition prompt went to GPT-4o at 4,475 ms.
The software also targets multi-turn agents through stage-based and escalation routing, using progress, failure and recent-turn signals to decide when extra model capability is worth the cost.
NVIDIA frames the tool around measurable trade-offs: a sample comparison showed router-based serving at $9 and 89% task success versus $20 and 92% for strong-only.