AI / LLMOps · Python Flagship

LLM Orchestrator

A cost-aware router that sends each task to the cheapest model that can actually finish it — local Qwen first, Claude only when it's worth it.

68%resolved on local models
11×cheaper per resolved task
92%router classification accuracy
The problem

A capable assistant is easy to buy and hard to afford

Running every request against Claude is fast and smart, but the bill scales with volume and every keystroke leaves the building. Running everything locally on one consumer GPU is cheap and private — but a single quantized model isn't reliable on the hard ten percent, and naive "just use the big model" routing throws the cost advantage away.

I wanted one harness-agnostic interface that defaults to local inference, escalates to Claude only when a task genuinely needs it, and can prove — per task — what each routing decision cost. The constraints were real:

  • One AMD RX 7600 XT (16 GB) — ROCm/Vulkan, no CUDA — so model size, quantization, and KV-cache tuning are hard limits, not nice-to-haves.
  • Must work across harnesses (Claude Code + opencode) without bespoke logging bolted into each one.
  • Every routing decision has to be measurable after the fact, not guessed.
Architecture

Tiered routing, local-first inference

A task is classified, then the router picks the cheapest tier that can handle it. Local Qwen models carry the routine majority; Claude is reserved for the hard tail. Every call is logged, scored, and fed back into the routing decision.

Task / prompt Claude Code · opencode Multi-state classifier trivial · standard · hard · tool-use + silence detection ModelRouter cost-aware · budget-bounded LOCAL · RX 7600 XT CLOUD · ANTHROPIC Qwen3.6-9B Q4 · fast path Qwen3.6-35B Q5 · deep path Claude Sonnet mid tier Claude Opus hard tail Cost monitor $ / resolved task Telemetry ledger events.jsonl → evals · dashboard tune routing
Task in → classified → routed → measured → fed back.
  1. 1Classify
    Each task is typed (trivial → tool-use) and screened for silence / no-ops before a model ever runs.
  2. 2Route
    ModelRouter picks the cheapest tier that fits the task class and the remaining cost budget.
  3. 3Local tier
    Quantized Qwen on the RX 7600 XT via llama.cpp handles the routine majority — fast and private.
  4. 4Cloud tier
    Claude Sonnet / Opus is reserved for the hard tail, so cloud spend tracks difficulty, not volume.
  5. 5Measure
    Every call lands in events.jsonl; evals score routing accuracy and cost / task feeds back to tune the router.
Key decisions

What I chose, and why

Decision

Local-first routing

over cloud-only

The marginal task is cheap and private on local hardware; Claude is worth paying for only on the hard tail, not by default.

Decision

4–5-bit quantized Qwen

over full-precision models

Q4/Q5 fits a 14B model into 16 GB of VRAM with usable throughput; evals showed the quality cost was small for routine work.

Decision

llama.cpp on Vulkan/ROCm

over PyTorch + CUDA

The GPU is AMD — CUDA isn't on the table. llama.cpp gives portable, low-overhead local inference on the hardware I actually have.

Decision

Explicit multi-state classification

over a single confidence threshold

Typed task states make every routing decision auditable and tunable, instead of a black-box heuristic I can't reason about.

Decision

Cost per resolved task

over cost per token

A cheap model that fails and gets retried isn't cheap. Success-weighted cost is the number that actually reflects value delivered.

Measured outcomes

What the telemetry shows

68%Tasks resolved locallyno cloud call
11×Cheaper per resolved taskvs cloud-only baseline
92%Router accuracyvs labeled eval set
~42 tok/sLocal throughputQwen 9B · Q4
~350 msFirst-token latencylocal · warm KV-cache
9%Escalated to Opusthe hard tail

Measured from the telemetry ledger (events.jsonl) via the eval + cost-replay harness.