Local Inference Stack Architecture

activereferenceverified 2026-07-19

Status: active Last verified: 2026-07-19 (against the current AI platform overview, evaluation contract, and attended vision-canary runbook) Scope: model routing, capability lanes, readiness checks, and promotion boundaries

The local inference stack provides private, OpenAI-compatible model access for chat, coding agents, retrieval-assisted tools, and unattended services. It is an internal platform rather than a public API.

This page describes capability boundaries and change controls. Deployment addresses, management endpoints, model paths, prompts, and credentials remain in private operations documentation.

Request path

LiteLLM is the stable front door. Callers request a capability alias instead of binding themselves to a particular model build or inference server. LiteLLM then forwards the request to the corresponding vLLM lane.

chat, agents, and internal services
                ↓
       LiteLLM model front door
          ↙               ↘
   fast vLLM lane      deep vLLM lane
   short tasks         reasoning, tools,
                       long context, vision

The aliases separate the client contract from the model lifecycle. A model can be evaluated, rejected, promoted, or rolled back behind a lane without changing every consumer.

Retrieval is adjacent to inference rather than embedded inside it. Retrieval services assemble grounded context; the selected model lane reasons over that context. A retrieval failure and an inference failure therefore remain distinguishable operational problems.

Lane roles

LaneIntended workDesign priority
FastClassification, routing, extraction, and short plumbing tasksLow startup and response cost
DeepComplex reasoning, code, long-context synthesis, native tool calls, and bounded image understandingCapability and answer quality

The lanes are roles, not permanent model identities. Current implementations use models from the Qwen family, but callers depend on fast and deep, not on a specific checkpoint name.

Workload owners choose the lane deliberately. Short mechanical work should not consume the deep lane merely because it is available, while tool use or difficult synthesis should not be sent to the fast lane without evidence that it satisfies the required contract.

Readiness is more than a running container

A lane is ready only after the complete request path has been exercised. Validation checks that:

  1. The front door advertises the expected capability aliases.
  2. A normal completion succeeds through the routed path.
  3. Native tool calls produce structurally valid arguments rather than tool-shaped prose.
  4. Required retrieval and agent consumers can still reach the model front door.
  5. The serving process has no new out-of-memory events, restart loops, or active health alerts.

Multimodal readiness adds bounded image cases while preserving the existing text, tool, and long-context behavior. Passing a new capability does not excuse a regression in an old one.

Candidate promotion

Model and serving changes use an attended, fail-closed canary:

  1. Freeze the evaluation corpus, answer keys, and numeric rejection thresholds.
  2. Verify that rollback material and monitoring are available.
  3. Capture an immediate production baseline with the same prompts and sampling settings.
  4. Apply one reversible candidate change.
  5. Record raw responses, transport failures, tool parsing, latency, truncation, and resource health.
  6. Promote only when every applicable hard gate passes.
  7. Restore the baseline before investigating a failed candidate.
  8. Recheck routed requests, downstream consumers, restart state, and alerts after either promotion or rollback.

The candidate model does not grade itself. Exact answers and structural checks are primary; any qualitative tie-break uses an independent judge with the raw judgments retained.

Deployment and evaluation are intentionally separate. The evaluation runner measures a candidate but does not deploy, restart, reroute, or restore services. Those mutations stay in an attended runbook with an explicit rollback checkpoint.

Failure boundaries

FailureExpected effect
One inference lane is unavailableRequests assigned to that capability fail unless the caller has an independently verified fallback
Shared model front door is unavailableBoth local capability aliases become unreachable through the normal path
Retrieval service is unavailableGrounded project or infrastructure context is unavailable even if raw inference still works
Candidate misses a hard gatePromotion stops and the known baseline is restored
Model behavior regresses after promotionRoll back the lane, then verify routing, consumers, process health, and alerts
GPU capacity is exhaustedAdmission slows or fails; health and resource evidence determine recovery rather than blind restart loops

The platform favors explicit capability contracts over transparent magic. Routing is simple enough to inspect, promotion evidence is preserved, and rollback is part of the change result rather than optional cleanup.