# AI Inference VM

> Current snapshot of a virtualized dual-3090 inference host with a vLLM/LiteLLM control plane.

Status: active · Kind: snapshot · Last verified: 2026-07-20 · Date: 2026-07-12

> Status: active
> Last verified: 2026-07-20 (against the current hardware inventory and AI platform overview)
> Scope: VM spec, inference stack, and service inventory for the primary AI inference host

## Purpose

This Proxmox guest is dedicated to local GPU inference and AI control-plane services. It provides OpenAI-compatible model access for editor integrations, chat interfaces, and automation agents.

RAG services (Qdrant, embeddings, and indexing) run on a separate compute appliance and no longer share these GPUs. Media serving also moved to a dedicated guest in July 2026. This VM is now inference and control-plane only.

## Current Shape

| Component     | Spec                                                                     |
| ------------- | ------------------------------------------------------------------------ |
| Hypervisor    | Proxmox VE on AMD EPYC 7402P (24C/48T)                                |
| vCPUs         | 48 (1:1 with host threads, no overcommit)                               |
| RAM           | 96 GiB dedicated (1G hugepages)                                         |
| GPUs          | 2x NVIDIA RTX 3090 (24 GB each) with NVLink bridge                      |
| GPU passthrough | Full PCIe passthrough (GPU + audio), no mediated devices              |
| OS            | Ubuntu 24.04.4 LTS, kernel 6.8.x (PREEMPT_DYNAMIC)                      |
| NVIDIA driver | 580.126.20                                                               |
| CUDA          | 12.9                                                                     |
| Container     | Docker                                                                   |
| Storage       | Local SSD-backed VM disk + NFS mounts from TrueNAS (models) and dev host |

## Inference Stack

| Layer | Model / Role |
|---|---|
| Deep vLLM lane | Qwen3.6-27B AWQ, tensor parallel across both GPUs (TP=2) |
| LiteLLM front door | Stable `deep` and `fast` capability aliases across separate inference lanes |

All inference runs in containers. No models run directly on the guest operating system.

The deep lane (vLLM TP=2) uses both GPUs for a single 27B model. The previous
one-model-per-GPU pattern (Qwen 2.5 7B + 14B) was replaced in favor of a unified
tensor-parallel serving. A 32B AWQ alternative mode exists but requires
CUDA graph capture narrowing (`max_cudagraph_capture_size=8`) to avoid Xid 13 faults
on dual 3090 hardware.

## Control Plane & Adjacent Services

| Service | Role |
|---|---|
| Open WebUI | Browser-based chat interface |
| LiteLLM | API router for deep and fast capability lanes |
| Chat service | Supplementary chat and skill dispatch |
| Voice notes | Browser capture and speech transcription |
| vLLM | Deep inference engine |
| Matrix service | Internal chat transport |
| SearXNG | Privacy-focused search engine |
| Agent services | Scheduled analysis and orchestration |

RAG, vector storage, embeddings, and indexing run on the separate retrieval appliance. Consumers reach them through private service contracts that are intentionally omitted here.

## Constraints

- **VM over bare metal:** Isolation, predictable recovery, easier lifecycle
- **No Kubernetes:** Single-node, GPU-bound workload; Docker Compose suffices
- **No HA:** Latency and simplicity over redundancy
- **GPU-bound:** Both 3090s are typically committed to the deep lane; no headroom for ad-hoc workloads
- **CUDA graph instability:** 32B+ TP=2 on dual 3090 requires narrowed graph capture or eager mode

## Drift / Unknowns

- NVIDIA driver and vLLM versions drift with upstream releases; pin versions when
  documenting rather than assuming latest
- The 27B AWQ model may be swapped for newer Qwen or other family members without
  architectural change
- Additional orchestration and chat services are experimental and may be
  pruned or promoted

## Related

- [AMD EPYC 7402P Proxmox Host](/homelab/hardware/epyc-7402p) — Bare-metal host