AI Inference VM
Status: active Last verified: 2026-07-20 (against the current hardware inventory and AI platform overview) Scope: VM spec, inference stack, and service inventory for the primary AI inference host
Purpose
This Proxmox guest is dedicated to local GPU inference and AI control-plane services. It provides OpenAI-compatible model access for editor integrations, chat interfaces, and automation agents.
RAG services (Qdrant, embeddings, and indexing) run on a separate compute appliance and no longer share these GPUs. Media serving also moved to a dedicated guest in July 2026. This VM is now inference and control-plane only.
Current Shape
| Component | Spec |
|---|---|
| Hypervisor | Proxmox VE on AMD EPYC 7402P (24C/48T) |
| vCPUs | 48 (1:1 with host threads, no overcommit) |
| RAM | 96 GiB dedicated (1G hugepages) |
| GPUs | 2x NVIDIA RTX 3090 (24 GB each) with NVLink bridge |
| GPU passthrough | Full PCIe passthrough (GPU + audio), no mediated devices |
| OS | Ubuntu 24.04.4 LTS, kernel 6.8.x (PREEMPT_DYNAMIC) |
| NVIDIA driver | 580.126.20 |
| CUDA | 12.9 |
| Container | Docker |
| Storage | Local SSD-backed VM disk + NFS mounts from TrueNAS (models) and dev host |
Inference Stack
| Layer | Model / Role |
|---|---|
| Deep vLLM lane | Qwen3.6-27B AWQ, tensor parallel across both GPUs (TP=2) |
| LiteLLM front door | Stable deep and fast capability aliases across separate inference lanes |
All inference runs in containers. No models run directly on the guest operating system.
The deep lane (vLLM TP=2) uses both GPUs for a single 27B model. The previous
one-model-per-GPU pattern (Qwen 2.5 7B + 14B) was replaced in favor of a unified
tensor-parallel serving. A 32B AWQ alternative mode exists but requires
CUDA graph capture narrowing (max_cudagraph_capture_size=8) to avoid Xid 13 faults
on dual 3090 hardware.
Control Plane & Adjacent Services
| Service | Role |
|---|---|
| Open WebUI | Browser-based chat interface |
| LiteLLM | API router for deep and fast capability lanes |
| Chat service | Supplementary chat and skill dispatch |
| Voice notes | Browser capture and speech transcription |
| vLLM | Deep inference engine |
| Matrix service | Internal chat transport |
| SearXNG | Privacy-focused search engine |
| Agent services | Scheduled analysis and orchestration |
RAG, vector storage, embeddings, and indexing run on the separate retrieval appliance. Consumers reach them through private service contracts that are intentionally omitted here.
Constraints
- VM over bare metal: Isolation, predictable recovery, easier lifecycle
- No Kubernetes: Single-node, GPU-bound workload; Docker Compose suffices
- No HA: Latency and simplicity over redundancy
- GPU-bound: Both 3090s are typically committed to the deep lane; no headroom for ad-hoc workloads
- CUDA graph instability: 32B+ TP=2 on dual 3090 requires narrowed graph capture or eager mode
Drift / Unknowns
- NVIDIA driver and vLLM versions drift with upstream releases; pin versions when documenting rather than assuming latest
- The 27B AWQ model may be swapped for newer Qwen or other family members without architectural change
- Additional orchestration and chat services are experimental and may be pruned or promoted
Related
- AMD EPYC 7402P Proxmox Host — Bare-metal host