Reference · 14 Sept 2026
Serving Qwen3.8-27B with vLLM on four RTX 3090s
The model, memory and gateway configuration behind the long-context language-model service running on mlrig.
Four RTX 3090s give the LLM VM on mlrig 96 GiB of physical VRAM. The normal serving stack is Docker with vLLM’s OpenAI-compatible server. At the time of writing, the main model is Qwen3.8-27B with FP16 weights distributed over all four GPUs using tensor parallelism.
The current configuration is designed to explore long-context inference rather than only maximise short-prompt throughput:
- 524,288-token advertised context length
- FP8 attention KV cache
- static YaRN scaling from the model’s native 262,144-token context
- automatic prefix caching
- up to two scheduled sequences
- per-request reasoning controls through an authenticated gateway
The FP16 weights preserve the model parameters without weight quantisation, while the FP8 KV cache spends less VRAM on remembered attention keys and values. That trade allows much longer contexts on 96 GiB of consumer GPU memory. It is also why I benchmarked FP8 and FP16 KV cache separately instead of assuming that the larger cache was free of quality costs.
The public-facing API does not connect directly to vLLM. A separate gateway handles API-key authentication, request limits, timeouts and usage accounting. It records prompt, generated, reasoning and cached-token statistics per key without exposing the inference server itself. Neat!