Reference · 13 Sept 2026
mlrig: six RTX 3090s under Proxmox
A technical overview of the GPU compute host used for local language models, numerical experiments and isolated rental workloads.
mlrig is the main GPU compute host in my homelab. It began as a bare-metal Ubuntu workstation for CUDA experiments and local language models, but grew into a six-GPU machine with several workloads that should not share the same trust boundary. It now runs Proxmox VE and divides the hardware between one trusted Ubuntu environment and two isolated VMs for other work.
The name is historical. The machine has been used for machine learning, but it is also a general-purpose CUDA laboratory: model serving, inference benchmarks, numerical software experiments and tests of GPU acceleration all run here!
Hardware
| Component | Configuration |
|---|---|
| Motherboard | ASRock Rack ROMED8-2T |
| Processor | AMD EPYC 7282, 16 cores / 32 threads |
| Memory | 256 GiB |
| GPUs | 6 x NVIDIA GeForce RTX 3090, 24 GiB each |
| Aggregate GPU memory | 144 GiB |
| Networking | Dual-port Intel X550 10 Gigabit Ethernet |
| Local storage | 1 TB and 500 GB NVMe SSDs |
The EPYC platform is important less for raw single- or multi-threaded performance than for its 128 gen 4 PCIe lanes. All cards negotiate PCIe 4.0 x16 links which is important for tensor parallelism. Each GPU and its audio function sits in an IOMMU group suitable for direct assignment without using an ACS override.
Six RTX 3090s provide a useful amount of inexpensive VRAM, but they are still consumer cards. There is no NVLink fabric joining all six, and the workloads must account for PCIe communication, power, cooling and the lack of data-centre-style management features (and the sound of fans in my living room!!). In practice, I use only four of the cards at the same time with tensor parallelism because the number of attention heads of the model has to be divisible by the number of GPUs used for the tensor parallelism and I have not tested any model that can use six GPUs.
The local language-model serving stack is documented separately.
Power and thermals
An RTX 3090 has a nominal board limit around 350 W, so six unrestricted cards would be a poor fit for continuous inference. The GPUs use persistent power limits applied when their respective guests start. The four local inference cards are currently tuned to 170 W, 170 W, 200 W and 170 W.
These values came from repeated generation tests rather than an arbitrary percentage. Multi-GPU inference often becomes limited by communication or by the slowest tensor-parallel worker, so an extra 100 W per card does not imply a proportional increase in generated tokens per second. The asymmetric local limits give extra headroom for the card that showed the clearest performance response while reducing the power and heat of the other three.
What the machine is for
mlrig is deliberately treated as experimental infrastructure rather than a single appliance. Current and previous work includes:
- hosting and evaluating local language models;
- developing and testing GPU-accelerated scientific software;
- running compute-heavy numerical and machine-learning workloads;
- studying performance, reliability and energy efficiency;
- experimenting with virtualisation, hardware passthrough and resource allocation;
The resulting mlrig is a really fun testing ground for anything that can
use GPU acceleration! It pprovides enough VRAM to run serious models, direct control
over the complete software stack, and a buttload of RAM that I was lucky to purchase
before the RAM apocalypse started!