AI Box on RTX 5090
A compact AI workstation for development, local LLMs, RAG and first production scenarios — without a data-centre footprint.
What it is
An AI Box built on the NVIDIA GeForce RTX 5090 is a workstation with 32 GB GDDR7 of GPU memory, sized for a single developer or a small team. It covers the path from experimentation to a small production loop: running open-weight models, local RAG, developer assistants, and agent prototyping.
When it fits
- Individual development and RAG over a small document set
- Prototyping AI agents and assistants
- Local coding assistant for a team
- Isolated perimeter for sensitive data
What to keep in mind
- Not sized for many concurrent users
- Models above ~32–70B run, but with limits on speed and context
- Single GPU — a single point of failure without HA
- For high load and vLLM, server-class GPUs (RTX PRO, L40S) are a better fit
Frequent questions
Can a 70B model run on an RTX 5090?+
Technically yes, at 4-bit quantisation. In practice, speed and context will be constrained, and concurrent users will feel the limits. For 70B we recommend 48–96 GB of VRAM.
Is the RTX 5090 suitable for fine-tuning?+
For LoRA and QLoRA up to ~13B — yes. For full fine-tuning of larger models, a server-class GPU with more memory is required.
How many users can one RTX 5090 AI Box serve?+
1–5 users on models between 14B and 32B. For 15+ users, consider RTX PRO or L40S.
Let’s discuss your task
Tell us what the AI contour needs to solve — we will match the configuration, infrastructure and next step.