Server for a local LLM
Selection of GPU, memory and inference runtime for a specific model, user count and required context length.
What it is
This is not a fixed product but an engineering sizing exercise. We start from the model — size, architecture, quantisation, context — and derive hardware and runtime requirements from it. That eliminates both overpaying for unused power and underestimating the load.
When it fits
- You have a specific model and a clear workload
- You need predictable latency and context
- Sensitive data must stay inside the perimeter
- Multiple models are planned for different tasks
What to keep in mind
- Not every task needs a local model — some are cheaper via API
- Quantisation affects quality; the trade-off is chosen per task
- Concurrency and context length drive VRAM as much as model size
- Runtime matters: the same model on Ollama and vLLM performs differently
Frequent questions
Can a 70B model run on one GPU?+
At 4-bit quantisation — yes, with limits on speed, concurrency and context length. For comfortable 70B operation we recommend 96 GB VRAM and above.
Ollama or vLLM?+
Ollama for quick start and light load. vLLM for production: higher throughput, better with concurrency and long context. Combining both is often the right answer.
Which model is best for Russian?+
Qwen, DeepSeek and Llama derivatives work well in Russian. Final choice follows testing on your own data, not benchmarks.
Let’s discuss your task
Tell us what the AI contour needs to solve — we will match the configuration, infrastructure and next step.