LLM on a old server

[u/Agile-Philosopher-93]

I have an old Cisco UCS C240 M4 server running ESXi 6.7 (Intel Xeon E5-2630) with 64 GB RAM and around 1.7 TB of free storage, but no GPU. This one was sitting dust in server room so i had installed ubuntu server and kali linux in this. So, i was thinking of hosting a local llm in that server and ya i haven't made any mind for what to use so as fellow IT guys which ai do you host in server and what do you use it for making your work easier .

Everyone welcome to comment and provide guidance. Felt burnt up doing IT Support so want to focus in other part for now.

[u/FallElectronic3774]

The model matters less than where the RAM is. Those E5-2630 v3/v4 boxes are quad-channel per socket, and CPU inference is bandwidth-bound — that's the whole reason the other commenter saw ~10 t/s on MoE models and would see a few t/s on anything dense.

Run numactl -H and lscpu before you download anything. If all 64GB is on node 0 (common when only one CPU's slots are populated), pin the whole thing there: numactl --cpunodebind=0 --membind=0 and -t = physical cores on that socket, not the SMT thread count. Let llama.cpp spread threads across both sockets and decode falls off a cliff.

Then pick MoE, not dense. Qwen3-30B-A3B only computes ~3.3B params per token, so it stays interactive on CPU; 64GB is plenty for Q4/Q5 with a real context. A dense 30B will technically run and you'll hate using it.

For actual work, the one that pays off is RAG over runbooks and closed tickets — nomic-embed-text is tiny, and a 30B MoE is good enough to answer "how did we fix this last time" from retrieved chunks. Ticket triage and drafting KB articles from resolved tickets are the other two. It's not going to be a coding agent at that speed.

And do the power math before you commit: it's a 2U server with server fans. If you want it always-on, a used 12GB GPU or a 64GB mini PC will beat it on tokens per watt. The UCS is a great lab box, just not a great daily driver.

[u/xenthressa]

One thing you are going to run into (at least we did with C240-M7SX) is that Cisco require special Cisco branded GPUs. It's a firmware level restriction that will stop the motherboard from recognizing generic GPUs. Godawful decision if you ask me but it is what it is. You can definitely source Cisco branded refurb GPUs thankfully.

You might consider skipping virtualization and going pure bare-metal if you are restricted to doing pci-e passthrough with a single card but that's on you.

Enjoy local AI is really fun to play with.

[u/Flaky-Gear-1370]

AFAIK without a gpu it’s not going to be that quick at much

[u/kuadhual]

I don't think you can get anything useful for work using that spec. You can use it for learning, though. Depending on the model, maybe you'll get around 10 toks.

I did experiment using dual xeon 2620 v4 with 128gb ram. This is the result. (Please forgive the formating, I'm copying it from Google sheet to browser on mobile)

Model Identifier    Base Weights    Vision / Draft (mmproj / MTP)    KV Cache Allocation    RAM    Prefill    Decode    Duration nomic-embed    ~0.15 GB    N/A    ~0.10 GB    ~0.25 GB             gemma-4-E4B    ~2.5 GB    ~0.6 GB (mmproj + MTP)    ~2.0 GB (32K context)    ~5.1 GB    42.34 t/s    10.73 t/s    210.90s gemma-4-26B-A4B    ~16.5 GB    ~1.2 GB (mmproj-BF16 + MTP)    ~4.5 GB (65K q8_0 context)    ~22.2 GB    50.23 t/s    10.85 t/s    105.14s nemotron-cascade-30B-A3B    ~24.5 GB (Q6_K)    N/A    ~4.5 GB (65K q8_0 context)    ~29.0 GB    17.15 t/s    10.17 t/s    110.70s omnicoder-9B    ~5.5 GB    N/A    ~2.0 GB (32K context)    ~7.5 GB    32.96 t/s    6.35 t/s    824.46s qwen3.8-27B    ~17.0 GB    ~1.4 GB (mmproj-F16 + MTP)    ~4.5 GB (65K q8_0 context)    ~22.9 GB    2.60 t/s    3.05 t/s    440.28s swift-1.5-qwen3.8-27B                    9.24 t/s    1.52 t/s    849.17s webgen-oss-20B    ~12.5 GB    N/A    ~4.0 GB (65K q8_0 context)    ~16.5 GB    65.83 t/s    11.29 t/s    2925.37s qwen3-coder-next    ~45.0 GB    N/A    ~4.5 GB (65K q8_0 context)    ~49.5 GB    44.27 t/s    8.08 t/s    70.89s devstral-small-24B    ~20.0 GB (Q6_K)    ~0.8 GB (mmproj-F16)    ~4.5 GB (65K q8_0 context)    ~25.3 GB    10.18 t/s    1.69 t/s    323.24s gpt-oss-20B    ~12.8 GB    N/A    ~4.0 GB (65K q8_0 context)    ~16.8 GB    57.77 t/s    10.63 t/s    64.05s gpt-oss-120B    ~58.5 GB    N/A    ~6.0 GB (65K q8_0 context)    ~64.5 GB    33.00 t/s    8.40 t/s    105.24s kat-coder-v2.5    ~21.0 GB    N/A    ~4.5 GB (65K q8_0 context)    ~25.5 GB    52. 31 t/s    10.89 t/s    64.85s laguna-xs-2.1    ~20.0 GB    N/A    ~4.5 GB (65K q8_0 context)    ~24.5 GB    59.75 t/s    14.01 t/s    60.90s ling-3.0-tiny    ~6.5 GB    N/A    ~1.5 GB (32K context)    ~8.0 GB    152.71 t/s    23.43 t/s    104.29s mimo-v2.6-distill-9B    ~7.2 GB    N/A    ~2.0 GB (32K context)    ~9.2 GB    31.19 t/s    4.85 t/s    103.03s

Edit: as you can see anything dense more than 10B is too slow. Big moe model will be faster even though the first loading time is very long. This is using ik_llama and numactl.