AI Lab · by Ren Suzuki · 2026-10-14
Tried running a 70b model fully on CPU just to see how bad it would be — results were actually not terrible
Got about 1.2 tokens/sec on a 32 core server with enough ram, which sounds awful until you realize it means you can run frontier-ish open models on hardware that costs way less than a gpu rig. For batch jobs where latency doesnt matter this might actually be a reasonable tradeoff. Anyone else experimenting with cpu-only inference seriously?
0 sparks · 9 comments
Comments
Olena Shevchenko · 0 likes
for batch overnight jobs this is a totally valid strategy, people sleep on cpu inference way too much
Felix Weber · 0 likes
1.2 tok/s would drive me insane for anything interactive but for a nightly summarization job, sure why not
Irene López · 0 likes
what quantization did you use, that number seems high for full precision on cpu
Valeria López · 0 likes
q4_k_m, full precision would be way slower, almost unusable honestly
Isha Menon · 0 likes
ram bandwidth is usually the real bottleneck here not core count, did you check memory bandwidth utilization
Kabir Khan · 0 likes
ddr5 helped a lot when i tried something similar, almost doubled my throughput over ddr4
Jisoo Park · 0 likes
this is a great option for privacy-sensitive use cases where you cant send data to an api and dont have gpu budget
Hamza Malik · 0 likes
love seeing these weird hardware experiments, more people should post the 'bad' results not just the wins
Paper River · 0 likes
tempted to try this on my old homelab server now, thanks for the writeup