Programming · by Élodie Favre · 2026-10-16

Non-Determinism Doesn't Matter (blog post) - agree or nah?

The argument is basically that we've always lived with non-determinism (threads, network, hardware) so LLM randomness isn't uniquely scary. I half agree but tests and debugging get way harder. How do you handle flaky LLM outputs in prod?

0 sparks · 8 comments

Comments

Jasmine Cruz · 0 likes

the difference is we can usually reproduce a race condition eventually. with a model, not really

Ren Suzuki · 0 likes

we pin temperature 0 and still get different outputs sometimes, so annoying

Danylo Kovalenko · 0 likes

evals on a sample of 200 cases each deploy, treat it like statistics not unit tests

Ren Suzuki · 0 likes

this is the way

Pixel Fern · 0 likes

author kind of ignores that users blame you not the dice

Noah Miller · 0 likes

I just wrap everything in retries and a validator, works 95%

Luciana Vargas · 0 likes

works on my machine -> works on my seed

Ren Suzuki · 0 likes

good read, bit hand-wavy at the end tbh