AI Lab · by Minji Kim · 2026-10-15
I ran the same 20 coding tasks through Claude Opus 5.5, GPT-6 Astra, and Gemini 4 Argon — here's what actually broke
Spent the weekend building a tiny eval harness instead of touching grass. 20 tasks: refactors, a few off-by-one bug hunts, one genuinely nasty concurrency bug, and some greenfield scaffolding. None of them got everything right. Opus 5.5 was the most conservative and asked clarifying questions more (good for prod code, annoying for quick iteration). Astra was fastest but hallucinated an import path twice. Argon nailed the concurrency bug on the first try, which surprised me. Anyone else doing structured comparisons like this instead of vibes-based picking?
0 sparks · 10 comments
Comments
Can Arslan · 0 likes
this matches my experience, opus models tend to be more cautious which is great until you just want it to shut up and write the function
Noah Miller · 0 likes
did you control for prompt wording across the three? thats usually where these comparisons fall apart, small wording changes shift results alot
Mehdi Benali · 0 likes
concurrency bugs being nailed by argon is interesting, i wouldve put money on opus for that honestly
Sleepy Coder · 0 likes
yeah that surprised me too, might just be luck on one sample tho, i wanna see this at n=100 before i trust it
Ananya Rao · 0 likes
we do this exact thing at work quarterly, rotate models through a fixed eval set. genuinely the only way to cut through marketing
Tola Ibrahim · 0 likes
importing a path that doesnt exist is such a classic LLM move lol, happens to me weekly no matter the model
Noé Petit · 0 likes
can you share the actual task list? would love to replicate this with the models we have access to internally
Sleepy Mango · 0 likes
this is the kind of post this sub needs more of honestly, less hype more data
Ren Suzuki · 0 likes
20 tasks is a good start but careful drawing big conclusions from such a small n, variance is real at this scale
Rohan Mehta · 0 likes
lol i did somthing similar and my main finding was that all three models are equally bad at admitting when they dont know something