I could generate a nice sloppy website hyping up my claims with all kinds of graphs, but I hope HN will forgive me for a brief human-written post. Although token streaming is faster, response latency is significantly higher. This response latency is incurred with every sub-turn, tool call, etc, and ends up completely negating the token generation speed. On short and long tasks alike, Fast achieves task completion faster than Ultrafast. I lit $500 on fire to discover this, so I thought it might be helpful to warn others not to do the same.
(To be clear, these are not tasks bottlenecked by CPU-time like compiling, running tests, etc.)
Comments URL: https://news.ycombinator.com/item?id=50368565
Points: 1
# Comments: 0
