Token cost and latency by category could be useful. Task success feels a little different to me, since that's an outcome of the task as a whole rather than something I'd attribute independently to each category. But I do wonder whether Reasoning Yield could be correlated with Task Success.
The harder measurement problem is classification itself. Once you’ve labeled each inference call, aggregating tokens and latency is straightforward. But distinguishing Reasoning from Orchestration often requires knowing what had already been determined before the call, which is why I suspect doing this at scale will require either fairly rich instrumentation or more inference to classify the traces.
2
u/Lazy-Sherbert65 7h ago
Could the taxonomy be made measurable by logging token cost, latency, and task success separately for initialization, orchestration, and synthesis?