📝 Paper Summary
Agent Evaluation
Agent Usability
The primary barrier to LLM agent adoption is not technical capability but low Agentic ROI in mass-market domains, where interaction costs often exceed the time required for manual execution.
Core Problem
Current LLM agents are technically capable but lack usability in mass-market applications because the cognitive load and time required to prompt them often exceed the effort of using existing optimized UIs.
Why it matters:
- Mass-market deployment is sparse (e.g., OpenAI Plus has ~10M users vs. recommendation engines with 700M+), indicating a huge untapped market
- Pure performance optimization ignores the socio-technical ecosystem: if an agent takes longer to prompt than the task takes to do manually, users won't use it regardless of accuracy
- There is a critical disconnect between high-demand domains (e-commerce, office work) and high-ROI domains (coding), creating a usability gap
Concrete Example:
In office work, describing a task like 'Schedule a 30-minute sync with the team next week avoiding Thursdays' requires back-and-forth disambiguation with an agent. In contrast, the same task can be completed in seconds using a modern calendar UI, resulting in negative or negligible time savings for the agent approach.
Key Novelty
Agentic Return on Investment (Agentic ROI)
- Formalize usability as a mathematical function of Information Gain and Time Savings divided by Cost, rather than just accuracy or success rate
- Propose a 'Zigzag' development trajectory: first scale up for capability (Info Gain/Time Savings), then scale down for cost efficiency, similar to processor evolution
- Introduce 'Sleep-time compute' as a strategy to improve quality without interaction latency by letting agents process user preferences and simulate paths during idle time
Architecture
Strategic roadmap for optimizing Agentic ROI, showing a Zigzag path
Evaluation Highlights
- Strong positive linear correlation (r=0.95) between user-reported usability ratings and calculated Agentic ROI across 5 domains
- Identified inverse relationship between market demand and Agentic ROI: high-demand areas like E-commerce and Personal Assistance currently show low Agentic ROI
- Survey of 34 users reveals that agent latency is not the primary barrier; prompting overhead and lack of net time savings are the main hurdles
Breakthrough Assessment
8/10
Provides a crucial theoretical re-framing of the agent usability problem. While not introducing a new model architecture, the Agentic ROI metric and Zigzag roadmap offer a strategic guide for the field.
