Open Agent Specification (Agent Spec): A Unified Representation for AI Agents

📝 Paper Summary

Agent Frameworks Interoperability Standards Agent Evaluation

Agent Spec is a declarative, framework-agnostic configuration language that allows AI agents to be defined once and executed or evaluated consistently across diverse runtimes like LangGraph and AutoGen.

Core Problem

The proliferation of incompatible agent frameworks (LangChain, AutoGen, CrewAI) has created fragmentation where agents defined in one system cannot be executed, reused, or compared in another due to differing abstractions and execution semantics.

Why it matters:

Organizations cannot reuse agent designs developed by different teams, leading to redundant development and lock-in to specific frameworks
Prototyping-to-deployment is slowed when porting agents from experimental stacks to enterprise runtimes requires complete rewrites
Reliable evaluation is impossible because there is no standardized harness to compare agent behavior consistently across different execution environments

Concrete Example: A multi-agent workflow defined in AutoGen using conversational patterns cannot be executed in LangGraph's stateful graph runtime without a complete reimplementation, making it impossible to directly compare their performance on benchmarks like BIRD-SQL.

Key Novelty

ONNX for AI Agents

Proposes a 'define-once, run-anywhere' standard (Agent Spec) that separates the logical definition of an agent (nodes, flows, tools) from its specific runtime implementation
Introduces a layer of 'Runtime Adapters' that translate the declarative Agent Spec JSON into framework-specific primitives (e.g., converting a Spec Flow into a LangGraph StateGraph) dynamically

Architecture

The position of Agent Spec within the broader AI ecosystem, differentiating it from resource protocols (MCP) and communication protocols (Agent2Agent)

Breakthrough Assessment

8/10

Addresses a critical infrastructure gap in the fragmented agent ecosystem. While not a modeling breakthrough, it enables necessary standardization for reproducibility and evaluation, similar to ONNX for neural networks.

⚙️ Technical Details

Problem Definition

Setting: Specification and execution of agentic workflows across heterogeneous runtime environments

Inputs: Declarative Agent Spec configuration (JSON/YAML)

Outputs: Executed agent workflow (actions, tool calls, final responses) within a specific target framework

Pipeline Flow

Agent Definition (Python SDK / YAML)
Serialization (Agent Spec JSON)
Runtime Adaptation (Conversion to Target Framework)
Execution (Target Runtime)

System Modules

Agent Spec Language

Defines the schema for Agents, Flows, Nodes, Tools, and connections

Model or implementation: Declarative JSON Schema

Runtime Adapter

Translates Agent Spec components into framework-specific objects

Model or implementation: Adapter Logic (e.g., for LangGraph, CrewAI)

Evaluation Harness

Standardizes assessment of agent behavior across runtimes

Model or implementation: Measurement Protocol

Novel Architectural Elements

Decoupling of agent definition from execution runtime via a standardized intermediate representation
Component-based architecture where 'Agents' and 'Flows' are distinct, composable primitives with explicit input/output contracts defined via JSON Schema

Modeling

Base Model: Framework-agnostic (supports connecting to any LLM via config)

Compute: Not reported in the paper

Comparison to Prior Work

vs. MCP: MCP handles the 'tools' layer; Agent Spec handles the 'behavior/logic' layer (Flows, Nodes, Planning)
vs. Agent2Agent: A2A handles communication protocols; Agent Spec handles the internal structural definition of the agent itself
vs. LangChain/LangGraph: These are specific runtimes; Agent Spec is an abstraction layer *above* them that can compile down to them

Limitations

The paper text provided does not include the quantitative results tables, limiting verification of the performance claims
Adapters rely on the target framework supporting the necessary primitives; features unique to one framework may not map cleanly to Agent Spec
The specification adds a layer of abstraction which may introduce overhead compared to native implementation

Reproducibility

Code: https://github.com/oracle/wayflow

All artifacts are publicly available: Reference runtime WayFlow (github.com/oracle/wayflow), Python SDK PyAgentSpec (pypi.org/project/pyagentspec/), and adapters for LangGraph/AutoGen/CrewAI.

📊 Experiments & Results

Evaluation Setup

Execution of standardized agent workflows across four distinct runtimes to compare performance and consistency

Benchmarks:

SimpleQA Verified (Question Answering)
Tau^2-Bench (Complex Reasoning)
BIRD-SQL (Text-to-SQL / Database Querying)

Metrics:

Accuracy
Query Time / Latency
Stability
Statistical methodology: Not explicitly reported in the paper

Experiment Figures

The Agent Spec ecosystem workflow, from SDK authoring to runtime execution

Main Takeaways

Qualitative Result: LangGraph and WayFlow (reference runtime) deliver reliable accuracy and query time stability across datasets.
Qualitative Result: AutoGen outperforms on complex reasoning tasks (e.g., Tau^2-Bench) but suffers from increased latency compared to other runtimes.
Qualitative Result: CrewAI's multi-agent coordination approach yields limited accuracy gains on these benchmarks despite having longer query times.
The ability to run the exact same agent definition across 4 frameworks confirms the feasibility of Agent Spec as a portability standard.

📚 Prerequisite Knowledge

Prerequisites

Understanding of AI agent architectures (Tools, Planning, Memory)
Familiarity with DAG (Directed Acyclic Graph) execution models
Knowledge of serialization formats (JSON, YAML)

Key Terms

Agent Spec: A declarative, framework-agnostic language for defining AI agents and workflows, designed to be portable across different runtimes

ONNX: Open Neural Network Exchange—a standard for representing machine learning models that allows them to be transferred between frameworks (used here as an analogy)

MCP: Model Context Protocol—an existing standard by Anthropic for resource and tool provisioning, which Agent Spec complements by handling agent behavior

Runtime Adapter: A software component that maps Agent Spec primitives (Agents, Flows, Nodes) to the specific execution constructs of a target framework (e.g., LangGraph, AutoGen)

WayFlow: The reference runtime implementation provided by the authors that natively supports Agent Spec execution

JSON Schema: A vocabulary that allows you to annotate and validate JSON documents, used by Agent Spec to define component properties and input/output structures