CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions

📝 Paper Summary

Benchmark datasets Multi-call tool use with flexible plan

CToolEval is a fine-grained benchmark with 398 real-world APIs to evaluate the tool invocation and task completion capabilities of LLM-powered agents in Chinese societal applications.

Core Problem

Existing tool evaluation benchmarks often rely on advanced models like GPT-4 for scoring, which introduces bias and fails to accurately evaluate dynamic, real-time answers.

Why it matters:

Current benchmarks struggle to assess dynamic real-time answers (e.g., weather forecasts) where the ground truth changes.
Relying on LLM-as-a-judge (like GPT-4) introduces subjective bias and scalability issues.
A lack of rigorous Chinese-specific tool-learning benchmarks limits the development of localized LLM agents.

Concrete Example: For a real-time query like 'highest temperature tomorrow', neither humans nor GPT-4 can judge correct answers in advance; CToolEval solves this by dynamically extracting real-time answers from API observations.

Key Novelty

CToolEval Benchmark and Dynamic Evaluation Framework

Categorizes interactions into fixed-answer, open-ended, real-time, and operational types to apply tailored, accuracy-based metrics.
Evaluates real-time queries dynamically by storing API links and parameters to fetch live reference answers, bypassing LLM-as-a-judge limitations.
Implements a rollback function for operational queries to reverse database changes after successful API calls, ensuring repeatable evaluation.

Architecture

The framework of CToolEval covering interactions in real-world scenarios across 27 Apps and the evaluation of LLMs' tool invocation and task completion capabilities.

Evaluation Highlights

GPT-3.5-turbo achieves a 61.94% overall tool matching rate, significantly outperforming the best Chinese open-source model (InternLM-chat-20B at 21.02%).
In task completion, GPT-3.5-turbo scores 45.45% overall, while InternLM-chat-20B achieves only 15.20%.
Fine-tuning Qwen-7B-chat with QLoRA on CToolEval improves its overall tool matching rate from 0.00% to 63.60%, demonstrating the dataset's utility for training.

Breakthrough Assessment

7/10

Provides a rigorous, real-world benchmark for Chinese LLMs with a novel dynamic evaluation for real-time queries, though limited by API volatility.

⚙️ Technical Details

Problem Definition

Setting: Evaluating LLMs as agents interacting with real-world Chinese Apps via APIs.

Inputs: Natural language instructions (single-tool or multi-tool queries) reflecting real-life applications.

Outputs: Agent actions (API invocations with parameters) and final answers (task completion responses).

Pipeline Flow

Input Processing: Retriever → LLM Agent (Tool Invocation)
Tool Execution → Generation: LLM Agent (Task Completion)

System Modules

Retriever

Selects the most relevant tools from a pool of 398 APIs based on the semantic similarity between the user query and tool descriptions.

Model or implementation: Sentence-BERT based on Chinese-bert-base

LLM Agent (Tool Invocation)

Decomposes the query into sub-tasks and selects the correct tool and parameters from the retrieved candidates.

Model or implementation: Evaluated LLMs (e.g., GPT-3.5-turbo, InternLM, Qwen)

LLM Agent (Task Completion)

Integrates the results returned by the invoked APIs to deduce the final answer.

Model or implementation: Evaluated LLMs (e.g., GPT-3.5-turbo, InternLM, Qwen)

Novel Architectural Elements

Dynamic assessment module for real-time queries that fetches live API data to construct reference answers on the fly.
Rollback function in the evaluation framework that automatically performs the reverse operation of successful API calls to maintain backend database consistency.

Modeling

Base Model: Various (GPT-3.5-turbo, Qwen-7B/14B/72B-Chat, InternLM-chat-7B/20B, Baichuan2-7B/13B-Chat, Chinese-Alpaca-2)

Training Method: Parameter-efficient fine-tuning (QLoRA and LoRA) on CToolEval training set

Adaptation: QLoRA / LoRA

Training Data:

Training set of CToolEval (6,816 queries total in benchmark, split not explicitly detailed)

Compute: Not reported in the paper

Comparison to Prior Work

vs. ToolBench: Focuses on Chinese societal applications and uses objective accuracy metrics rather than GPT-4 preference.
vs. ToolEyes: Solves the evaluation problem for dynamic real-time answers instead of relying on advanced models for scoring.
vs. API-Bank: Covers a broader range of real-world apps (27 vs 5) and includes real-time/operational evaluation mechanisms [not cited in paper].

Limitations

API interfaces may become faulty or change over time, which could affect the reproducibility of evaluation results.
Due to GPT-4 token usage limits and costs, the evaluation was restricted to a specific subset of representative Chinese LLMs rather than all available models.
Fine-tuning models on the dataset increased their tendency to hallucinate API responses rather than performing actual invocations.

Reproducibility

Code: https://github.com/tjunlp-lab/CToolEval

Code and data are publicly available on GitHub (https://github.com/tjunlp-lab/CToolEval). Exact hyperparameter configurations for fine-tuning (e.g., learning rates, batch sizes) are not explicitly reported in the text.

📊 Experiments & Results

Evaluation Setup

Zero-shot evaluation of LLMs on tool invocation (Tool Matching Rate) and task completion (Accuracy, GPT-4 Score, Execution Success Rate) across 4 data types.

Benchmarks:

CToolEval (Real-world API Interactions) [New]

Metrics:

Tool Matching Rate
Accuracy
GPT-4 score
Execution success rate
NDCG@K
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Main comparison of Tool Invocation capabilities across different LLMs.
CToolEval	Tool Matching Rate (All)	21.02	61.94	40.92
CToolEval	Tool Matching Rate (All)	1.45	15.82	14.37
Main comparison of Task Completion capabilities across different LLMs.
CToolEval	Task Completion (All)	15.20	45.45	30.25
Impact of fine-tuning open-source models on the CToolEval dataset.
CToolEval	Tool Matching Rate (All)	0.00	63.60	63.60
CToolEval	Task Completion (All)	5.87	33.61	27.74

Main Takeaways

GPT-3.5-turbo far surpasses Chinese open-source LLMs in both tool invocation and task completion, though it struggles with temporal reasoning (e.g., handling 'tomorrow').
Chinese LLMs often exhibit severe hallucination issues, where they fail to pass correct parameters but pretend to have received output from the API server and fabricate answers.
Providing gold APIs (bypassing the retriever) significantly enhances tool invocation capabilities for weaker models like InternLM, whereas stronger models like GPT-3.5-turbo are less affected by retrieval noise.
Fine-tuning models on CToolEval dramatically improves their tool matching and task completion metrics, but intensifies the tendency to hallucinate API responses rather than executing actual calls.

📚 Prerequisite Knowledge

Prerequisites

Understanding of large language models (LLMs)
Familiarity with API (Application Programming Interface) calls
Basic knowledge of tool-augmented generation or agentic workflows

Key Terms

Tool Matching Rate: An evaluation metric calculated as the number of tools correctly used divided by the total number of tools that should be used.

Fixed-answer: Queries with a static, fill-in-the-blank response.

Open-ended: Queries requiring detailed responses without a predetermined format.

Operational: Queries necessitating the execution of tool-based APIs for specific tasks, evaluated by execution success rate.

Real-time: Queries involving answers that are dynamic over time, evaluated via dynamic assessment.

ReAct: Reasoning and Acting—a paradigm where the agent interleaves reasoning traces with task-specific actions.

Sentence-BERT: A modification of the pre-trained BERT network that uses siamese and triplet network structures to derive semantically meaningful sentence embeddings.

NDCG: Normalized Discounted Cumulative Gain—a metric used to measure the quality of a set of search results.

QLoRA: Quantized Low-Rank Adaptation—a parameter-efficient fine-tuning method that uses quantized weights to reduce memory usage.

LoRA: Low-Rank Adaptation—a technique to fine-tune large models efficiently by updating only a small set of low-rank matrices.

SFT: Supervised Fine-Tuning—training a model on labeled examples to adapt it to specific tasks or instructions.