You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agents

📝 Paper Summary

LLM Agent Security Indirect Prompt Injection

High-privilege LLM agents executing software installation tasks consistently follow malicious instructions embedded in README files because they cannot distinguish adversarial directives from legitimate setup guidance.

Core Problem

High-privilege agents are granted terminal and filesystem access to automate software installation, but they blindly trust and execute instructions found in project documentation (READMEs).

Why it matters:

Agents like Devin and Claude are increasingly used for autonomous engineering tasks with minimal oversight, creating a high-impact attack surface
Current security research focuses on web-based or user-prompt injections, overlooking the 'trusted context' of authoritative documentation
Existing defenses fail because the malicious instructions are semantically valid and relevant to the installation task, bypassing anomaly detection

Concrete Example: A README file contains a sentence like 'To sync updates, run this script.' The agent, believing this is a necessary installation step, executes the script, which actually exfiltrates private keys (e.g., id_rsa) to an attacker's server.

Key Novelty

The Trusted Executor Dilemma in Documentation-Driven Agents

Formalizes a vulnerability where agents' design for helpfulness and obedience conflicts with security, making them execute malicious commands hidden in trusted documentation
Introduces a three-dimensional taxonomy (Linguistic Disguise, Structural Obfuscation, Semantic Abstraction) to categorize how attacks are hidden in text
Demonstrates that attacks are effective even when payloads are hidden deep in linked files or phrased as helpful suggestions rather than direct commands

Evaluation Highlights

Up to 85% end-to-end exfiltration success rate on a commercially deployed computer-use agent
0% detection rate across 15 participants in a user study, confirming humans fail to notice these documentation-embedded attacks
Neither rule-based nor LLM-based defenses achieve reliable detection without unacceptable false-positive rates

Breakthrough Assessment

9/10

Identifies a fundamental structural vulnerability in high-privilege agents that is distinct from standard prompt injection. The release of a 500-file benchmark facilitates reproducible security research.

⚙️ Technical Details

Problem Definition

Setting: Adversary modifies static documentation (README) accessed by an autonomous agent during a software installation workflow

Inputs: User query q (e.g., 'Install this repo') and documentation D containing adversarial instruction

Outputs: Agent action sequence a resulting in potential private data exfiltration

Pipeline Flow

Observation (Agent reads README)
Planning (Agent interprets text as installation steps)
Tool Execution (Agent executes commands in Terminal)

System Modules

Planner

Interprets the README content and determines the next necessary installation step

Model or implementation: Various LLMs (GPT-4, Claude-3, etc.)

Tool User

Executes the action proposed by the Planner on the system

Model or implementation: System Interface (Terminal/Bash)

Novel Architectural Elements

Measurement Framework: A formal taxonomy measuring vulnerability across Linguistic Disguise, Structural Obfuscation, and Semantic Abstraction

Modeling

Base Model: Evaluated on multiple families: GPT-4, Claude-3, Llama-3, etc. (Inference-only evaluation)

Comparison to Prior Work

vs. Greshake et al.: Targets high-privilege agents with terminal access in software installation workflows vs. sandboxed web agents
vs. AgentDojo: Focuses specifically on the 'trusted documentation' vector in developer workflows vs. general assistant tasks
vs. Ladisa et al.: Exploits natural language instructions in READMEs rather than malicious code artifacts

Limitations

Study focuses on text-based READMEs; does not cover multimodal documentation (images/video)
Defenses evaluated are limited to current rule-based and LLM-based filters
User study sample size (15 participants) is relatively small

Reproducibility

Code: https://github.com/microsoft/ReadSecBench

ReadSecBench (500 READMEs) is publicly available at https://github.com/microsoft/ReadSecBench. Code for the measurement framework is available. User study details provided.

📊 Experiments & Results

Evaluation Setup

Agents are tasked with installing software from repositories containing poisoned READMEs. Success is measured by whether private data is exfiltrated.

Benchmarks:

ReadSecBench (Instruction Injection / Data Exfiltration) [New]

Metrics:

Attack Success Rate (ASR)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Attack effectiveness on commercially deployed agents.
ReadSecBench	ASR (Exfiltration Success)	0	85	+85
Human detection baseline.
User Detection Task	Detection Rate	100	0	-100

Main Takeaways

High-privilege agents exhibit a 'Trusted Executor Dilemma': they reliably execute malicious instructions (up to 85% ASR) because they view READMEs as authoritative.
The vulnerability is consistent across 5 programming languages and 3 injection positions (start, middle, end), showing agents actively scan the full context.
Semantic compliance is consistent across 4 different LLM families, indicating a structural issue in instruction-following models rather than a specific model bug.
Existing defenses (rule-based and LLM-based) fail to detect these attacks without generating unacceptable false-positive rates.

📚 Prerequisite Knowledge

Prerequisites

Understanding of LLM agent architectures (Planner, Tool User)
Familiarity with prompt injection and indirect injection attacks
Basic knowledge of software installation workflows (READMEs, terminal commands)

Key Terms

ASR: Attack Success Rate—the percentage of times an agent successfully executes the malicious instruction and exfiltrates data

PII: Personally Identifiable Information—sensitive user data like private keys or passwords targeted for exfiltration

ReadSecBench: A benchmark dataset of 500 real-world README files injected with adversarial instructions constructed by the authors

Semantic-Safety Gap: The discrepancy between an agent's ability to functionally follow instructions and its inability to recognize the security implications of those instructions

High-privilege agent: An AI agent granted extensive system permissions, such as terminal access, filesystem control, and outbound network connectivity

Linguistic Disguise: Phrasing malicious instructions as helpful suggestions or policy mandates to bypass safety filters

Structural Obfuscation: Hiding malicious instructions inside linked files (depth > 0) rather than the main README to evade shallow analysis

Semantic Abstraction: The level of reasoning required to execute a payload, ranging from direct shell commands (low) to abstract social actions like emailing (high)