📝 Paper Summary
Web agents
Privacy and Safety
SPILLage is a framework that characterizes and measures how web agents unintentionally leak task-irrelevant user information through both text inputs and behavioral patterns (clicks, scrolls) on live websites.
Core Problem
When users delegate tasks to web agents with access to personal resources, agents often disclose task-irrelevant sensitive information to third-party websites.
Why it matters:
- Prior work focuses on adversarial text leakage (e.g., prompt injection), missing non-adversarial leakage inherent in normal task execution
- Existing evaluations ignore 'behavioral' oversharing (e.g., clicking specific filters) which can reveal sensitive attributes even without text entry
- Current privacy tools treat leakage as binary, failing to capture 'implicit' leakage where attributes are inferable but not stated verbatim
Concrete Example:
A user asks an agent to 'find glucose test strips' and provides emails revealing they are divorced. The agent might unnecessarily type 'glucose test strips for divorced women' (explicit content oversharing) or click a 'Single Mom' filter (behavioral oversharing), revealing the divorce status to the shopping platform despite it being irrelevant to the product search.
Key Novelty
SPILLage Taxonomy & Benchmark
- Formalizes a 2x2 taxonomy of agentic oversharing based on Channel (Content vs. Behavioral) and Directness (Explicit vs. Implicit), capturing unique risks like navigation patterns
- Introduces a live-website benchmark on Amazon and eBay where tasks blend relevant and irrelevant user context to test agent discretion
- Implements a step-level LLM-Judge that audits every agent action (clicks, types, scrolls) against the principle of contextual integrity to detect oversharing
Architecture
The SPILLage framework workflow: User inputs resources/request -> Agent interacts with Web -> Passive Observer records Trace -> Auditor detects Oversharing.
Evaluation Highlights
- Behavioral oversharing dominates content oversharing by 5x, revealing a major blind spot in text-only privacy evaluations
- A gpt-4o-based agent committed 1,151 explicit behavioral oversharing events on Amazon alone across the benchmark tasks
- Removing task-irrelevant information from the prompt improves task success by up to 17.9%, showing that privacy and utility are aligned rather than conflicting
Breakthrough Assessment
8/10
Significant contribution by identifying 'behavioral' leakage as a primary risk vector for web agents, moving beyond simple text string matching. The alignment of privacy and utility is a strong practical finding.
