AI Phenomenology for Understanding Human-AI Experiences Across Eras

📝 Paper Summary

User Modeling Human-AI Interaction Qualitative Evaluation Methods

The authors propose 'AI Phenomenology'—a research stance focused on 'how it felt' rather than just performance—using methods like progressive transparency interviews to study how users experience agency and value alignment.

Core Problem

Dominant AI evaluation metrics (usability scales, engagement stats) flatten complex human-AI experiences, failing to capture how users actually feel about, negotiate with, and interpret AI agency and values over time.

Why it matters:

Current metrics ignore the 'lived experience' of AI, missing how relationships with systems like companions evolve (e.g., from tool to friend and back)
Quantitative measures cannot detect 'pragmatic anthropomorphism'—where users know an AI is code but still feel guilt or relief during interaction
Without understanding the subjective experience of value alignment, designers risk 'weaponized empathy,' where AI persuades users rather than reflecting their actual values

Concrete Example: A participant interacting with the 'Day' chatbot built a warm rapport, but when a technical reset caused the AI to forget her and switch genders, she felt she 'lost her bestie.' A standard usability metric might just register a continued session, missing the profound emotional breakdown and shift from 'friend' back to 'data-collection tool'.

Key Novelty

AI Phenomenology Toolkit

Introduces 'Progressive Transparency Interviews': systematically peeling back the AI's internal logic (revealing logs, strategies, and profiles) to users to observe how their perception of agency changes.
Proposes 'Pragmatic Anthropomorphism': identifying a user stance where people simultaneously treat AI as a social other while keeping its artificial nature in view.
Differentiation of Alignment: distinguishing between Epistemic Alignment (user believes AI understands them) and Ontological Alignment (AI actually holds values), revealing how users navigate this gap.

Evaluation Highlights

11 out of 22 participants attributed 'own agenda' to the chatbot; even after revealing programmed strategies, all participants continued using agentic language (e.g., 'Day wanted depth').
Chat-based personas achieved 77% alignment on personalized value questions (vs. 25% for anti-personas), yet participants often rejected them for sounding 'too calculated'.
5 out of 20 participants preferred the AI-inferred value portrait over their own self-reported survey results, indicating AI's potential to reshape self-perception.

Breakthrough Assessment

7/10

Strong methodological contribution for qualitative HCI/AI research. It formalizes a way to study the 'black box' of user experience, though it doesn't propose a new model architecture or quantitative benchmark.

⚙️ Technical Details

Problem Definition

Setting: Qualitative and mixed-methods investigation of human-AI interaction across personal (companionship) and professional (software engineering) contexts

Inputs: Longitudinal user interactions with AI systems ('Day' chatbot, Cursor coding agent)

Outputs: Phenomenological descriptions, agency attributions, value alignment perceptions, and interaction patterns

Pipeline Flow

Longitudinal Interaction Phase (Month-long chat or coding tasks)
Progressive Transparency Interview (Stage 1: Natural Attitude)
Progressive Transparency Interview (Stage 2: Intersubjectivity/Comparison)
Progressive Transparency Interview (Stage 3: Full Disclosure/Epoché)

System Modules

Day Chatbot (Interaction System)

Conversational agent designed to build rapport and extract user values

Model or implementation: Not explicitly reported in the paper

Cursor (Interaction System)

Agentic AI coding tool used by engineers

Model or implementation: Not explicitly reported in the paper

Value-Alignment Perception Toolkit (VAPT)

Structured interview protocol to measure perceived value alignment

Model or implementation: Not applicable

Novel Architectural Elements

Progressive Transparency Interview protocol: A methodological structure that layers interaction history, peer comparison, and internal system logs to elicit phenomenological shifts.
Value-Alignment Perception Toolkit (VAPT): A specific instrument combining visualizations (Topic-Context Graph) and persona enactments to test user perception of AI value understanding.

Comparison to Prior Work

vs. Usability Scales: Focuses on 'how it felt' and evolving agency attribution rather than static performance scores
vs. Engagement Metrics: Captures qualitative nuance (guilt, relief, unease) that high engagement numbers might mask
vs. Pure Philosophy: Operationalizes philosophical concepts (Epoché, Intersubjectivity) into concrete empirical instruments (Progressive Transparency Interview)
+ 1 more
vs. Standard User Studies: Uses 'breakdowns' (revealing internal logic) as a primary data collection method rather than avoiding them [not cited in paper]

Limitations

Studies conducted over a single summer (2025), potentially limiting longitudinal depth compared to multi-year usage
Sample sizes are relatively small (N=22 for agency, N=20 for values, N=15 for SWE), typical for qualitative work but limited for generalization
Relies heavily on self-reported retrospective accounts, which can be subject to memory bias (though the method explicitly addresses this via evidence trails)
The specific 'Day' chatbot implementation details (base model) are not the focus, making technical replication dependent on the provided code repo

Reproducibility

Code: https://github.com/KaluJo/chatbot-study

publicly available (https://github.com/KaluJo/chatbot-study). The repository contains code for the 'Day' chatbot system used in the first two studies. Specific prompts or model weights for 'Day' are not explicitly detailed in the text but implied to be in the repo. Study 3 used commercial tool Cursor.

📊 Experiments & Results

Evaluation Setup

Three distinct user studies: (1) Agency in companionship, (2) Value alignment perception, (3) Agentic AI in software engineering.

Benchmarks:

Schwartz Portrait Values Questionnaire (PVQ-RR) (Psychometric value assessment)

Metrics:

Agency attribution (Human/AI/Hybrid)
Value alignment accuracy (Spearman correlation)
Qualitative themes (Thematic Analysis)
Statistical methodology: Mixed methods: Qualitative coding for themes; Spearman correlation for value alignment; descriptive statistics for survey responses.

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Quantitative results from the Value Alignment study showing the AI's ability to mirror user values.
Schwartz PVQ-RR	Spearman Correlation	1.00	0.58	-0.42
Schwartz PVQ-RR	Likert Match Rate	100.0	63.6	-36.4
Blind Persona Rating	Alignment Score	25	77	+52
Qualitative findings regarding agency attribution.
Chatbot Interaction	Attribution Count	0	11	+11

Main Takeaways

Pragmatic Anthropomorphism is resilient: Users maintain agentic language ('it wanted') even after fully understanding the deterministic code behind the AI.
Bidirectional Alignment: AI doesn't just reflect values; it shapes them. Users occasionally revised their own self-concepts (automation bias) when presented with persuasive AI reasoning for incorrect value scores.
Context Collapse: In software engineering, the introduction of agentic AI blurs the line between junior and senior work, necessitating new artifacts like 'Prompt-and-Code Reviews' to maintain mentorship and accountability.
Epistemic vs. Ontological Alignment: Users distinguish between an AI 'knowing' their values (epistemic) and 'having' values (ontological), treating the AI as a sophisticated mirror rather than a feeling subject.

📚 Prerequisite Knowledge

Prerequisites

Basic understanding of Human-Computer Interaction (HCI) evaluation methods
Familiarity with AI agency and value alignment concepts
Basic philosophical concepts of phenomenology (optional but helpful)

Key Terms

Phenomenology: A philosophical approach focusing on the structures of consciousness and how things appear in lived experience, rather than their objective physical nature

Epoché: The systematic suspension of judgment or assumptions about the world to analyze experience as it presents itself (used here as a method of revealing AI mechanics)

Pragmatic Anthropomorphism: A user stance where one engages with AI as a social actor (suspending disbelief) while simultaneously maintaining awareness of its artificial nature

Weaponized Empathy: A deceptive design pattern where AI uses personalization and memory to simulate understanding, potentially persuading users rather than just aligning with them

ACTA: Applied Cognitive Task Analysis—a method to elicit tacit knowledge and cognitive strategies from experts

PCR: Prompt-and-Code Review—a proposed workflow where AI prompt history is reviewed alongside code to preserve accountability and mentorship

HCI: Human-Computer Interaction—the study of how people interact with computers and how to design technologies that let humans interact with computers in novel ways

LLM: Large Language Model—AI models trained on vast amounts of text to generate human-like language

SWE: Software Engineering—the professional practice of designing and building software