A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms

📝 Paper Summary

Autonomous Driving (AD) Large Language Models (LLMs) in Robotics Neuro-symbolic AI

This survey identifies the lack of robust reasoning as the primary bottleneck in autonomous driving and proposes a three-level Cognitive Hierarchy to guide the integration of LLMs as a central cognitive engine.

Core Problem

Autonomous driving systems have mastered structured perception and control but consistently fail in long-tail, complex social scenarios due to a deficit in robust, generalizable reasoning.

Why it matters:

The bottleneck in AD has shifted from sensing physical limitations to 'planning discrepancy,' where systems fail to understand context
Current modular pipelines (Perception → Prediction → Planning) suffer from information loss and cannot handle implicit social rules
Without human-like reasoning, vehicles cannot safely navigate unpredictable scenarios like construction zones or non-verbal negotiation with pedestrians

Concrete Example: When a ball rolls onto a street, a standard perception system detects 'ball' and 'road.' A reasoning-enabled system uses commonsense to infer a 'hidden child likely following' and preemptively slows down—a capability often missing in purely rule-based or reactive AD systems.

Key Novelty

The Cognitive Hierarchy Framework

Deconstructs driving into three levels of increasing complexity: (1) Sensorimotor (reflexive control), (2) Egocentric Reasoning (planning relative to other agents), and (3) Social-Cognitive (negotiation and commonsense)
Proposes elevating reasoning from a modular component to the 'Cognitive Core' of the system, using LLMs to coordinate perception and control rather than just processing text
Systematizes seven core reasoning challenges, including the 'Responsiveness-Reasoning Tradeoff' and 'Social-Game Reasoning'

Architecture

The proposed Cognitive Hierarchy framework deconstructing the autonomous driving task

Breakthrough Assessment

7/10

A significant conceptual contribution that structures the chaotic landscape of LLM-based driving. While it offers no new algorithm, the 'Cognitive Hierarchy' provides a necessary roadmap for moving beyond Level 2/3 autonomy.

⚙️ Technical Details

Problem Definition

Setting: Integration of Large Language Models (LLMs) and Multimodal LLMs (MLLMs) into Autonomous Driving stacks

Inputs: Multimodal sensor data (cameras, LiDAR) and textual context

Outputs: High-level driving decisions, trajectory plans, and control signals

Limitations

Fundamental tension between high latency of LLM reasoning and millisecond-level safety requirements of vehicle control
Lack of systematic frameworks to verify the safety of probabilistic LLM outputs in critical traffic scenarios
Current 'glass-box' approaches often lack the closed-loop robustness to handle the full symbolic-to-physical translation required for real-world driving

Reproducibility

This is a survey paper; no specific code or model weights are associated with the contribution.

📊 Experiments & Results

Evaluation Setup

Survey and analysis of existing literature; no new experimental benchmarks were conducted.

Metrics:

Statistical methodology: Not explicitly reported in the paper

Main Takeaways

The field is shifting from 'system-centric' engineering to 'cognitive-centric' agent design, prioritizing generalizable reasoning over sensor precision
There is a clear trend toward 'glass-box' agents that offer interpretability, but they currently struggle with the real-time constraints of the 'Sensorimotor Level'
A critical unresolved challenge is the 'Responsiveness-Reasoning Tradeoff': the deeper the reasoning (LLM-based), the slower the reaction time, creating safety risks
Future success depends on bridging the 'symbolic-to-physical gap': creating verifiable architectures that can ground high-level social reasoning into safe, low-level control

📚 Prerequisite Knowledge

Prerequisites

Standard Autonomous Driving pipeline (Perception, Prediction, Planning, Control)
Capabilities of Large Language Models (LLMs) and Multimodal LLMs (MLLMs)
Basic concepts of reinforcement learning and game theory in traffic

Key Terms

Cognitive Hierarchy: The paper's proposed framework dividing driving into Sensorimotor (basic), Egocentric (interaction), and Social-Cognitive (intent/norms) levels

Glass-box agents: AI systems designed to be interpretable, allowing humans to see the internal reasoning process (e.g., Chain-of-Thought) behind a driving decision

Symbolic-to-physical gap: The difficulty of translating high-level linguistic or symbolic reasoning (e.g., 'slow down carefully') into precise continuous control signals (steering angle, brake pressure)

Long-tail scenarios: Rare, edge-case driving situations (e.g., weird loads, ambiguous gestures) that traditional data-driven models struggle to generalize to

Neuro-symbolic: Architectures attempting to combine the learning capability of neural networks (perception) with the logical robustness of symbolic systems (reasoning)

Social-Game Reasoning: Modeling traffic interactions as a game-theoretic process where agents must predict and negotiate based on the likely actions of others