MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks

📝 Paper Summary

3D Scene Generation Embodied AI Simulation Procedural Content Generation

MANSION combines large language models with geometric solvers to generate interactive, building-scale multi-floor environments from text, enabling the evaluation of long-horizon embodied agents in complex spatial settings.

Core Problem

Existing embodied AI benchmarks are confined to single-floor, residential layouts that lack the vertical structure and scale required to test long-horizon spatial reasoning and planning.

Why it matters:

Real-world tasks (delivery, hospital transport) are inherently building-scale and multi-floor, requiring agents to navigate elevators and stairs
Current scene generation methods focus on single rooms or apartments, failing to model vertical constraints or cross-floor connectivity
Scanned 3D datasets are expensive to collect, hard to modify for specific tasks, and lack interactive semantic elements needed for simulation

Concrete Example: A standard generator might create a house where the second floor's footprint doesn't align with the first, or stairs lead to a ceiling. MANSION enforces vertical alignment and connectivity, allowing an agent to successfully navigate from a first-floor lobby to a second-floor office.

Key Novelty

Hybrid MLLM-Geometry Pipeline for Vertical Structures

Decouples high-level semantic planning (via MLLMs) from low-level geometric validity (via constrained solvers) to ensure structural correctness
Enforces vertical alignment as a hard constraint, ensuring elevator shafts and stairwells physically connect across multiple floors
Introduces a Task-Semantic Scene Editing Agent that modifies static environments via tool usage to satisfy preconditions for specific language-defined tasks

Architecture

The hierarchical multi-agent framework of MANSION, showing the flow from user instruction to final 3D scene.

Evaluation Highlights

Generated MansionWorld dataset containing over 1,000 diverse multi-floor buildings (offices, hospitals, schools) with >10,000 rooms
State-of-the-art embodied agents show sharp performance degradation when moving from single-floor to multi-floor tasks in MANSION
Framework supports export to AI2-THOR, Blender, and NVIDIA Isaac Sim for broad simulator compatibility

Breakthrough Assessment

9/10

First framework to successfully generate navigable, building-scale multi-floor environments from language, addressing a major gap in embodied AI simulation for long-horizon planning.

⚙️ Technical Details

Problem Definition

Setting: Generating 3D navigable building layouts and object placements from natural language prompts, subject to vertical structural constraints.

Inputs: Natural language description of building function/style and building footprint polygon

Outputs: Interactive 3D simulation environment (AI2-THOR) with vertically aligned floorplans and semantic objects

Pipeline Flow

Building-Level Planning: Determine global style and cross-floor zones
Floor Planning: Generate per-floor bubble diagrams with vertical core constraints
Cutting MLLM: Propose initial room seeds on rasterized grids
Geometric Solver: Hierarchical single-cut growth to realize room polygons
Scene Instantiation: Place objects and architectural elements

System Modules

Building-Level Planning Node (Planning)

Determines cross-floor functional zones, target areas, and global style from user prompts

Model or implementation: MLLM (unspecified specific variant)

Floor-Planning Node (Planning)

Generates bubble diagrams for each floor, specifying adjacencies and vertical core connections

Model or implementation: MLLM

Cutting MLLM Node (Geometry Generation)

Proposes coarse initial seeds (coordinates) for room growth on a rasterized grid

Model or implementation: MLLM

Single-Cut Solver (Geometry Generation)

Realizes room partitions via topology-aware constrained growth

Model or implementation: Algorithmic Solver (Lopes-style constrained growth)

Task-Semantic Scene Editing Agent

Modifies the generated scene to satisfy specific task preconditions (e.g., adding missing objects)

Model or implementation: MLLM Controller

Novel Architectural Elements

Hierarchical splitting strategy orchestrated by LangGraph to manage combinatorial complexity of room layout
Explicit integration of vertical structural constraints (stairs/elevators) as 'hard' constraints in the generation pipeline
Check-and-Provision workflow in the editing agent to dynamically modify scenes for task feasibility

Comparison to Prior Work

vs. Holodeck: MANSION generates multi-floor buildings with vertical alignment constraints, whereas Holodeck is single-floor only
vs. SceneCraft: MANSION models inter-floor portals (elevators/stairs) explicitly for navigation
vs. DiffuScene: MANSION uses a hybrid MLLM-geometry approach for open-world scalability, while DiffuScene is limited to closed-world training data distributions

Limitations

Reliance on proprietary MLLMs (implied) may limit full open-source reproducibility without API costs
Geometric solver assumes orthogonal polygons, potentially limiting organic architectural shapes
Evaluation focuses on navigation/interaction capability rather than photorealism compared to scanned datasets

Reproducibility

Code: https://mansion-project.github.io

Publicly available code and dataset at https://mansion-project.github.io. The paper mentions using MLLMs but does not specify the exact model versions (e.g., GPT-4V vs. Claude 3) in the main text.

📊 Experiments & Results

Evaluation Setup

Evaluation of both the generation quality (floorplan validity) and the utility of the environments for embodied agents.

Benchmarks:

Floorplan Generation Benchmark (Layout Synthesis) [New]
Embodied Navigation & Manipulation (Long-horizon agent tasks) [New]

Metrics:

Success Rate (SR)
Path Length (PL)
IoU (Intersection over Union for layout similarity)
Generalized IoU
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
MansionWorld Multi-Floor Tasks	Performance Degradation	Not reported in the paper	Not reported in the paper	Not reported in the paper
MansionWorld	Number of Buildings	Not reported in the paper	1000	Not reported in the paper

Experiment Figures

Visual examples of the generated MansionWorld dataset.

The workflow of the Task-Semantic Scene Editing Agent.

Main Takeaways

MansionWorld provides the first large-scale (>1,000 buildings) dataset of diverse, multi-floor environments (hospitals, offices, schools) for embodied AI.
The constrained-growth solver allows for 'open-world' generation that generalizes beyond the residential distributions found in training data for diffusion models.
State-of-the-art agents struggle significantly with the spatial reasoning required for cross-floor navigation (elevators/stairs), validating the benchmark's difficulty.
The Task-Semantic Scene Editing Agent successfully converts static scenes into valid task environments by checking preconditions and inserting necessary objects.

📚 Prerequisite Knowledge

Prerequisites

Procedural content generation (PCG) concepts
Multimodal Large Language Models (MLLM)
Embodied AI navigation and planning
Graph-based scene representations

Key Terms

AI2-THOR: An interactive 3D environment for embodied AI agents to navigate and manipulate objects

Bubble Diagram: A graphical representation of room layouts where nodes represent rooms and edges represent adjacencies

LangGraph: A library for building stateful, multi-actor applications with LLMs, used here to orchestrate the generation pipeline

MLLM: Multimodal Large Language Model—an AI model capable of processing and generating both text and images/spatial data

Vertical Cores: Structural elements like elevator shafts and stairwells that must align vertically across floors

Constrained Growth: A geometric algorithm that expands room boundaries while respecting hard constraints like walls and adjacencies

Reachability: A constraint ensuring that placed objects have enough clearance for a robot agent to navigate to and interact with them