ODIN: A NL2SQL Recommender to Handle Schema Ambiguity

📝 Paper Summary

User-profile based personalization Multi-turn w. user interactions

Odin resolves database schema ambiguity by generating diverse SQL options via iterative schema masking, filtering them with conformal prediction, and learning user preferences to personalize future queries.

Core Problem

Enterprise databases often contain ambiguous schemas (e.g., similarly named tables/columns), causing standard NL2SQL models to guess user intent incorrectly and produce wrong data.

Why it matters:

Ambiguity can lead to structurally different queries (e.g., joins vs. pre-aggregated lookups) with different runtime costs and results
Standard diversity techniques like high-temperature sampling produce only 'cosmetic' variations (syntax changes) rather than 'semantic' diversity (logic changes), failing to expose real alternatives to the user
Users have specific business logic preferences (e.g., using 'gross_sales' vs 'net_sales') that generic models fail to capture over time

Concrete Example: A user asks 'What is the revenue per customer?'. The database has an `orders` table (requiring aggregation) and a `customers` table (with a `total_revenue` column). A standard model might guess one arbitrarily. Odin generates both options, allowing the user to select the pre-aggregated column, which Odin then learns as a preference.

Key Novelty

Generate-Select-Personalize Paradigm for Ambiguity

**Forced Diversity via Masking:** Instead of random sampling, the Generator explicitly hides schema elements (columns/tables) used in previous queries to force the LLM to find alternative valid paths (e.g., hide `birthplace` to force use of `origin`)
**Conformal Selection:** Uses conformal prediction to filter the candidate set, providing statistical guarantees that the correct query is retained while minimizing the number of options shown to the user

Architecture

Overview of Odin's workflow handling a user question about 'hometown of students'.

Evaluation Highlights

Improves the likelihood of generating the correct SQL query by 1.5–2x compared to baselines on the AmbiQT benchmark
Maintains a result set size that is 2–2.5x smaller than baselines while preserving accuracy
Successfully generates semantically diverse queries where traditional high-temperature sampling produces only superficial syntactic variations

Breakthrough Assessment

8/10

Addresses a critical, often-overlooked real-world enterprise problem (schema ambiguity) with a clever algorithmic solution (masking) rather than just bigger models. The integration of conformal prediction for quality control is robust.

⚙️ Technical Details

Problem Definition

Setting: Natural Language to SQL (NL2SQL) with ambiguous schemas

Inputs: Natural language question Q, Database Schema

Outputs: A set of candidate SQL queries {SQL_1, ..., SQL_K}

Pipeline Flow

Generator (Iterative Schema Masking)
Selector (Conformal Prediction Filtering)
Personalizer (Hint Generation)

System Modules

Generator

Produce a set of semantically diverse SQL queries by iteratively forcing the LLM to use different schema elements

Model or implementation: Not reported in the paper

Selector

Filter out incorrect or irrelevant queries from the candidate set to reduce user cognitive load

Model or implementation: Not reported in the paper

Personalizer

Learn user preferences from their selection to guide future query generation

Model or implementation: Not reported in the paper

Novel Architectural Elements

Iterative Schema Masking loop: A feedback mechanism where previous generation outputs dictate the input schema for the next generation pass to ensure diversity
Personalization via Hint Injection: Dynamic generation of schema linking hints based on user selection, which are injected into prompts for future queries

Modeling

Base Model: Large Language Models (Specific architecture not reported in the paper)

Training Method: In-context learning / Inference-time algorithm

Compute: Not reported in the paper

Comparison to Prior Work

vs. SC-Prompt/MAC-SQL: Odin explicitly handles schema ambiguity and generates sets of queries rather than a single output, and learns from user feedback
vs. AmbiQT: Odin uses schema masking for diversity rather than beam search infilling, and incorporates personalization
vs. Standard LLM Sampling: Odin produces semantic diversity (different logic) via masking, whereas sampling produces cosmetic diversity (syntax changes)

Limitations

Depends on the availability of a scoring function that accurately reflects schema relevance (SBERT used here)
Iterative generation requires multiple LLM calls, potentially increasing latency and cost (mitigated by greedy search)
Requires user interaction to resolve ambiguity and provide feedback for personalization

Reproducibility

No replication artifacts mentioned in the paper. The paper describes algorithms (masking, conformal prediction) but does not provide code, model weights, or specific prompts.

📊 Experiments & Results

Evaluation Setup

Evaluation on NL2SQL tasks with ambiguous schemas

Benchmarks:

AmbiQT (Modified) (Ambiguous NL2SQL) [New]

Metrics:

Likelihood of generating correct SQL (Recall)
Average Result Set Size
Statistical methodology: Conformal prediction used for selection guarantees

Main Takeaways

Odin improves the likelihood of finding the correct SQL query by 1.5–2x compared to baselines on the AmbiQT benchmark.
The system maintains a result set size 2–2.5x smaller than baselines, reducing user effort in selecting the correct query.
High-temperature sampling in LLMs fails to generate semantically diverse queries (often producing valid but identical execution results), whereas Odin's schema masking forces structural diversity.

📚 Prerequisite Knowledge

Prerequisites

Understanding of NL2SQL pipelines
Basic knowledge of database schemas (tables, columns, joins)
Familiarity with Large Language Models (LLMs) and prompting

Key Terms

NL2SQL: Natural Language to SQL—systems that translate human language questions into database queries

Schema Ambiguity: Situations where multiple tables or columns have similar names or semantic roles (e.g., 'sales' vs 'gross_sales'), making user intent unclear

Schema Masking: The technique of temporarily removing specific columns or tables from the schema provided to an LLM to force it to generate queries using different data sources

Conformal Prediction: A statistical framework used to construct prediction sets that contain the true label with a guaranteed probability (recall control)

Nucleus Sampling: A text decoding method that samples from the top probability tokens, often used to generate diverse text

SBERT: Sentence-BERT—a modification of the BERT network to derive semantically meaningful sentence embeddings that can be compared using cosine similarity

Schema Linking: The process of identifying which words in a natural language question correspond to which columns or tables in a database schema