A Novel Hierarchical Multi-Agent System for Payments Using LLMs

📝 Paper Summary

Multi-agent collaboration Agentic payments

HMASP is a hierarchical multi-agent framework that modularizes payment tasks into distinct agent roles (Conversation, Supervisor, Routing, Summary) to enable secure, end-to-end agentic payments with minimal infrastructure changes.

Core Problem

Current LLM agents cannot perform end-to-end payments because traditional payment infrastructure (anti-bot, fraud controls, PCI DSS) blocks them, and existing solutions require major structural changes to payment rails.

Why it matters:

Regulatory frameworks like PCI DSS strictly govern AI use in payments, making direct integration of standard agents difficult due to data exposure risks
Traditional payment networks employ anti-bot mechanisms that inherently reject automated agentic interactions
LLM hallucination risks pose severe threats to financial transaction reliability and security

Concrete Example: When an external agent tries to "Register a new card," it fails at the network boundary due to authentication controls. In HMASP, a dedicated Routing Agent triggers a specific, isolated registration subgraph that validates the 16-digit card checksum (Luhn Check) before execution, preventing invalid data from reaching the payment rail.

Key Novelty

Hierarchical Multi-Agent System for Payments (HMASP)

Deconstructs payment processing into four hierarchical levels: a conversational entry point, domain supervisors, task-specific routers, and process summarizers
Isolates sensitive payment data in decoupled state variables that agents can access but not hallucinate, ensuring deterministic execution of critical financial tasks
Uses a structured handoff protocol where agents pass control downstream to execute tasks and propagate summary outcomes back upstream without exposing raw internal states

Architecture

The hierarchical structure of HMASP, detailing the flow from user input to payment execution and back

Evaluation Highlights

99.6% task success rate on payment checkout tasks using GPT-4.1, with open-weight model Qwen2.5:32b achieving a comparable 95.6%
99.9% F1-score for agent handoffs (routing requests correctly) with GPT-4.1, demonstrating reliable orchestration
100% rejection rate of irrelevant inputs (e.g., "tell me a joke") across top performing models (GPT-4.1, Qwen2.5:32b), preventing unauthorized workflow triggers

Breakthrough Assessment

7/10

Significant as the first proposed framework for end-to-end agentic payments that respects existing rails. High practicality, though evaluation is simulation-based.

⚙️ Technical Details

Problem Definition

Setting: End-to-end execution of payment-related workflows (registration, retrieval, checkout) via natural language commands

Inputs: Natural language user request (e.g., "Complete my payment")

Outputs: Completed payment transaction or status response (e.g., "Your payment has been successfully completed")

Pipeline Flow

CPA receives external request -> Hands off to Supervisor
Supervisor identifies domain -> Hands off to Routing Agent
Routing Agent triggers Task-Specific Workflow (Subgraph)
Process Summary Agent compiles outcome -> Propagates back to Supervisor -> CPA responds

System Modules

Conversational Payment Agent (CPA)

Central entry point handling external requests and coordinating high-level handoffs

Model or implementation: Various (GPT-4.1, Qwen, Llama tested)

Supervisor Agent

Decision-maker for domain-specific workflows (e.g., payments vs. cards domain)

Model or implementation: Same as CPA

Routing Agent

Final decision layer to trigger task-specific function modules or reject requests

Model or implementation: Same as CPA

Process Summary Agent

Generates structured summaries of workflow outcomes (success/failure)

Model or implementation: Same as CPA

Novel Architectural Elements

Four-level hierarchical agent structure specific to payments (CPA -> Supervisor -> Router -> Summary)
Role-based decoupled message states (agents only see their own message history to reduce token usage and exposure)
Interrupt mechanism for secure Human-in-the-Loop input collection (e.g., card details) integrated directly into the agent graph

Modeling

Base Model: Evaluated with GPT-4.1 (proprietary baseline) and various open-weights (Qwen2.5, Qwen3, Mistral, Llama3.1)

Compute: Open-weight models run locally using Ollama on NVIDIA H100 GPU (96 GB VRAM)

Comparison to Prior Work

vs. LLMPA: HMASP executes end-to-end payments including checkout, whereas LLMPA stops at the app navigation level
vs. Fraud Agents: HMASP is a full payment execution framework, not just a monitoring tool
vs. Tokenized Credentials [not cited in paper]: HMASP focuses on natural language orchestration rather than just credential management infrastructure

Limitations

Evaluated in a simulated environment using function mocks rather than live production payment rails
Dataset is relatively small (1000 points) and proprietary
Production complexities like actual issuer authentication latency were not modeled

Reproducibility

Dataset (1000 data points) is part of Mastercard's research and 'can be made available based on request'. Code is not explicitly linked. Payment modules were simulated using functions.

📊 Experiments & Results

Evaluation Setup

Simulation of payment workflows (Registration, Retrieval, Checkout) and irrelevant inputs

Benchmarks:

Mastercard Research Dataset (Payment intent classification and execution) [New]

Metrics:

Task Success Rate (Triggering correct workflow + saving/retrieving correct info)
Agent Handoff F1-Score (Accuracy of routing between agents)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Task Success Rate measures if the correct workflow was triggered and the correct information (e.g., transaction details) was saved.
Mastercard Research Dataset (Task 3 - Payment Processing)	Task Success Rate	100	95.6	-4.4
Mastercard Research Dataset (Task 1 - Card Registration)	Task Success Rate	99.6	99.6	0.0
Agent Handoff F1-Score measures the reliability of the routing logic between the CPA, Supervisors, and Routers.
Mastercard Research Dataset (Task 3 - Payment Processing)	Handoff F1-Score	100	99.0	-1.0
Mastercard Research Dataset (Task 4 - Irrelevant Inputs)	Handoff F1-Score	100	11.4	-88.6

Main Takeaways

Proprietary models (GPT-4.1) deliver near-perfect performance (99.6-100%) across all payment tasks
Specific open-weight models (Qwen2.5:32b) are competitive alternatives, achieving >95% success rates across tasks
Llama 3.1 models struggled significantly with orchestrating handoffs, often failing to reject irrelevant inputs or trigger the correct payment workflow
The hierarchical architecture successfully isolates payment logic, as evidenced by high success rates in validated environments

📚 Prerequisite Knowledge

Prerequisites

Understanding of LangGraph state management (shared vs. private states)
Basic knowledge of payment security standards (PCI DSS, 3D-Secure)
Familiarity with hierarchical multi-agent orchestration

Key Terms

HMASP: Hierarchical Multi-Agent System for Payments—the proposed framework organizing agents into four levels

CPA: Conversational Payment Agent—the top-level agent acting as the single entry point for user requests

PCI DSS: Payment Card Industry Data Security Standard—strict regulations for handling credit card information

LangGraph: An orchestration framework used here to manage agent states and structured handoffs

Luhn Check: A checksum formula used to validate a variety of identification numbers, such as credit card numbers

3D-Secure: A security protocol that adds an extra layer of authentication for online credit card and debit card transactions

Decoupled Message States: An architectural pattern where agents only access their own role-specific message history, minimizing token usage and data exposure

Interrupt: A mechanism that pauses workflow execution to request and validate human input (e.g., asking for card details) before resuming