TrueNorth: PERMA+ 4 and Conversational Agentic RAG to Optimize Long-Term STEM Engagement

📝 Paper Summary

Agentic RAG pipeline Modularized RAG pipeline

TrueNorth is a multi-agent dialogue system that integrates a positive psychology framework (PERMA+4) with Retrieval-Augmented Generation to provide evidence-based mentoring for early-career STEM professionals.

Core Problem

General-purpose Large Language Models often lack the context-specific domain expertise, empirical rigor, and psychological grounding necessary to provide effective mentoring for STEM professionals facing workplace attrition.

Why it matters:

The 'leaky pipeline' in STEM causes significant workforce attrition, particularly among early-career professionals and underrepresented groups
Traditional mentoring is resource-intensive and often inaccessible, while standard LLMs may provide generic or hallucinated advice that fails to address complex organizational behavior

Concrete Example: When a user asks about handling credit attribution in a lab setting, a standard LLM might give generic platitudes. TrueNorth retrieves specific conflict resolution strategies grounded in PERMA+4 (e.g., Engagement and Relationships) to offer actionable, psychologically valid guidance.

Key Novelty

PERMA+4-Grounded Agentic RAG

Embeds the PERMA+4 positive psychology framework (Positive Emotion, Engagement, Relationships, Meaning, Accomplishment + Health, Mindset, Environment, Economic Security) directly into the knowledge retrieval and generation process
Utilizes a nine-agent architecture where specialized modules (e.g., Hallucination Checker, Relevance Grader) autonomously coordinate to ensure factual integrity and psychological relevance

Architecture

Multi-agent RAG Dialogue System architecture integrating the PERMA+4 Knowledge Base and Tavily Web Search

Evaluation Highlights

No significant performance variation (p > 0.05) across 11 of 14 evaluation dimensions (79%), indicating robustness across diverse STEM challenge categories
Achieved high Accessibility score (Mean = 4.7/5.0), demonstrating the system effectively translates complex psychological concepts into clear guidance
Trustworthiness score of 4.4/5.0 confirms the effectiveness of the RAG pipeline in grounding responses in peer-reviewed literature

Breakthrough Assessment

7/10

Strong application of agentic RAG to a specific, high-impact domain (STEM retention) with a novel psychological framework integration, though technical novelty lies more in system orchestration than new model architectures.

⚙️ Technical Details

Problem Definition

Setting: Conversational mentoring for early-career STEM professionals using domain-specific knowledge retrieval

Inputs: Natural language user queries regarding workplace challenges (e.g., burnout, discrimination, career planning)

Outputs: Contextually accurate, evidence-based mentoring responses grounded in PERMA+4 principles

Pipeline Flow

Query Router (Decides pathway: internal RAG, web search, or chat)
Query Rewriter (Reformulates ambiguous queries)
Document Retriever (Fetches chunks via vector search)
Relevance Grader (Filters chunks via MMR/RRF)
Answer Generator (Synthesizes response)
Hallucination Checker (Validates facts/URLs)
Answer Evaluator (Quality assurance)

System Modules

Query Router

Analyzes query intent to route to Retrieval, Web Search (Tavily), or Chitter-Chatter agent

Model or implementation: Google Gemini 2.0 Flash

Document Retriever (Retrieval & Selection)

Performs semantic similarity search on PostgreSQL vector database

Model or implementation: Google Gemini text-embedding-004

Relevance Grader (Retrieval & Selection)

Filters retrieved documents using Maximal Marginal Relevance (MMR) and Reciprocal Rank Fusion (RRF)

Model or implementation: Google Gemini 2.0 Flash (as logic engine)

Answer Generator

Synthesizes final response integrating retrieved context and PERMA+4 principles

Model or implementation: Google Gemini 2.0 Flash

Hallucination Checker

Cross-references outputs against sources and validates URLs via HTTP HEAD requests

Model or implementation: Google Gemini 2.0 Flash (as logic engine)

Novel Architectural Elements

Integration of PERMA+4 framework directly into the knowledge base structure and retrieval logic
Nine-agent orchestration using LangGraph specifically for psychological mentoring context

Modeling

Base Model: Google Gemini 2.0 Flash

Training Method: Prompt Engineering and RAG (Inference-only)

Key Hyperparameters:

temperature: 1.0
top_p: 0.95
top_k: 64
+ 2 more
candidate_count: 1
RRF_constant_k: 60

Compute: Not reported in the paper

Comparison to Prior Work

vs. Standard LLMs: TrueNorth grounds responses specifically in PERMA+4 literature via RAG to reduce hallucination and increase relevance [not cited in paper]
vs. Traditional RAG: Uses agentic workflow (9 agents) including explicit hallucination checking and relevance grading rather than simple retrieve-then-generate
vs. Mental Health Chatbots (e.g., Woebot): Focuses specifically on STEM career retention and organizational behavior rather than general CBT [not cited in paper]

Limitations

Evaluation relies on synthetic AI-as-a-judge (Claude-3.5-Haiku) rather than human users
Knowledge base is relatively small (32 documents), potentially limiting breadth
Personal Visual Engagement metric showed significant variation, suggesting inconsistency in personalized interactions
Interface implementation in Streamlit has limitations regarding polished user experience

Reproducibility

Code: https://github.com/jovalie/TrueNorth-AIxHEART-2025

📊 Experiments & Results

Evaluation Setup

AI-as-a-judge evaluation of generated responses to synthetic user scenarios

Benchmarks:

Custom STEM Challenge Dataset (Mentoring response generation) [New]

Metrics:

Usefulness
Goal Facilitation
Relatability
Trustworthiness
Accessibility
Human-likeness
Attractivity
Statistical methodology: Chi-squared tests to examine category-dependent performance variation across challenge types

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
TrueNorth demonstrates high performance across key conversational quality metrics when evaluated by an AI judge.
Custom STEM Challenge Dataset	Accessibility (1-5 Likert)	Not reported in the paper	4.7	Not reported in the paper
Custom STEM Challenge Dataset	Goal Facilitation (1-5 Likert)	Not reported in the paper	4.5	Not reported in the paper
Custom STEM Challenge Dataset	Trustworthiness (1-5 Likert)	Not reported in the paper	4.4	Not reported in the paper
Custom STEM Challenge Dataset	Usefulness (1-5 Likert)	Not reported in the paper	4.3	Not reported in the paper
Custom STEM Challenge Dataset	Personal Visual Engagement (p-value)	0.05	0.002	-0.048

Main Takeaways

System achieves consistently high scores (>4.0/5.0) on Accessibility, Goal Facilitation, and Trustworthiness, suggesting the RAG pipeline effectively retrieves and simplifies complex advice
Performance is robust across 79% of evaluation dimensions (11 of 14 metrics showed no significant variation across challenge categories)
Weaknesses identified in 'Personal Visual Engagement' and 'Inclusive Community' metrics, which showed significant variation depending on the specific STEM challenge type
The integration of Hallucination Checker and Relevance Grader likely contributed to the high Trustworthiness score (4.4)

📚 Prerequisite Knowledge

Prerequisites

Understanding of Retrieval-Augmented Generation (RAG) architectures
Familiarity with agentic AI workflows (LangGraph/LangChain)
Basic knowledge of vector search and embeddings

Key Terms

RAG: Retrieval-Augmented Generation—AI systems that answer questions by first searching for relevant documents before generating a response

PERMA+4: A positive psychology framework extending Seligman's model (Positive Emotion, Engagement, Relationships, Meaning, Accomplishment) with Physical Health, Mindset, Environment, and Economic Security

Agentic RAG: A RAG system where autonomous software agents dynamically control the retrieval and generation process (e.g., deciding when to search or rewrite queries)

MMR: Maximal Marginal Relevance—a ranking algorithm used to select documents that are both relevant to the query and diverse from each other to reduce redundancy

RRF: Reciprocal Rank Fusion—a method for combining multiple ranked lists of search results into a single unified ranking

pgvector: An extension for PostgreSQL databases that allows for storing and searching vector embeddings

hallucination: In AI, when a model generates incorrect or fabricated information not supported by its training data or retrieved context

LangGraph: A library for building stateful, multi-agent applications with Large Language Models, enabling cyclic workflows

LangChain: A framework designed to simplify the creation of applications using Large Language Models

STEM: Science, Technology, Engineering, and Mathematics fields