RAG Evaluation Is Model Risk Management for Financial Research

Retrieval-augmented generation has become the default answer to a familiar problem: large language models can write fluent research summaries without enough evidence underneath them. In a finance or commercial real estate workflow, that failure mode is expensive. A memo that cites the wrong paper, uses a stale regulatory source, or summarizes a lease abstract against the wrong document is not just a weak draft. It creates review burden and credibility risk.

RAG is useful because it changes the information architecture. The original Retrieval-Augmented Generation paper framed the method as a way to combine a language model's parametric memory with an external non-parametric memory, so the system can retrieve passages and generate from them. That is the right direction for institutional research. A model should not rely only on what it absorbed during training when the user is asking about current guidance, a specific paper, a data-room document, or a market memo.

But retrieval is not a compliance program. It is not a substitute for model validation. It is a component that needs its own tests.

The evaluation target is not only answer quality

Generic answer-quality evaluation is too thin for financial research. A response can be readable, concise, and still unsafe to use. The more important questions are operational:

• Did the system retrieve the right source?
• Did it retrieve enough of the source to answer the question?
• Did the final answer stay inside what the retrieved material supports?
• Did it preserve provenance so a reviewer can inspect the claim?
• Did it fail closed when the source set was incomplete?

The RAGAS paper is useful because it separates RAG evaluation into multiple dimensions, including retrieval context and answer faithfulness. That separation matters. A research agent can fail at retrieval before generation begins, or it can retrieve the right evidence and still overstate the conclusion. Those are different defects with different fixes.

Finance teams should therefore avoid a single "RAG score." A better evaluation dashboard has at least four lanes: retrieval coverage, citation fidelity, answer faithfulness, and workflow traceability. The unit of review is not only the final paragraph. It is the path from question to source set to extracted evidence to generated answer.

Unsupported claims still occur with retrieval

RAG reduces one class of unsupported output, but it does not eliminate it. RAGTruth was built around this problem: even when systems use retrieval, generated answers may include claims that are unsupported by or contradictory to the retrieved content. The paper presents a corpus of nearly 18,000 RAG responses with manual annotations for hallucination analysis across domains and tasks.

That finding lines up with what finance users see in practice. A system may retrieve a relevant paper and then describe a stronger result than the paper reports. It may cite a regulator's guidance and apply it to the wrong entity type. It may answer a portfolio question using a source that is current for public equities but irrelevant for private real estate. In each case, the retrieval step creates a trail, but the trail still needs verification.

This is where institutional workflows differ from consumer search. The goal is not to make the answer sound helpful. The goal is to make it reviewable by someone with responsibility for the decision.

Model risk language already fits the problem

Financial institutions do not need to invent an entirely new governance vocabulary for RAG. They already have a model risk frame, even though it was not written with RAG in mind. In April 2026, the Federal Reserve, OCC, and FDIC issued revised model risk management guidance for the banking organizations they supervise, superseding SR 11-7 and covering model development and use, validation and monitoring, governance and controls, and third-party products. The OCC bulletin summarizes the same guidance for national banks, federal savings associations, and federal branches and agencies. Both agencies were explicit that the guidance's formal scope covers traditional quantitative and statistical models, not generative or agentic AI systems, which they described as novel and still evolving.

That scope limit does not remove the analogy for finance and CRE research teams. It sharpens it. Most teams building RAG-based research tools are not themselves Federal Reserve, OCC, or FDIC-supervised banking organizations, and even inside a bank, a RAG research assistant may sit outside the guidance's formal reach. But the reasoning the guidance sets out for supervised institutions, covering how a model is developed and used, how it is validated and monitored, who governs it, and how third-party dependencies are controlled, is exactly the reasoning a RAG system needs applied to it. Finance and CRE teams should borrow that framework in that spirit, as a design pattern for their own evaluation and governance, rather than treat it as a compliance obligation that already covers them.

The model-risk question is practical: what can go wrong, how would we know, and who can stop it before it becomes part of a decision record?

Evaluation should be built into the workflow

For a finance or CRE research team, RAG evaluation should live where work happens. A useful operating pattern has five steps.

First, maintain a source registry. Separate public regulatory sources, internal research, transaction documents, market data, and academic papers. Track retrieval permissions, update cadence, and source owner. A stale source can be more dangerous than no source because it gives the answer borrowed authority.

Second, build query sets around real work. Use representative questions from investment committee memos, market screens, research briefs, rent roll reviews, loan covenant checks, and policy summaries. Synthetic questions help at scale, but the first benchmark should come from tasks the team already performs.

Third, score retrieval before generation. If the system cannot find the right passage, improving the prompt will not solve the core issue. Measure whether the retrieved set contains the relevant document, the relevant section, and enough surrounding context to support the answer.

Fourth, review citation fidelity. Every material claim should map to a source. The test is not whether the source is plausible. The test is whether the source supports the specific sentence. This matters in finance because a single citation can carry more weight than the surrounding prose.

Fifth, log the trace. The reviewer should be able to see the question, retrieved documents, extracted snippets, model output, warnings, and any human edits. NIST's Generative AI Profile, a companion to the AI Risk Management Framework, is useful here because it pushes organizations toward risk management across design, development, use, and evaluation rather than treating evaluation as a one-time test.

Regulation is moving toward accountability, not novelty

The regulatory signal is not that AI systems require a special exception. It is that existing obligations still matter. FINRA Regulatory Notice 24-09 reminded member firms that AI use does not relieve them from obligations under securities laws, FINRA rules, and other applicable regulations. The U.S. Treasury's 2024 report on AI in financial services also describes industry experimentation with generative AI for internal business operations and flags risk-management themes across the sector.

For RAG systems, that means the most durable product design is boring in the right places. Permissions matter. Source freshness matters. Human review matters. Logs matter. So do controls around third-party models and data providers. The more consequential the workflow, the less acceptable it is to treat a generated answer as a standalone artifact.

The practical standard

A finance RAG system is ready for serious work when it can answer three questions before the user trusts the output.

What evidence did it use? The answer should expose source names, dates, passages, and links or internal document identifiers.

What evidence did it not have? The system should be able to say when the corpus is incomplete, stale, out of scope, or permission-limited.

How was the answer checked? The workflow should show retrieval tests, citation checks, reviewer actions, and known limitations.

That standard is not anti-AI. It is the condition under which AI becomes useful in institutional settings. Fast prose is not enough. Source-grounded research infrastructure needs retrieval, evaluation, and governance working together so the team can defend what it writes.