The Reproducibility Gap in Financial AI Agents: What Three New Papers Show

Three papers published between April and June 2026 converge on a single, unglamorous problem: the AI agents entering trading desks and research workflows are usually evaluated on whether their final answer looks right, not on whether the process that produced it can be checked. For finance and commercial real estate teams deciding how much to trust an agent's output, that distinction is the whole ballgame.

Taken together, a systematic review of LLM trading agents, a new benchmark for financial-research agents, and a grounding pipeline for financial document question answering describe the same failure pattern from three different angles: plausible outputs, thin audit trails.

A systematic review finds most trading-agent studies cannot be checked

The most striking of the three is Agentic Trading: When LLM Agents Meet Financial Markets, a May 2026 paper that reframes LLM-based trading agents as expert-system decision pipelines and builds an audit-oriented evidence map of the literature. The authors screened studies through March 9, 2026, and included 77 in their review. Only 19 of those met a minimum empirical bar of producing a tradable action and reporting a closed-loop evaluation; the rest were retained as background and design context rather than evidence.

Inside that primary subset of 19 studies, the paper's central finding is what it calls protocol incomparability. Just 2 of the 19 report an extractable, time-consistent train-test split. Only 1 documents an explicit transaction-cost model. Only 1 addresses universe selection or survivorship bias. Eleven report execution timing or semantics at all. Fifteen of the 19 fall into the lowest reproducibility tier the authors define, and none reach the highest tier they set out. In plain terms, most published claims that an LLM trading agent beats a benchmark cannot currently be independently reproduced or even fairly compared against one another, because the studies do not report the details a reproduction would require.

That is not a claim that LLM trading agents do not work. It is a claim that the evidence base supporting performance claims is thinner than the confident tone of many papers and product pitches would suggest. A backtest without a documented cost model or survivorship-bias check is not a backtest a risk committee should rely on.

A benchmark shows the destination matters less than the path

BigFinanceBench, released in June 2026, applies a similar audit-first logic to research agents rather than trading agents. The authors argue that a financial-research answer is only decision-relevant if another analyst can audit how it was produced: which source was used, which period and accounting definition applied, which assumptions were made, and how the calculation was performed. Most existing finance benchmarks score the final answer, or an isolated subskill, and leave that derivation unmeasured.

BigFinanceBench is a 928-item, expert-authored benchmark of open-ended financial-research tasks. Each item pairs a ground-truth reference answer with a point-weighted rubric that breaks the derivation into independently checkable steps, totaling 36,241 rubric points across the benchmark. When the authors evaluated ten current frontier and open-weight agents against it, the best system reached only 58.8 percent of the available rubric score. The paper's headline methodological point is that final-answer accuracy is a useful but lossy proxy for derivation quality, and that model capability varies unevenly across different parts of a research workflow rather than moving up or down as a single number.

For a research or underwriting team, that finding has a direct implication. A model can land on the correct answer to a question, such as a company's leverage ratio or a property's effective rent growth, while getting there through a chain of reasoning that would not survive review. Grading only the destination hides exactly the failures a reviewer most needs to catch.

Grounding claims to source documents cuts hallucination, but does not remove the need for review

The third paper addresses a narrower but closely related failure mode: fabricated numbers and invented citations when an LLM answers questions against financial filings. FinGround, posted in April 2026, is a three-stage verify-then-ground pipeline built specifically for financial document question answering. It performs finance-aware hybrid retrieval across text and tables, decomposes a draft answer into atomic claims, verifies each claim through type-specific strategies that include reconstructing the underlying formula, and rewrites any unsupported claim with a precise citation back to the source filing.

The authors also propose what they call retrieval-equalized evaluation, a methodology that holds the retrieval step constant across systems so that comparisons isolate the grounding and verification stage rather than conflating it with differences in search quality. Under that controlled comparison, the verify-then-ground pipeline reduced hallucination rates by 68 percent relative to the strongest baseline the authors tested, and by 78 percent relative to GPT-4o when the full pipeline was compared end to end. A distilled 8-billion-parameter version of the claim detector preserved most of that accuracy at a reported cost of roughly $0.003 per query, and the authors report validating the system through a four-week pilot with working analysts.

FinGround is the most encouraging of the three papers because it shows hallucination in financial document QA is addressable with current techniques, not an unsolvable property of LLMs. But a 68 to 78 percent reduction is a reduction, not a guarantee. A pipeline that still produces some unsupported claims after grounding is safer than one without grounding, but it is not a substitute for a reviewer checking the citation trail on anything that will inform a decision.

What this means for practitioners

Read together, these three papers argue for the same operating discipline regardless of whether the agent in question is trading a book, drafting a research memo, or answering a question against a 10-K. First, ask what tier of evidence supports a performance claim. A trading agent's backtest is not evidence of edge if it lacks a transaction-cost model or a survivorship-bias check, per the Agentic Trading review's own scoring. Second, do not accept a correct final answer as proof of a sound process. BigFinanceBench shows the gap between final-answer accuracy and full-derivation accuracy can be wide, and that gap is exactly where a bad assumption hides. Third, treat citation-grounded output as a floor, not a ceiling. FinGround's results show grounding substantially reduces fabrication, but the residual error rate after grounding is still a number a review process needs to catch, not zero.

None of the three papers argue against using AI agents in financial workflows. They argue for the same thing model risk teams have always argued for: know what evidence backs a claim, know how the claim was derived, and know what happens when the underlying data or assumptions are stale or wrong. In 2026, that discipline applies as much to an agent that drafts a rent-roll summary or a credit memo as it does to one that places a trade.