Skip to main content
What Client-Ready Means: Evaluating AI Agents on Deliverables, Not Demos

What Client-Ready Means: Evaluating AI Agents on Deliverables, Not Demos

Agentic Assets

Agentic Assets Research Team

Agentic Assets Research

August 27, 2026

6 min read

ai-agentsevaluationmodel-riskfinancial-researchgovernancebenchmarks

Every institution weighing an AI agent for analytical work eventually asks the same question: can we delegate this and rely on what comes back. A demo cannot answer it. A rubric can.

BankerToolBench, an open-source benchmark posted to arXiv in April 2026, is a useful model for how to ask. Its authors built it with 502 investment bankers from leading firms, then required agents to execute senior banker requests end to end: navigating data rooms, using a market data platform and an SEC filings database, and producing multi-file deliverables including Excel models, slide decks, and written reports. Deliverables are scored against more than 100 rubric criteria defined by veteran bankers. The paper notes that completing one of these tasks takes bankers up to 21 hours.

The results are the part worth sitting with. Across nine frontier models, the best performer failed nearly half of the rubric criteria, and bankers rated zero percent of its outputs as client-ready. The benchmark is a preprint and has not been peer reviewed, and it measures investment banking workflows rather than commercial real estate ones, so the scores do not transfer directly. The structure of the finding does.

The failure mode is consistency, not fluency

The paper's failure analysis points to breakdowns in cross-artifact consistency: the agent produces a set of deliverables that do not agree with each other. Anyone who has assembled a credit package will recognize the risk immediately. A cash flow model, a committee memo, and a summary deck that each look competent in isolation but carry three different numbers for the same line item is worse than an obviously incomplete draft, because the error survives review.

Fluency is not the constraint. Agents write well. What they do not reliably do is hold one version of the truth across several documents produced in one pass.

No supervisor is going to hand you the bar

It would be convenient if a validation standard already existed to point at. It does not. When the federal banking agencies issued revised model risk guidance on April 17, 2026, the Office of the Comptroller of the Currency release stated that generative AI and agentic AI models "are novel and rapidly evolving" and that "they are not within the scope of this guidance." The agencies said they plan to issue a request for information covering model risk management and bank use of AI.

Being outside the scope of that guidance is not the same as being outside supervision. It means the institution writes its own evidence standard, and then defends it later.

Industry bodies are filling the gap

Parts of real estate finance are not waiting. MISMO released FRAME in June 2026, a governance toolkit that includes a policy template, an AI system inventory, and a risk assessment, positioned as a risk management tool rather than a new regulation. At its Fall Summit in August, MISMO launched two AI governance certifications, one a product-level validation for technology providers that assesses each AI use case on consumer impact, compliance, influence over decisions, and operational risk, the other for advisory partners. Both are mortgage-focused rather than commercial, so a CRE lender adapting them is doing translation work.

What to carry across

Four things in the benchmark's method are worth importing into an internal evaluation, whatever the asset class:

  • Score the deliverable, not the conversation. The artifact is what goes to committee.
  • Write the rubric with practitioners who already do the work, and keep the criteria specific enough to fail.
  • Test cross-artifact consistency explicitly. Ask whether the model, the memo, and the deck agree.
  • Record a client-ready rate, not an average score. An output that is eighty percent right still needs a person, and the rate tells you how much review you are actually buying back.

The useful question is not whether an agent is impressive. It is how much verification its output still requires, measured the same way every time, so the number can move.

Share this article