Can LLMs be trusted to automatically detect inconsistencies within documents?
Kensho's FIND dataset and accompanying ACL 2026 research show that top AI models reliably surface over 60% of inconsistencies in complex documents, and even catch errors the document authors missed.
In financial documents, a single misplaced digit can lead to millions of dollars lost. Every day, S&P Global produces and ingests large amounts of long, complex documents such as bond disclosures, financial analyses, governmental filings, and presale reports. If something in a document doesn’t add up, then something in the document is wrong.
Manually auditing the information in a long, technical document takes time, effort, and domain expertise. S&P Global invests significant resources in doing just that. Experts interviewed during our research study, “On Finding Inconsistencies in Documents”, noted that reviewing just the tables in a company’s annual report takes about four hours. The reviewing and error-checking processes when preparing a 10-K can take a large team weeks.
A natural question is whether AI tools can assist the auditing process, either by accelerating reviews or checking a larger volume of documents than a human ever could. After all, LLMs perform well on domain-specific tasks such as financial modeling, QA, and reasoning. However, no existing datasets measured how well models could actually perform.
We built and released the FIND dataset (Finding Inconsistencies in Documents) to evaluate how well AI systems like LLMs can detect inconsistent information in professional documents including financial documents, academic research papers, and non-fiction books. Financial experts edited real documents to introduce inconsistencies, applying specialized domain knowledge to test everything from figure references to derived table values (like sums).
The top-performing models that we evaluated found over half of the inserted inconsistencies, as well as inconsistencies in the original documents that went undetected by their authors. Our team presented our work and dataset at this year’s Association for Computational Linguistics conference (ACL 2026).
Building FIND
FIND contains 500 documents (125 for model development, 375 for model evaluation) from numerous sources: reports from the Bureau of Labor Statistics, presale reports from S&P Global, 10-Q filings, bond disclosures from the Municipal Securities Rulemaking Board (MSRB), and non-fiction books from Project Gutenberg. Supplementary analysis of model performance drew upon an additional set of miscellaneous financial reports (such as annual reports) and academic research papers from arXiv.
The documents in FIND are long: the median length is about 60 pages (35k tokens), and a quarter of the documents are about 130 pages (78k tokens) or longer. We formalized inconsistencies as consisting of snippets of text called evidence spans and a natural language description of how those evidence spans are inconsistent with each other.
The task is to find inconsistencies within an input document. This example shows five distinct evidence spans and a generated description summarizing the inconsistency.
To construct FIND, financial experts edited documents to introduce a wide array of realistic inconsistencies. The annotators deliberately varied how close together or far apart the inconsistent evidence spans were, whether the inconsistencies involved numeric, non-numeric, or structural information (like references to page numbers or figure numbers), and whether the information was presented in tables, text, or both.
By inserting inconsistencies into documents, then prompting the models to return all the inconsistencies they could find, we could track whether the AI systems found the known inserted ones. Even in cases where we can’t verify particular information using only what’s in the document, such as companies’ self-reported revenue claims, we can at least check the necessary condition of whether the information in the document is self-consistent.
Beyond the benchmark: Real inconsistencies in real documents
| Model | BLS Reports | S&P Presales | Form 10-Q | Bond Disclosures | Books | Avg |
|---|---|---|---|---|---|---|
| GPT-5 mini | 80 | 53 | 47 | 37 | 39 | 51 |
| GPT-5 | 87 | 67 | 73 | 43 | 52 | 64 |
| GPT-5.4 mini | 85 | 72 | 69 | 49 | 61 | 67 |
| GPT-5.5 | 92 | 83 | 84 | 60 | 63 | 76 |
| Gemini 2.5 Flash | 75 | 48 | 36 | 25 | 31 | 43 |
| Gemini 2.5 Pro | 88 | 71 | 61 | 40 | 45 | 61 |
Performance of a selection of LLMs at correctly identifying inconsistencies on the FIND dataset. Higher is better. Scores are presented as percentages (out of 100). Results on more models are available in the paper.
Of the models we tested, GPT-5 performed the best, finding 64% of the inconsistencies inserted by the experts. Gemini 2.5 Pro was not far behind, finding 61% of them. Newer models released after the writing of the paper perform even better, with GPT-5.5 identifying 76% of the inconsistencies. We also found a negative correlation between document length and model performance, with models performing much better on shorter documents like Bureau of Labor Statistics reports than on longer ones such as bond disclosures.
Initially, we observed that models weren’t finding the inconsistencies we expected. However, when we looked closer and inspected the models’ predictions, we found that many of the inconsistencies identified were not added by our experts but actually present in the original source documents. We dug deeper and analyzed predictions from the best model (GPT-5) that did not correspond to the inconsistencies we had inserted. These were candidates for real inconsistencies in the original unedited documents.
More than half of these analyzed predictions were judged by financial experts to represent a legitimate inconsistency. Two thirds of the predictions revealed either inconsistencies or near-inconsistencies that were helpful to point out (such as confusing or ambiguous information that might be inconsistent depending on outside context). We judged about 70% of GPT-5’s predictions on research papers to be legitimate inconsistencies, with over 90% of the predictions being either inconsistencies or near-inconsistencies. On average, GPT-5 found about 1 undiscovered inconsistency per financial document and about 3 undiscovered inconsistencies per research paper.
Models still make mistakes and can produce incorrect predictions. At their current capabilities, the models seem to demonstrate legitimate utility when used with caution. In fact, we ran GPT-5 and Gemini 2.5 Pro on drafts of our own research paper, and they caught minor inconsistencies (none of which affected the conclusions of the work).
A legitimate inconsistency found by Gemini 2.5 Pro on a draft of our research paper, “On Finding Inconsistencies in Documents”. The issue arose due to an out-of-date table in the paper’s appendix.
Building toward more powerful document AI
This work represents an early but promising milestone for AI-assisted inconsistency detection, and marks another branch in Kensho's broader research into document AI capabilities. Some of the best models available show promise as a tool to supplement domain experts’ careful manual review of complex financial documents. These models also surfaced real inconsistencies in source documents that a more mature tool could help prevent, though there’s still much room for model improvement. Failure modes remain, particularly on long documents, late-appearing evidence spans, and structural rather than substantive inconsistencies. None of these insights would have been possible without FIND.
We plan to explore additional research opportunities including impact studies on real workflows, incorporating visual modalities like charts and figures, and expanding detection across multiple documents. Each of these paths will help us push the document AI frontier forward and deepen our understanding of how well AI systems handle complex, information-dense documents.
Fixing an inconsistency presents an even harder challenge: because many plausible resolutions exist, choosing the right one requires ground truth from outside the document. S&P Global's trusted data puts Kensho in a strong position to take that problem on.
We enjoyed the warm reception to our paper, “On Finding Inconsistencies in Documents” at ACL 2026, and we’re excited to see how the community uses the FIND dataset in their own work.