Extract and Summarise With AI

Extractive summarisation copies sentences directly from the source; abstractive summarisation generates new sentences that paraphrase and condense — the two approaches trade factual precision against readability and compression.

Extractive summarisation — used by tools like Scholarcy — ranks sentences by keyword frequency, position, and information density, then reproduces the top-scoring sentences without modification. Because no new text is generated, factual errors are structurally impossible: the output is always a subset of the source. Abstractive summarisation — used by GPT-4o (OpenAI), Claude Sonnet 4.6 (Anthropic), and Gemini 3.1 Pro (Google DeepMind) — generates new text that synthesises and paraphrases, producing more readable output at the cost of a higher hallucination risk.

NotebookLM (Google DeepMind) leads for source-grounded, cited summarisation of uploaded documents; Claude Sonnet 4.6 (Anthropic) leads for synthesis, cross-document analysis, and complex reasoning; GPT-4o (OpenAI) leads for fast, flexible general-purpose summarisation.

On Vectara's Hughes Hallucination Evaluation Model (HHEM) — the standard benchmark for document summarisation faithfulness, tested across 831 documents per model — the top performers in 2025 were:

These rates represent a 96% improvement from 2021, when the top models scored 21.8% hallucination rates on the same task. However, these numbers apply only to grounded summarisation — where the model is anchored to a source document. Open-domain factual recall produces hallucination rates of 3–33% across the same models.

Tested in PromptQuorum — 25 document summarisation prompts dispatched across three models: Claude Sonnet 4.6 produced the most analytically complete summaries (identifying implications and connections between documents) in 20 of 25 cases. GPT-4o produced the most concise, immediately usable summaries in 18 of 25 cases. Gemini 3.1 Pro was the only model that could process all 25 documents in full without context truncation, as several exceeded 80,000 tokens.

A structured summarisation prompt — one that specifies the document type, output format, length constraint, and explicit instruction to flag unverifiable claims — produces directly usable outputs; an unstructured prompt produces a generic paragraph that misses critical information.

The most common prompt engineering failure in summarisation is treating "summarise this" as a complete instruction. Every assumption the model makes about length, format, perspective, and level of detail is a potential mismatch with what you actually need. The 5-block prompt structure — Role, Task, Input, Constraints, Output Format — applies directly to extraction tasks.

Bad prompt — unstructured, produces generic unusable output:

The structured prompt produces a document directly usable in a briefing. The open prompt produces a narrative paragraph that omits segment data, buries guidance changes, and requires 30 minutes of restructuring.

With 1M token context windows now standard across GPT-4o, Claude Sonnet 4.6, and Gemini 3.1 Pro, most single documents fit within the context window without chunking. Chunking remains essential for: (1) multi-document synthesis exceeding 800 pages, (2) smaller or local models with limited context (Mistral 7B: 32K, LLaMA 3.3 8B: 128K), and (3) improving faithfulness on very long documents where "lost in the middle" degradation occurs — models pay most attention to the beginning and end of long contexts.

For documents exceeding the model's context window, chunking — splitting the document into segments of 500–2,000 tokens, summarising each chunk, then synthesising the chunk summaries — preserves information that would otherwise be truncated or degraded.

For documents with clear section structures (legal contracts, annual reports, academic papers), thematic chunking produces the most coherent final synthesis. For unstructured documents (email threads, transcripts), paragraph-based chunking at 500-token intervals is the recommended default.

Iterative summarisation — generating an initial summary, then refining it with a second targeted prompt — improves factual completeness and reduces omissions compared to single-pass generation.

Iterative summarisation generates an initial summary, then applies a second prompt to catch missing claims. The two-step structure:

Grounded summarisation hallucination rates have dropped 96% since 2021 — from 21.8% to 0.7% for the top models — but a 2025 mathematical proof confirmed that hallucinations cannot be fully eliminated under current LLM architectures.

The architecture reason is fundamental: LLMs generate statistically probable next tokens based on pattern matching across training data, not by retrieving verified facts. Even when given a source document, a model occasionally "blends" source content with training knowledge in a way that produces a plausible but unfaithful sentence — what researchers call a "mixed context hallucination." This is one of the core AI limitations that grounded summarisation workflows must account for.

The failure modes in AI summarisation, ordered by frequency:

Note: The Vectara HHEM benchmark results are from 2025, tested on previous-generation models (GPT-4o, Gemini 2.0 Flash). Current frontier models (GPT-4o, Claude Sonnet 4.6, Gemini 3.1 Pro) are expected to achieve equal or better faithfulness scores. Updated benchmarks will be incorporated when published by their respective vendors.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation), BERTScore, and faithfulness metrics measure different and non-overlapping dimensions of summary quality — no single metric is sufficient to evaluate whether an AI summary is trustworthy.

ROUGE measures n-gram overlap between a generated summary and a reference summary — useful for benchmarking but blind to semantic meaning and factual accuracy. BERTScore uses cosine similarity between BERT embeddings of the generated and reference summaries, capturing semantic similarity rather than exact word matches. Faithfulness metrics (HHEM, FaithJudge) measure whether the summary contains only claims supported by the source document — the most relevant metric for production summarisation use cases.

European enterprises processing documents under GDPR cannot send sensitive content to external API endpoints without compliance review. Mistral AI (France) provides locally deployable models — Mistral Large and Mistral Small — that perform abstractive summarisation entirely on-premise, with zero data leaving the organisation's infrastructure, satisfying EU data residency requirements under Article 46 of GDPR.

Chinese enterprises increasingly use Qwen 3 (Alibaba) and DeepSeek V3 for document extraction tasks across Chinese-language corpora. Both models tokenise Chinese characters (CJK scripts) at a more efficient ratio than Western-trained models — a 10,000-character Chinese document consumes roughly 40% fewer tokens in Qwen 3 than in GPT-4o, making large-scale Chinese document processing significantly cheaper. China's Interim Measures for Generative AI (2023) require AI-generated summaries used in official contexts to be labelled as AI-generated.

Japanese enterprises operating under METI data governance guidelines frequently deploy Ollama with LLaMA 4 models for local document summarisation. LLaMA 4 7B requires 8GB RAM for local inference and produces zero external API calls — meeting strict data residency requirements for sensitive legal and financial documents.