DoCoG: Mask-based Multi-Type Grounded Chain-of-Thought for Document QA (ECCV 2026 Spotlight)

Paper

Abstract

Existing methods for Visual Document Question Answering (DocVQA) often predict answers without explicit intermediate analysis or produce ungrounded intermediate steps. The few approaches which employ grounding produce rectangular bounding boxes which fail to accurately represent complex structures such as pie-chart wedges, flowchart nodes, and curved text. We present DoCoG, a mask-based multi-type grounded step-wise Chain-of-Thought architecture for Document QA.

The model supports multi-type grounding — it localizes both textual content and fine-grained graphical elements such as legends, pie slices, and flowchart nodes. It generates step-wise intermediate outputs and final answers grounded through precise segmentation masks. DoCoG integrates a Vision-Language Model with a promptable segmenter via a novel Grounding Interaction Module (GIM). During decoding, special [GND] tokens trigger mask prediction, linking each step and the final answer to supporting evidence.

We introduce DoCoG-QA, a large-scale diverse dataset comprising 325K documents and 1.5M step-wise grounded QA pairs. We further release DoCoG-PQA with 20K preference pairs for DPO alignment, and DoCoG-Bench, a human-verified benchmark. Experiments show that DoCoG achieves state-of-the-art performance across multiple benchmarks, improving both answer accuracy and grounding precision.

Architecture

Method

DoCoG architecture: Given a document image \(I\) and question \(q\), the VLM 1 encodes visual features and autoregressively generates step-wise outputs interleaved with [GND] tokens 2. Their hidden states 3 pass to GIM 4, which refines them via cross-attention with segmenter features 6 from 5, producing prompt embeddings decoded into evidence masks (2c).

🔗 Decoupled Language & Spatial Grounding

DoCoG decouples language modeling from spatial prediction. The VLM focuses on step-wise reasoning, while a dedicated segmentation module handles fine-grained, geometry-aware mask localization — yielding cleaner text outputs and more precise evidence masks.

🔄 Grounding Interaction Module (GIM)

GIM refines [GND] token embeddings through self-attention and cross-attention with visual features from the segmenter. This ensures grounding tokens remain mutually consistent and spatially aligned with the correct document regions.

Interactive Architecture Walkthrough

Press play to watch the forward pass unfold — tokens flow from the VLM through the Grounding Interaction Module into the segmenter, and each [GND] token lands as an evidence mask on the document.

Drag to pan, scroll to zoom · Open fullscreen

Step-wise Multi-Type Grounding

DoCoG generates intermediate reasoning steps and final answers, each grounded to supporting evidence spanning textual and graphical elements.

📝

Textual Evidence

Words, lines, paragraphs, table cells, headers, and form fields

📊

Graphical Elements

Chart bars, lines, pie slices, legends, axis labels, and plot marks

🔀

Structural Elements

Flowchart nodes, connectors, arrows, and diagram components

Step-wise CoT with [GND] tokens grounds each reasoning step to pixel-accurate evidence masks

Qualitative Results

Model outputs across charts, flowcharts, tables, and documents — each reasoning step grounded to pixel-level evidence.

01 / 06

Try It Interactively

Step through DoCoG's grounded reasoning — each [GND] token sweeps a mask over the supporting evidence in the image.

Training Objectives

DoCoG is trained in two stages: SFT on DoCoG-QA, then DPO on DoCoG-PQA.

𝓛_total = λ_lm𝓛_lm + λ_seg𝓛_seg + λ_anchor𝓛_anchor

𝓛_lm — Language Modeling

Cross-entropy over all output tokens for accurate step-wise CoT generation and answer prediction.

𝓛_seg — Segmentation (Dice + BCE)

Supervises mask predictions against pixel-level ground truth for each [GND] token.

𝓛_anchor — Anchor Loss

Preserves [GND] token compatibility with the frozen segmenter during DPO, preventing mask degradation.

DPO — Preference Optimization

Trained on DoCoG-PQA (20K pairs) to align step quality, answer accuracy, and grounding fidelity jointly.

Data

DoCoG-QA Dataset & Benchmark

Dataset release coming soon. We are preparing DoCoG-QA, DoCoG-PQA, and DoCoG-Bench for public release.

📦 DoCoG-QA

Large-scale training dataset with step-wise grounded Chain-of-Thought, built through a unified data engine across diverse document types with pixel-level mask supervision.

325K

Documents

1.5M

QA Pairs

Step

Wise CoT

Mask

Grounding

ChartsTablesFlowchartsDocumentsCurved textFormsReportsWebpages

📊 DoCoG-Bench

Human-verified benchmark for jointly evaluating answer accuracy, step-wise CoT quality, and mask-level grounding fidelity across diverse document types.

3K

Documents

10K

QA Pairs

Human

Verified

4

Doc Types

AQ (Answer Quality)CQ (CoT Quality)GQ (Grounding Quality)G-Eval

🎯 DoCoG-PQA

20K preference pairs for Direct Preference Optimization (DPO), constructed using a reward function combining AQ, CQ, and GQ scores to select chosen/rejected samples for alignment training.

Evaluation

Results

DoCoG consistently outperforms both commercial and open-source baselines. While commercial models achieve competitive answer accuracy, their grounding quality remains substantially lower.

85.8

AQ (All)
DoCoG-Bench

82.0

CQ (All)
DoCoG-Bench

81.9

GQ_fa (All)
DoCoG-Bench

Hover bars for score · Sorted by performance (high → low)

Unified Evaluation on DoCoG-Bench

AQ and GQ_fa per split. "All" reports AQ, G_eval, CQ, GQ_all, and GQ_fa.

Model	Params	Charts		Flowcharts		Docs		Tables		All
Model	Params	AQ	GQ_fa	AQ	GQ_fa	AQ	GQ_fa	AQ	GQ_fa	AQ	G_eval	CQ	GQ_all	GQ_fa
GPT-5	–	82.4	3.8	78.1	4.0	86.7	5.0	84.3	6.4	82.9	84.8	76.8	2.4	4.8
Sonnet 4.5	–	79.1	5.7	75.3	13.3	83.8	6.8	81.6	2.6	79.9	80.5	71.3	3.3	7.1
Gemini 3 Pro	–	83.2	11.2	79.4	10.1	88.1	11.7	86.0	12.6	84.2	84.1	75.2	15.4	11.4
Gemini 3 Flash	–	79.6	12.7	75.8	12.5	84.5	35.3	82.4	33.9	80.6	82.8	73.4	32.8	23.6
InternVL 3.5	8B	73.4	4.1	67.2	1.5	77.1	7.2	78.0	3.2	73.9	78.2	33.2	2.9	4.0
Qwen3-VL	8B	75.2	3.9	69.4	11.0	78.6	6.3	79.5	0.9	75.7	77.6	35.9	3.5	5.5
VisualCoT	7B	52.3	5.5	44.6	3.9	59.7	3.6	55.4	1.0	53.0	52.1	51.2	2.4	3.5
InternVL 3.5 FT	8B	74.6	65.0	68.8	67.9	82.0	64.3	79.1	61.2	76.1	80.4	72.2	61.4	64.6
Qwen3-VL FT	8B	75.8	63.9	70.2	66.9	83.4	61.2	80.6	55.6	77.5	79.8	74.8	57.8	61.9
DoCoG (Ours)	9B	83.5	80.3	82.2	79.2	89.7	85.4	87.8	82.7	85.8	85.2	82.0	78.2	81.9SOTA

Domain-Specific Benchmark Results

Click the metric pills to switch charts. Each domain card shows detailed per-benchmark breakdowns.

📊 Charts

Hover for score · sorted high → low

Model	ChartQA	CQA-Pro	ChartBench	CharXiv	Bench AQ
GPT-5	89.0	42.7	69.4	76.3	82.4
Chart-R1	91.0	44.0	34.1	46.2	74.2
InternVL 3.5	86.7	34.7	51.4	68.8	73.4
Qwen3-VL	89.6	32.9	65.3	64.7	75.2
DoCoG	92.7	52.1	79.4	73.1	83.5

🔀 Flowcharts

Hover for score · sorted high → low

Model	FlowVQA	AI2D	MISS-QA	Bench AQ	Bench GQ_fa
GPT-5	91.9	89.5	71.7	78.1	4.0
InternVL 3.5	78.4	84.0	48.5	67.2	1.5
Qwen3-VL	81.0	85.7	46.2	69.4	11.0
DoCoG	85.5	88.3	78.4	82.2	79.2

📋 Tables

Hover for score · sorted high → low

Model	TabFact	WTQ	TabMWP	Bench AQ	Bench GQ_fa
GPT-5	87.7	78.8	93.7	84.3	6.4
InternVL 3.5	78.1	55.6	55.7	78.0	3.2
Qwen3-VL	81.8	57.4	65.6	79.5	0.9
DoCoG	89.4	64.4	80.7	87.8	82.7

📄 Documents

Hover for score · sorted high → low

Model	DOGR F1g	MMDB IoU	VCoT	BD F1g	Bench AQ
GPT-5	3.6	9.3	89.7	5.4	86.7
Gemini 3 Pro	65.3	55.4	87.4	73.5	88.1
Qwen3-VL	27.6	28.7	86.7	44.5	78.6
DoCoG	75.4	70.5	88.9	85.4	89.7