(1) a medical triage flowchart — each step's [GND] masks the exact node or connector it walks: cough → sputum → fever → throat pain.
DoCoG · In one picture
Multi-type evidence: whatever shape the step needs
(2a)–(2c) one chart, three grounding behaviours: axis labels (2a) · every team's legend entry + line segment (2b) · one team's, in the final answer (2c).
DoCoG · Architecture
VLM + segmenter, joined by GIM
① VLM (Qwen3-VL-8B) emits steps S = (S₁…Sₙ, Sf) with [GND] ② · hidden states G˜ ③ → GIM ④ cross-attends segmenter features ⑤ Fenc(I) ⑥ · prompt encoder ⑦ + mask decoder → one mask per [GND]
DoCoG · Interactive
Watch the forward pass — tokens flow through GIM into masks
Click inside to drive it · Space pause · R restart · F fit — then use ‹ › to continue the talkopen standalone ↗
DoCoG · The new piece
GIM — because evidence tokens must agree with each other
M3Grounder grounds one query at a time — independent heads suffice. DoCoG emits a [GND] in every step + the answer — refined in isolation, they can contradict.
81.9 → 69.6
final-answer grounding without GIM
78.2 → 73.5
GQall if GIM loses image cross-attention
DoCoG · Training
Training in two stages: SFT → DPO
Stage 1 is plain supervised fine-tuning, end-to-end (VLM + GIM + mask decoder): cross-entropy on the text · Dice + BCE on every predicted evidence mask.
Why a stage 2? SFT only imitates the single annotated chain — fixed-style supervision. It never compares the model's own better and worse chains — DPO adds that.
DoCoG · Training stage 2 · the idea
DPO in one minute: prefer the better answer
No new labels, no reward model — the model learns from its own outputs, ranked better vs worse.
In DoCoG this update touches only the VLM — the part that writes the text. Why everything else stays frozen → next slide.
DoCoG · Training stage 2 · the objective
The DPO formula — every term is text
𝓛DPO = −log σ( β [
logπθ(y✓ | I, q)πref(y✓ | I, q)
the better chain — pushed up
−
logπθ(y✗ | I, q)πref(y✗ | I, q)
the worse chain — pushed down
] )
π(y | I, q) is a text likelihood
y is the chain, as tokens — π is the LM head's probability of writing it. θ = tuned · ref = frozen SFT copy · β = push strength.
So only the VLM can move
Every π comes from the LM head → gradients reach only the 🔥 VLM. GIM + segmenter appear nowhere in the formula → ❄ frozen.
Why no mask term?
A preference needs a likelihood — a mask has none: the decoder emits one mask, not a distribution. Ill-posed, so left out.
The catch — πref leashes the text, not the hidden [GND] states the frozen mask pipeline reads → the drift trap, next.
DoCoG · Training stage 2
DPO on text only — and the drift trap
What DPO does here
From each preference pair, DPO pushes the VLM to make the chosen ✓ chain more likely than the rejected ✗ one, against a frozen reference. Only the VLM's weights change — mask preferences are ill-posed, so GIM + segmenter stay frozen.
70.5 → 40.3
mask IoU without the fix
81.9 → 41.7
GQfa without the fix
The fix — anchor loss
𝓛anchor = (1/K) Σk ‖Gkθ − Gkref‖²
A moving VLM drags the [GND] states with it — this keeps them where the frozen GIM + segmenter expect them.
DoCoG · Training stage 2 · the fix
The anchor loss, unpacked
𝓛opt = 𝓛DPO + λanchor·𝓛anchor
𝓛anchor = (1/K) Σk ‖Gkθ − Gkref‖² · λanchor = 1
Read the symbols
Gk = the k-th GIM-refined prompt embedding — what the mask decoder consumes · ref = frozen SFT copy · θ = the model DPO tunes · mean over the K [GND] tokens.
Why it works
Drift becomes differentiable and punished: DPO improves the reasoning only in directions that keep the grounding interface where the frozen modules expect it.
70.5 kept
mask IoU with the anchor — 40.3 without it
DoCoG · Training stage 2
Stage 2, end to end
84.1 → 85.8
AQ — what stage 2 adds
70.5 held
mask IoU — the anchor at work
One objective: better chains, unmoved grounding interface.
DoCoG · Stage-1 data
DoCoG-QA: 325K docs · 1.5M QA — multi-type by construction
Composition (source documents)
The engine cites element IDs, never coordinates. Verified: gpt-oss-120B judge (a different family, on purpose) · ROSCOE · 33,500 masks checked by hand.
