Researchers Propose Evidence-Order Calibration for Vision-Language Models

In a new research preprint published on arXiv, researchers investigated the answer-level reliability of vision-language models (VLMs) under progressive visual evidence degradation. Using the frozen Qwen2.5-VL-3B-Instruct model, the authors evaluated 176 trajectories derived from GQA across 880 masking conditions.

The study showed that masking question-critical regions reduced model accuracy by 28.2 percentage points, compared to just 0.6 percentage points for non-critical regions. However, the model's native sequence confidence exhibited an evidence monotonicity violation rate (EMVR) of 0.436, with 92.0% of trajectories containing at least one adjacent violation.

To address this inconsistency, researchers trained a lightweight post-hoc reliability head. Adding evidence-order supervision reduced the masking EMVR from 0.330 to 0.303, and from 0.449 to 0.402 under unseen local Gaussian blur. The authors noted that the method specifically improves evidence-order consistency rather than establishing general confidence superiority.