Researchers Combine Vision and Language Models for Chest X-Ray Classification

A scientific preprint published on arXiv introduces a framework for multi-label chest X-ray classification that integrates unimodal visual representations with vision-language data. The approach combines embeddings from RAD-DINO and BioViL-T within a latent space to classify 14 labels in the MIMIC-CXR-JPG dataset.

In testing, the hybrid fusion framework outperformed standalone models, achieving a mean AUROC of 0.840 and a mean Average Precision (mAP) of 0.467. The results demonstrate that the visual and clinical semantic representations provide complementary diagnostic signals.

The authors note that the study relied exclusively on internal evaluation using MIMIC-CXR-JPG, meaning the model's generalizability across data from external healthcare institutions remains to be verified.