We are developing AI systems that read medical images together with the patientโs clinical data to improve diagnostic accuracy. Some field notes from our latest experiments.
Why multimodal matters
A chest X-ray alone is often ambiguous. Combining it with the patientโs history, lab values and notes gives the model the context a radiologist would have. In a retrospective cohort, our multimodal model improved the AUC on a hard-to-diagnose subset from 0.82 to 0.89.
Fusion strategy we chose
| Strategy | AUC | Note |
|---|---|---|
| Image only | 0.82 | Strong baseline |
| Late fusion | 0.86 | Concatenate pooled features |
| Cross-attention | 0.89 | Image attends over EHR text |
Cross-attention wins but costs 2x GPU memory. We are exploring efficient attention to close the gap.
Open problems
- Handling missing modalities at inference time
- Calibrating confidence for high-stakes screening
- Domain shift across hospitals and scanner vendors
Next step: an external validation study with a partner hospital. Stay tuned.