EVLF-FM

EVLF-FM is a medical multimodal model for disease diagnosis, visual grounding, and visual question answering. It is based on InternVL3-8B. This version is trained on publicly available datasets.

Code · Evaluation

Performance

Benchmark	Metric	Score
MedMNIST	Mean accuracy / F1	85.13 / 81.63
BiomedParse	Mean IoU / accuracy at IoU 0.5	74.44 / 84.44
NIH	Mean class accuracy / F1	75.10 / 54.80
VinDr	Mean class accuracy / F1	92.60 / 63.70

Usage

Python 3.9 and a CUDA GPU with BF16 support.

pip install torch==2.1.0 torchvision==0.16.0 --index-url https://download.pytorch.org/whl/cu121
pip install huggingface-hub==0.34.4
hf download yangbai123/EVLF-FM --local-dir EVLF-FM
pip install -r EVLF-FM/requirements.txt
pip install flash-attn==2.4.0.post1 --no-build-isolation

CUDA_VISIBLE_DEVICES=0 python EVLF-FM/inference.py \
  --image image.jpg \
  --question "What modality is used to take this image?" \
  --task vqa

Tasks: vqa, cls, grounding, or raw. Use --no-flash-attn to run without FlashAttention. Decoding is greedy with up to 512 new tokens.

import sys
sys.path.insert(0, "EVLF-FM")
from inference import load_model, predict

model, tokenizer = load_model("EVLF-FM")
answer = predict(model, tokenizer, "image.jpg",
                 "What modality is used to take this image?", task="vqa")
print(answer)

Batch inference

from inference import predict_batch

answers = predict_batch(
    model, tokenizer,
    image_paths=["image1.jpg", "image2.jpg"],
    questions=["What modality is used to take this image?", "Is there a fracture?"],
    task="vqa", batch_size=8,
)

For JSONL input (question or text fields are accepted):

{"question_id": "1", "image": "image1.jpg", "question": "What modality is used to take this image?"}
{"question_id": "2", "image": "image2.jpg", "question": "Is there a fracture?"}

CUDA_VISIBLE_DEVICES=0 python EVLF-FM/inference.py \
  --input questions.jsonl --image-root /path/to/images \
  --output predictions.jsonl --task vqa --batch-size 8

JSON arrays from the original evaluator are also supported. Output preserves the input order and fields, adding response.

Citation

@article{bai2026evlffm,
  title   = {A clinical multimodal vision language foundation model with fine-grain explainability},
  author  = {Bai, Yang and Cheng, Haoran and Zhou, Yang and Zhou, Jun and
             Thirunavukarasu, Arun and Ke, Yuhe and Yao, Jie and Fukutsu, Kanae and
             Quek, Chrystie Wan Ning and Hong, Ashley Shuen Ying and
             Gutierrez, Laura and Berni, Alessandro and Teo, Zhen Ling and
             Ting, Darren Shu Jeng and Soetikno, Brian T. and Nielsen, Christopher S. and
             Elze, Tobias and Li, Zengxiang and Sharma, Sumit and Ruthmann, Nicholas and
             Linh, Le Dinh and Chan, Hiok Hong and Koh, Victor and Tan, Marcus and
             Li, Kelvin Z. and Yip, Leonard and Cheng, Ching Yu and Tham, Yih Chung and
             Tan, Gavin Siew Wei and Schmetterer, Leopold and Kim, Rae-Young and
             Wong, Damon and Ang, Marcus and Hussain, Rahat and Mehta, Jodhbir S. and
             Aung, Tin and Cheng, Lionel Tim-Ee and Tran, Nguyen Tuan Anh and
             Cheng, Chee Leong and Wong, Tien Yin and Moshfeghi, Darius M. and Liu, Nan and
             Tan, Iain Beehuat and Lim, Soon Thye and Klang, Eyal and Lim, Tony Kiat Hon and
             Goh, Rick Siow Mong and Liu, Yong and Ting, Daniel Shu Wei},
  journal = {Nature Medicine},
  year    = {2026},
  doi     = {10.1038/s41591-026-04655-5},
  url     = {https://www.nature.com/articles/s41591-026-04655-5}
}