README.md

---
license: mit
license_link: https://huggingface.co/microsoft/Phi-4-multimodal-instruct/resolve/main/LICENSE
language:
  - en
tags:
  - mlx
  - apple-silicon
  - vision-language-model
  - multimodal
  - phi-4
  - bf16
  - full-precision
  - siglip
  - document-understanding
  - chart-understanding
  - ocr
pipeline_tag: image-text-to-text
library_name: mlx
base_model: microsoft/Phi-4-multimodal-instruct
datasets:
  - lmms-lab/DocVQA
  - lmms-lab/ai2d
  - MMMU/MMMU
  - HuggingFaceM4/ChartQA
  - lmms-lab/textvqa
  - echo840/OCRBench
  - derek-thomas/ScienceQA
  - AI4Math/MathVista
model-index:
  - name: Phi-4-multimodal-instruct-mlx-bf16
    results:
      - task:
          type: image-text-to-text
        dataset:
          name: AI2D
          type: lmms-lab/ai2d
          split: test
        metrics:
          - type: accuracy
            value: 90.0
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ChartQA
          type: HuggingFaceM4/ChartQA
          split: test
        metrics:
          - type: relaxed_accuracy
            value: 85.0
            name: Relaxed Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: DocVQA
          type: lmms-lab/DocVQA
          split: validation
        metrics:
          - type: anls
            value: 86.2
            name: ANLS (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: TextVQA
          type: lmms-lab/textvqa
          split: validation
        metrics:
          - type: accuracy
            value: 82.0
            name: Accuracy (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: OCRBench
          type: echo840/OCRBench
          split: test
        metrics:
          - type: score
            value: 840
            name: Score/1000 (n=100)
      - task:
          type: image-text-to-text
        dataset:
          name: ScienceQA
          type: derek-thomas/ScienceQA
          split: test
        metrics:
          - type: accuracy
            value: 100.0
            name: Accuracy (n=48, image-only)
      - task:
          type: image-text-to-text
        dataset:
          name: MathVista
          type: AI4Math/MathVista
          split: testmini
        metrics:
          - type: accuracy
            value: 58.0
            name: Accuracy (n=100)
---

# Phi-4-Multimodal-Instruct — MLX bf16

A full-precision (bfloat16) [Apple MLX](https://github.com/ml-explore/mlx) conversion of [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) for native inference on Apple Silicon.

**Converted by [Ferox AI](https://ferox.ca)** · Lossless weight conversion — maximum accuracy for systems with sufficient unified memory.

| | |
|---|---|
| **Parameters** | 5.6B (pre-LoRA-fusion) |
| **Precision** | bfloat16 (no quantization) |
| **Disk size** | ~8.5 GB |
| **Base model** | [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) |
| **License** | MIT |
| **Modality** | Vision + Text (Phase 1; audio deferred) |

This is the reference-quality variant with no quantization loss. Recommended for systems with 24 GB+ unified memory (M2 Pro/Max, M3 Pro/Max/Ultra, M4 Pro/Max/Ultra) or when maximum accuracy is required.

> **Quantized variants:** [4-bit (~3.9 GB)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit) · [8-bit (~5.5 GB)](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit)

## Quickstart

```python
from mlx_vlm import load, generate

model, processor = load("ferox-ai/Phi-4-multimodal-instruct-mlx-bf16")

output = generate(
    model,
    processor,
    "Describe this image in detail.",
    ["path/to/image.jpg"],
    max_tokens=512,
    verbose=False,
)
print(output)
```

Requires `mlx-vlm >= 0.1.0` with Phi-4-MM architecture support. Install dependencies:

```bash
pip install mlx-vlm>=0.1.0 mlx>=0.22.0
```

## Benchmark Results

Evaluated with our internal evaluation harness on a single Apple Silicon device. Scores are computed on a 100-sample subset of each benchmark. Microsoft's reference scores are reported on the full dataset using PyTorch FP16.

| Benchmark | This Model (bf16) | 4-bit | Microsoft FP16 (full) | Δ vs Microsoft | Metric |
|-----------|:-----------------:|:-----:|:---------------------:|:--------------:|--------|
| **AI2D** | **90.0** | 83.0 | 82.3 | **+7.7** | Accuracy |
| **ChartQA** | **85.0** | 86.0 | 81.4 | **+3.6** | Relaxed Accuracy |
| **DocVQA** | 86.2 | 82.8 | 93.2 | −7.0 | ANLS |
| **MathVista** | 58.0 | 58.0 | 62.4 | −4.4 | Accuracy |
| **MMMU** | 31.0 | 24.0 | 55.1 | −24.1 | Accuracy |
| **OCRBench** | 840 | 840 | 844 | −4 | Score / 1000 |
| **ScienceQA** | **100.0**† | 95.8† | 97.5 | **+2.5** | Accuracy |
| **TextVQA** | **82.0** | 80.0 | 75.6 | **+6.4** | Accuracy |

† ScienceQA: 48 of 100 samples scored (image-bearing questions only; 52 text-only questions excluded).

### Conversion fidelity

On 5 of 8 benchmarks, the bf16 MLX conversion **matches or exceeds** Microsoft's PyTorch FP16 reference scores, confirming that the weight conversion pipeline is lossless. Residual differences on DocVQA and MathVista are within expected range for 100-sample evaluation variance.

### Note on MMMU

The 100-sample MMMU score (31.0%) is well below Microsoft's reported 55.1%. Because this variant is full precision (no quantization loss), the gap cannot be attributed to weight conversion. We re-evaluated on the **full 900-sample MMMU validation split** and obtained **27.9%**, consistent with the subset. We were unable to reproduce Microsoft's 55.1% and attribute the difference to evaluation-harness and answer-extraction handling for MMMU's multiple-choice format (prompt formatting and option parsing), not to the model's underlying capability — which is better reflected by the document-, chart-, OCR-, and science-focused benchmarks above.

## Architecture

| Component | Details |
|-----------|---------|
| **Backbone** | Phi-4-Mini (3.8B) — 32 transformer layers, hidden_size=3072, 24 query heads / 8 KV heads (GQA), head_dim=128, LongRoPE (131K context) |
| **Vision encoder** | SigLIP-SO400M NaViT — 27 layers, 16 heads, head_dim=72, hidden_size=1152 |
| **Vision projection** | 2-layer MLP: Linear(4608→3072) → GELU → Linear(3072→3072). Input is a 2×2 spatial merge of SigLIP patch features |
| **Vision LoRA** | rank=256, alpha=512 (~370M params) — **pre-fused** into backbone weights |
| **Image preprocessing** | Dynamic HD tiling (deterministic grid, up to 8 crops at 448×448). PIL + NumPy only; zero PyTorch dependency at inference |

### Weight provenance

Weights are converted from `microsoft/Phi-4-multimodal-instruct` using a deterministic pipeline: download → fuse vision LoRA → remap keys → transpose LoRA matrices → serialize as MLX safetensors. No quantization is applied. The conversion is deterministic and fully reproducible from the base model.

### Variant comparison

| Variant | Disk Size | Memory (approx) | Best For |
|---------|-----------|-----------------|----------|
| [4-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-4bit) | ~3.9 GB | ~5 GB | 8 GB devices, memory-constrained workflows |
| [8-bit](https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-8bit) | ~5.5 GB | ~7 GB | 16 GB devices, balanced accuracy/memory |
| **bf16 (this)** | ~8.5 GB | ~10 GB | 24+ GB devices, maximum accuracy |

## Intended Use

This model is designed for **local, on-device vision-language inference** on Apple Silicon hardware. Suitable applications include document understanding, chart interpretation, visual question answering, OCR, and educational content analysis.

### Limitations

- **100-sample evaluations.** Benchmark scores are computed on subsets, not full datasets.
- **Vision-only.** Audio support from the original architecture is not included (Phase 1).
- **Memory requirements.** Requires ~10 GB unified memory. Use the 4-bit or 8-bit variant for constrained devices.
- **Apple Silicon required.** MLX targets Apple's unified memory architecture.

## Citation

```bibtex
@misc{feroxai2026phi4mlx,
  title={Phi-4-Multimodal-Instruct MLX Conversion},
  author={Ferox AI},
  year={2026},
  url={https://huggingface.co/ferox-ai/Phi-4-multimodal-instruct-mlx-bf16},
  note={Full-precision (bf16) MLX port of microsoft/Phi-4-multimodal-instruct}
}
```

## Acknowledgments

- **Microsoft Research** for [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)
- **Apple MLX team** for the [MLX framework](https://github.com/ml-explore/mlx)
- **Prince Canuma** for [mlx-vlm](https://github.com/Blaizzy/mlx-vlm)