---
library_name: transformers
tags:
- onnx
- vlm
- gui-automation
- pix2struct
- transformers.js
- webgpu
license: apache-2.0
base_model: google/pix2struct-base
---

# Pix2Act ONNX

ONNX export of [google/pix2struct-base](https://huggingface.co/google/pix2struct-base) 
for GUI automation via Transformers.js/WebGPU.

## Model Details

| Property | Value |
|----------|-------|
| **Original Model** | google/pix2struct-base |
| **Architecture** | Pix2Struct (Vision2Seq) |
| **Parameters** | ~282M |
| **Export Format** | ONNX (opset 17) |
| **Quantization** | INT8 dynamic |
| **Browser Eligible** | Yes |

## Description

Pix2Struct is a pretrained image-to-text model designed for understanding visually 
situated language. It uses a pixel-only approach making it ideal for:

- Screenshot understanding
- GUI automation
- Document parsing
- Web interaction

## Usage

### With ONNX Runtime (Python)

```python
import onnxruntime as ort
from transformers import AutoProcessor
from PIL import Image

# Load processor
processor = AutoProcessor.from_pretrained("google/pix2struct-base")

# Load ONNX model
session = ort.InferenceSession("encoder_model.onnx")

# Process image
image = Image.open("screenshot.png")
inputs = processor(images=image, return_tensors="np")

# Run inference
outputs = session.run(None, dict(inputs))
```

### With Transformers.js (Browser)

```javascript
import { pipeline } from "@xenova/transformers";

// Load the ONNX model
const pipe = await pipeline("image-to-text", "bollscoasts/pix2act-onnx");

// Run inference on screenshot
const result = await pipe(image);
console.log(result);
```

## Export Details

Exported using [HotelBench](https://github.com/bsneider/hotelbench) ONNX export utilities.

```bash
hotelbench export-onnx pix2act -o models/pix2act-onnx --quantize
```

## License

Apache 2.0