
[RolmOCR](https://huggingface.co/reducto/RolmOCR) from Reducto is a powerful, open-source document OCR solution that delivers superior performance while requiring fewer resources than comparable models. Compared to its predecessor, [allenai/olmOCR-7B-0225-preview](https://huggingface.co/allenai/olmOCR-7B-0225-preview), this new model is built on a better base model, Qwen2.5-VL-7B instead of Qwen2.0-VL-7b. This model isn't trained to take in metadata information and with some 15% rotation of the documents, which enables it to be more flexible under different conditions. Overall, it excels at OCR on PDF's, documents, forms, and invoices. Because it is open source, companies can build on top of this model for their own proprietary pipelines while not sending data to other providers or model hosting services.

Vast.ai offers a GPU marketplace where you can rent compute power at lower costs than major cloud providers, with the flexibility to select specific hardware configurations optimized for specific models and GPU RAM needs, while keeping all of your company's data private.

In this guide, we'll demonstrate how to extract structured pricing data from invoice images using this new model. To follow along, we also have a notebook version you can find [here](https://nbviewer.org/urls/bitbucket.org/%21api/2.0/snippets/jsbcannell/nEbMzk/c6b76123dbef385aa3576dcda376eb2db876fc28/files/reducto_rolmocr_vast.ipynb)

## Deploying RolmOCR on Vast

### Install Vast

First, we'll install and set up the Vast API.

You can get your API key from the [Account Page](https://cloud.vast.ai/account/) in the Vast Console and set it below in `VAST_API_KEY`.

```bash
pip install vastai==0.2.6
```

```bash
export VAST_API_KEY="" # Your API key here
vastai set api-key $VAST_API_KEY
```

### Search for an Instance

Next, we'll search for an instance to host the model. While `reducto/RolmOCR` requires at least 16GB VRAM, selecting an instance with 60GB VRAM will accommodate larger documents and enable a wider context window.

```bash
vastai search offers "compute_cap >= 750 \
geolocation=US \
gpu_ram >= 60 \
num_gpus = 1 \
static_ip = true \
direct_port_count >= 1 \
verified = true \
disk_space >= 80 \
rentable = true"
```

### Deploy the Instance

Using the instance ID from the search, deploy the model to the instance.

Note: we set `VLLM_USE_V1=1` to use the v1 engine for vLLM, which `reducto/RolmOCR` requires.

```bash
export INSTANCE_ID= # insert instance ID here
vastai create instance $INSTANCE_ID --image vllm/vllm-openai:latest --env '-p 8000:8000 -e VLLM_USE_V1=1' --disk 80 --args --model reducto/RolmOCR
```

## Using the OpenAI API to Call `reducto/RolmOCR`

### Download Dependencies

While waiting for the instance to finish setting up and starting to serve the model, we'll install the required Python packages locally.

```bash
pip install --upgrade openai datasets pydantic
```

### Download Sample Invoice Data

Now, we'll use the `datasets` library to load a subset of invoice data from the `katanaml-org/invoices-donut-data-v1` dataset on Hugging Face. This dataset contains 500 annotated invoice images with structured metadata for training document extraction models.

```python
from datasets import load_dataset

# Stream the dataset
streamed_dataset = load_dataset("katanaml-org/invoices-donut-data-v1", split="train", streaming=True)

# Take the first 3 samples
subset = list(streamed_dataset.take(3))
```

Next, we'll create an `encode_pil_image` function to convert images from the dataset into a base64 encoded string format suitable for the OpenAI API.

```python
import base64
import io
from PIL import Image

def encode_pil_image(pil_image):
    # Resize image while maintaining aspect ratio
    max_size = 1024  # Maximum dimension
    ratio = min(max_size / pil_image.width, max_size / pil_image.height)
    new_size = (int(pil_image.width * ratio), int(pil_image.height * ratio))
    resized_image = pil_image.resize(new_size, Image.Resampling.LANCZOS)

    # Convert to bytes with JPEG format and reduced quality for smaller size
    img_byte_arr = io.BytesIO()
    resized_image.save(img_byte_arr, format='JPEG', quality=85)
    img_byte_arr = img_byte_arr.getvalue()
    return base64.b64encode(img_byte_arr).decode("utf-8")
```

### Enforcing Structured Data Extraction

We'll create an `Invoice` Pydantic model for use to store the invoice number and the amount.

```python
from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_number: str
    invoice_amount: str

json_schema = Invoice.model_json_schema()
```

## Configuring the API Client

Here we're creating a function to interface with the RolmOCR endpoint via the OpenAI API.

Set your instance's IP address and port from the [Instances tab](https://cloud.vast.ai/instances/) of the Vast AI Console.

You can see that we're leveraging the json schema fromt he above `Invoice` Model to pull have our model generate structured data.

```python
from openai import OpenAI

VAST_IP_ADDRESS = ""  # Insert your instance IP address
VAST_PORT = ""        # Insert your instance port
base_url = f"http://{VAST_IP_ADDRESS}:{VAST_PORT}/v1"

client = OpenAI(api_key="", base_url=base_url)

model = "reducto/RolmOCR"

def ocr_page_with_rolm(img_base64):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/png;base64,{img_base64}"},
                    },
                    {
                        "type": "text",
                        "text": "Return the invoice number and total amount for each invoice as a json: {invoice_number : str, invoice_amount: str}",
                    },
                ],
            }
        ],
        extra_body={"guided_json": json_schema},
        temperature=0.2,
        max_tokens=500
    )
    return response.choices[0].message.content
```

### Extract and Display Invoice Data

Iterate over the sample subset of invoices, display the image, extract the invoice number and amount with RolmOCR, and compare the extracted data to the ground truth.

```python
import matplotlib.pyplot as plt
import json

invoices = []
ground_truth = []
for sample in subset:
    # Display invoice image
    plt.figure(figsize=(10, 14))
    plt.imshow(sample['image'])
    plt.axis('off')
    plt.show()

    # Encode and process with OCR
    img_base64 = encode_pil_image(sample['image'])
    result = ocr_page_with_rolm(img_base64)
    result_dict = json.loads(result)
    invoices.append(result_dict)

    # Extract ground truth fields
    ground_truth_i = json.loads(sample["ground_truth"])
    ground_truth_dict = {
        "invoice_number": ground_truth_i["gt_parse"]["header"]["invoice_no"],
        "invoice_amount": ground_truth_i["gt_parse"]["summary"]["total_gross_worth"]
    }
    ground_truth.append(ground_truth_dict)

    print("Ground Truth")
    print(ground_truth_dict)

    print("Extracted Info")
    print(result_dict)
```

### Example Outputs

#### Invoice 1

![Invoice 1](/uploads/reducto_rolmocr_output_1.png)

```
Ground Truth
{'invoice_number': '40378170', 'invoice_amount': '$8,25'}
Extracted Info
{'invoice_number': '40378170', 'invoice_amount': '$8.25'}
```

#### Invoice 2

![Invoice 2](/uploads/reducto_rolmocr_output_2.png)

```
Ground Truth
{'invoice_number': '61356291', 'invoice_amount': '$ 212,09'}
Extracted Info
{'invoice_number': '61356291', 'invoice_amount': '$212.09'}
```

#### Invoice 3

![Invoice 3](/uploads/reducto_rolmocr_output_3.png)

```
Ground Truth
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
Extracted Info
{'invoice_number': '49565075', 'invoice_amount': '$96,73'}
```

As observed, the output from `reducto/RolmOCR` closely matches the expected ground truth, demonstrating its accuracy in extracting key invoice information. The differences between the extracted and ground truth in these cases is around commas versus decimal points, which can be easily solved for with simple downstream methods.

## Conclusion

In this walkthrough, we demonstrated how to deploy and use RolmOCR on Vast to extract structured data from invoice images efficiently.

RolmOCR delivers high accuracy for critical fields such as invoice numbers and amounts, while staying more robust with image-based proxessing and reduced RAM usage. When combined with Vast.ai's flexible and cost-effective GPU marketplace, this forms a powerful solution for scalable, privacy-focused document processing workflows.

This approach can be extended to extract a wide range of structured data from diverse document formats, unlocking advanced automation and insights for businesses of all sizes.


Learn how to deploy Reducto’s RolmOCR on Vast.ai to extract structured invoice data using affordable GPUs and accurate open-source document OCR.

Posts about: Ocr

Running RolmOCR on Vast