From 9d260791a0bf915c04bebf779937bdf1d251de1c Mon Sep 17 00:00:00 2001 From: Jake Poznanski Date: Tue, 17 Jun 2025 19:58:06 +0000 Subject: [PATCH] README updates --- README.md | 65 +++++++++++++++++++++--------------------- olmocr/bench/README.md | 7 +++-- 2 files changed, 37 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 8067aa6..817be6b 100644 --- a/README.md +++ b/README.md @@ -49,18 +49,30 @@ We also ship a comprehensive benchmark suite covering over 7,000 test cases acro Model - AR - OSM - TA - OS - HF - MC - LTT + ArXiv + Old Scans Math + Tables + Old Scans + Headers and Footers + Multi column + Long tiny text Base - Overall Score + Overall + + Marker v1.7.5 (base) + 76.0 + 57.9 + 57.6 + 27.8 + 84.9 + 72.9 + 84.6 + 99.1 + 70.1 ± 1.1 + MinerU v1.3.10 75.4 @@ -75,43 +87,32 @@ We also ship a comprehensive benchmark suite covering over 7,000 test cases acro Mistral OCR API - 77.2 + 77.2 67.5 60.6 29.3 93.6 71.3 77.1 - 99.4 + 99.4 72.0 ± 1.1 - Marker v1.7.5 (base) - 76.0 - 57.9 - 57.6 - 27.8 - 84.9 - 72.9 - 84.6 - 99.1 - 70.1 ± 1.1 - - - olmOCR v0.1.68 (pipeline.py) - 75.6 - 75.1 - 70.2 - 44.5 - 93.4 - 79.4 - 81.7 - 99.0 - 77.4 ± 1.0 + olmOCR v0.1.75 (Anchored) + 74.9 + 71.2 + 71.0 + 42.2 + 94.5 + 78.3 + 73.3 + 98.3 + 75.5 ± 1.0 + ### Installation Requirements: diff --git a/olmocr/bench/README.md b/olmocr/bench/README.md index 58e0ff3..18bd4aa 100644 --- a/olmocr/bench/README.md +++ b/olmocr/bench/README.md @@ -121,7 +121,7 @@ to run it against your own OCR tools. Your tool just needs to support Markdown o Gemini Flash 2 (Anchored) 54.5 56.1 - 72.1 + 72.1 34.2 64.7 61.5 @@ -158,7 +158,7 @@ to run it against your own OCR tools. Your tool just needs to support Markdown o 71.5 71.4 71.4 - 42.8 + 42.8 94.1 77.7 71.0 @@ -172,7 +172,7 @@ to run it against your own OCR tools. Your tool just needs to support Markdown o 71.0 42.2 94.5 - 78.3 + 78.3 73.3 98.3 75.5 ± 1.0 @@ -180,6 +180,7 @@ to run it against your own OCR tools. Your tool just needs to support Markdown o + There was a small drop in scores from olmOCR v0.1.68 (77.4), which is due to two factors. One, is that we have adjusted our benchmark code to not include any "fallback" mechanism when measuring benchmark scores (though it still exists when you run olmocr.pipeline). Second, there is a small drop in scores as we have updated from sglang 0.4.2 to vllm 0.9.1. In net, we think the upgrade to vllm is the right choice, given that sglang 0.4.6 had even lower scores by one point, and vllm comes with a