yujunjun/olmocr
1
0
Fork 0
mirror of https://github.com/allenai/olmocr.git synced 2025-07-24 09:27:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
582 Commits 49 Branches 18 Tags
Commit Graph

582 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jake Poznanski
c2909f314e run pipeline 2024-10-09 19:55:45 +00:00
Jake Poznanski
954b19a5d4 Stuff 2024-10-09 19:55:04 +00:00
Jake Poznanski
991b213cf5 Refactoring, startng to write run_pipeline 2024-10-09 18:48:31 +00:00
Jake Poznanski
4bf6e7a430 Refactoring 2024-10-09 18:11:18 +00:00
Jake Poznanski
0c56dec704 Adding diff to tinyhost 2024-10-09 17:53:26 +00:00
Jake Poznanski
400e92180b Unifying some of the pdf rendering stuff 2024-10-09 16:57:13 +00:00
Jake Poznanski
dc6440d068 Cleaning up anchor text to deal with abnormally long lines 2024-10-09 16:29:20 +00:00
Jake Poznanski
b6b74b7832 Rewriting prompts to eval with new model 2024-10-09 16:04:39 +00:00
Jake Poznanski
7c19a9a856 fix 2024-10-08 23:54:17 +00:00
Jake Poznanski
ad10add6c1 try lower lr 2024-10-08 23:52:56 +00:00
Jake Poznanski
230c8a9f9a Trying new run that will rewrite the prompts as it goes 2024-10-08 22:10:18 +00:00
Jake Poznanski
97291b3f6a Anchor is fixed to sample text elements better 2024-10-08 21:51:43 +00:00
Jake Poznanski
c8a4d14c57 Adding image merging to pdf report/hint/anchor 2024-10-08 21:23:21 +00:00
Jake Poznanski
57d9a21eeb Adding prompt length histogram to a script 2024-10-08 18:22:56 +00:00
Jake Poznanski
adc702c918 FIxing wandb key 2024-10-08 18:16:39 +00:00
Jake Poznanski
085937859f Lower lr 2024-10-08 17:52:00 +00:00
Jake Poznanski
4b30dd867b Fixing eval script, working FSDP config 2024-10-08 16:56:07 +00:00
Jake Poznanski
f5fd9ff53a Trying grad checkpoint 2024-10-08 16:11:31 +00:00
Jake Poznanski
4fb7e9b184 Updated eval script 2024-10-08 16:09:25 +00:00
Jake Poznanski
fb4e585e9f Trying out non-lora training 2024-10-08 15:20:37 +00:00
Jake Poznanski
ec09408ca9 Filtering based on cpu count 2024-10-07 15:40:29 -07:00
Jake Poznanski
a90eb94951 Fix dataloader bug 2024-10-07 15:25:48 -07:00
Jake Poznanski
3d36545fa5 loading fix for parquets again... 2024-10-07 14:48:53 -07:00
Jake Poznanski
fdcd77eadd typo 2024-10-07 14:32:47 -07:00
Jake Poznanski
7416b42023 Adding support for parquet datasets which are precached 2024-10-07 21:14:33 +00:00
Jake Poznanski
dc26541da2 Starting code to build parquets... 2024-10-07 20:59:43 +00:00
Jake Poznanski
4557a5b296 Typo 2024-10-07 13:03:31 -07:00
Jake Poznanski
e973de7ba9 Typo 2024-10-07 13:01:43 -07:00
Jake Poznanski
ebd40f9084 Hopefully fixing dataloader for now 2024-10-07 12:59:27 -07:00
Jake Poznanski
5d35461dd2 Fix for unicode errors in big datasets for the future 2024-10-07 17:01:59 +00:00
Jake Poznanski
44bcdc771b Hopefully can use weka for the train datasets now 2024-10-07 16:14:28 +00:00
Jake Poznanski
d8e459c9f3 Weird issue with surrogate pairs in json 2024-10-07 09:04:13 -07:00
Jake Poznanski
98020cabbb Allow loading files locally 2024-10-07 07:49:16 -07:00
Jake Poznanski
13123ddea4 Pinning datasets to work around weird issue 2024-10-06 03:56:27 +00:00
Jake Poznanski
568dd48509 Prepping for qwen2vl full training run 2024-10-05 04:04:45 +00:00
Jake Poznanski
6065da268b Hopefully working better 2024-10-04 18:06:04 +00:00
Jake Poznanski
a2ff849a78 checkpoint on new runner for openai batches 2024-10-04 17:32:35 +00:00
Jake Poznanski
2da901d433 new better runopenaibatch script 2024-10-04 16:58:38 +00:00
Jake Poznanski
35ec67c427 Hopefully finishing touches 2024-10-04 16:10:19 +00:00
Jake Poznanski
db36608b42 Fix 2024-10-04 16:05:08 +00:00
Jake Poznanski
f25cb6c261 Fixes 2024-10-04 15:54:00 +00:00
Jake Poznanski
4630f7b1cb Bugfixes 2024-10-04 15:35:52 +00:00
Jake Poznanski
e87729a653 New send silver script for testing 2024-10-04 15:27:43 +00:00
Jake Poznanski
6e1094ee8a Support for more evals and output formats 2024-10-03 20:19:52 +00:00
Jake Poznanski
974ddd3773 I'm pretty sure we only need to save on rank0 2024-10-03 11:30:44 -07:00
Jake Poznanski
8f1fa4f796 Running a mini config again with metric 2024-10-03 11:12:30 -07:00
Jake Poznanski
046d4a4534 Adding eval on start and seed params 2024-10-03 10:54:25 -07:00
Jake Poznanski
2227605bfb Mini train config 2024-10-03 10:32:15 -07:00
Jake Poznanski
4505a49420 Pinning to normal transformers version now 2024-10-03 09:00:53 -07:00
Jake Poznanski
78e3a94173 Adding pluto ib 2024-10-03 15:33:17 +00:00