yujunjun/olmocr
1
0
Fork 0
mirror of https://github.com/allenai/olmocr.git synced 2025-08-18 13:52:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
693 Commits 55 Branches 24 Tags
Commit Graph

693 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jake Poznanski
49b5b233c3 Working on new pipeline script 2024-10-10 22:10:26 +00:00
Jake Poznanski
a8b50ae8fa Preloading the datasets directly 2024-10-10 19:57:51 +00:00
Jake Poznanski
85f2dc6d26 Fixes 2024-10-10 18:52:42 +00:00
Jake Poznanski
2864f907e1 Dataloader fix with nicer tests 2024-10-10 16:58:45 +00:00
Jake Poznanski
b7c80cd17f Fix up some tests but I don't see why this isn't working 2024-10-10 16:58:40 +00:00
Jake Poznanski
3245990216 Faster eval script 2024-10-10 15:22:33 +00:00
Jake Poznanski
931f48c3d1 Allow eval script to support one more type of jsonls, runpipeline multiglobs, other fixes 2024-10-09 23:39:13 +00:00
Jake Poznanski
c6bdf69d8f First stab at document assembly 2024-10-09 22:19:16 +00:00
Jake Poznanski
847064f46f Taking notes, starting on document assembly 2024-10-09 22:14:28 +00:00
Jake Poznanski
8e5809da71 runpipeline 2024-10-09 20:29:59 +00:00
Jake Poznanski
a90feda42f bugfixes 2024-10-09 20:20:06 +00:00
Jake Poznanski
c2909f314e run pipeline 2024-10-09 19:55:45 +00:00
Jake Poznanski
954b19a5d4 Stuff 2024-10-09 19:55:04 +00:00
Jake Poznanski
991b213cf5 Refactoring, startng to write run_pipeline 2024-10-09 18:48:31 +00:00
Jake Poznanski
4bf6e7a430 Refactoring 2024-10-09 18:11:18 +00:00
Jake Poznanski
0c56dec704 Adding diff to tinyhost 2024-10-09 17:53:26 +00:00
Jake Poznanski
400e92180b Unifying some of the pdf rendering stuff 2024-10-09 16:57:13 +00:00
Jake Poznanski
dc6440d068 Cleaning up anchor text to deal with abnormally long lines 2024-10-09 16:29:20 +00:00
Jake Poznanski
b6b74b7832 Rewriting prompts to eval with new model 2024-10-09 16:04:39 +00:00
Jake Poznanski
7c19a9a856 fix 2024-10-08 23:54:17 +00:00
Jake Poznanski
ad10add6c1 try lower lr 2024-10-08 23:52:56 +00:00
Jake Poznanski
230c8a9f9a Trying new run that will rewrite the prompts as it goes 2024-10-08 22:10:18 +00:00
Jake Poznanski
97291b3f6a Anchor is fixed to sample text elements better 2024-10-08 21:51:43 +00:00
Jake Poznanski
c8a4d14c57 Adding image merging to pdf report/hint/anchor 2024-10-08 21:23:21 +00:00
Jake Poznanski
57d9a21eeb Adding prompt length histogram to a script 2024-10-08 18:22:56 +00:00
Jake Poznanski
adc702c918 FIxing wandb key 2024-10-08 18:16:39 +00:00
Jake Poznanski
085937859f Lower lr 2024-10-08 17:52:00 +00:00
Jake Poznanski
4b30dd867b Fixing eval script, working FSDP config 2024-10-08 16:56:07 +00:00
Jake Poznanski
f5fd9ff53a Trying grad checkpoint 2024-10-08 16:11:31 +00:00
Jake Poznanski
4fb7e9b184 Updated eval script 2024-10-08 16:09:25 +00:00
Jake Poznanski
fb4e585e9f Trying out non-lora training 2024-10-08 15:20:37 +00:00
Jake Poznanski
ec09408ca9 Filtering based on cpu count 2024-10-07 15:40:29 -07:00
Jake Poznanski
a90eb94951 Fix dataloader bug 2024-10-07 15:25:48 -07:00
Jake Poznanski
3d36545fa5 loading fix for parquets again... 2024-10-07 14:48:53 -07:00
Jake Poznanski
fdcd77eadd typo 2024-10-07 14:32:47 -07:00
Jake Poznanski
7416b42023 Adding support for parquet datasets which are precached 2024-10-07 21:14:33 +00:00
Jake Poznanski
dc26541da2 Starting code to build parquets... 2024-10-07 20:59:43 +00:00
Jake Poznanski
4557a5b296 Typo 2024-10-07 13:03:31 -07:00
Jake Poznanski
e973de7ba9 Typo 2024-10-07 13:01:43 -07:00
Jake Poznanski
ebd40f9084 Hopefully fixing dataloader for now 2024-10-07 12:59:27 -07:00
Jake Poznanski
5d35461dd2 Fix for unicode errors in big datasets for the future 2024-10-07 17:01:59 +00:00
Jake Poznanski
44bcdc771b Hopefully can use weka for the train datasets now 2024-10-07 16:14:28 +00:00
Jake Poznanski
d8e459c9f3 Weird issue with surrogate pairs in json 2024-10-07 09:04:13 -07:00
Jake Poznanski
98020cabbb Allow loading files locally 2024-10-07 07:49:16 -07:00
Jake Poznanski
13123ddea4 Pinning datasets to work around weird issue 2024-10-06 03:56:27 +00:00
Jake Poznanski
568dd48509 Prepping for qwen2vl full training run 2024-10-05 04:04:45 +00:00
Jake Poznanski
6065da268b Hopefully working better 2024-10-04 18:06:04 +00:00
Jake Poznanski
a2ff849a78 checkpoint on new runner for openai batches 2024-10-04 17:32:35 +00:00
Jake Poznanski
2da901d433 new better runopenaibatch script 2024-10-04 16:58:38 +00:00
Jake Poznanski
35ec67c427 Hopefully finishing touches 2024-10-04 16:10:19 +00:00