yujunjun/olmocr
1
0
Fork 0
mirror of https://github.com/allenai/olmocr.git synced 2025-08-31 12:25:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Decent set of todos for monday

Browse Source
This commit is contained in:
Jake Poznanski 2024-11-18 04:54:12 +00:00
parent 2f1664f3d7
commit b4ca5636bc
1 changed files with 8 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
pdelfin/beakerpipeline.py
View File

@ -804,5 +804,11 @@ if __name__ == "__main__":
asyncio.run(main())
# TODO
# Possible future addon, in beaker, discover other nodes on this same job
# Send them a message when you take a work item off the queue
# - Refactor the work queue into its own file so it's reusable and generic, and it makes temporary work files (prevent issue where if a work item is done, then it stalls because queue was just emptied)
# - Fix the queue release mechanism so that it just does a timeout, based on zero queue size only, so you don't block things
# - Add the page rotation check and mechanism
# - Sglang commit a fix for the context length issue
# - Get a solid benchmark on the stream vs non stream approach
# - sglang error on s3://ai2-s2-pdfs/73ee/35e7ed5c2fb113ceba652284aaa51db7c2fc.pdf-2
# - Client error on attempt 0 for s3://ai2-s2-pdfs/e13c/9e03ce463ba53bfb15b26dbfd55c0bbc5568.pdf-1: 400, message='Bad Request',
# - Fix loading of the model checkpoints, it's so flakey now, maybe use datasets