112 Commits

Author	SHA1	Message	Date
Jake Poznanski	0b72eda794	Move form check into exception handler, don't mark the work item as done if it had an exception on it	2024-12-04 19:08:21 +00:00
Jake Poznanski	e9c3c21731	Skipping files which are not found	2024-12-04 17:56:45 +00:00
Jake Poznanski	535181d8e0	Moving to manual HTTP Post, have had succss with 10k page files now	2024-12-03 10:48:52 -08:00
Jake Poznanski	fe04db828e	Better error handling	2024-12-02 23:56:45 +00:00
Jake Poznanski	35502bc6f2	Limit the number of retries on the server process	2024-12-02 23:46:46 +00:00
Jake Poznanski	b3ca86a182	More robust to errors when reading logs which had caused freezes	2024-12-02 13:59:42 -08:00
Jake Poznanski	5e3080db28	Sglang based unit test	2024-11-25 09:48:05 -08:00
Jake Poznanski	46fe4acc0b	Trying fixes for live lock	2024-11-23 21:41:49 +00:00
Jake Poznanski	41accfe867	Error out if you see a broken process pool, might need a better check for this	2024-11-22 22:07:43 +00:00
Jake Poznanski	a95487e44c	Adding check for possible sglang livelock	2024-11-22 21:50:45 +00:00
Jake Poznanski	f8dcdf625a	Better catching of httpx errors and retrying them	2024-11-21 23:35:42 +00:00
Jake Poznanski	d6a00135a7	Faster init by caching pdf filter	2024-11-21 23:23:11 +00:00
Jake Poznanski	a91befc4ad	Fix for fallback stuff	2024-11-21 11:08:42 -08:00
Jake Poznanski	b8b786e003	Applying pdf filter	2024-11-21 10:20:58 -08:00
Jake Poznanski	af8ce518ac	Merge branch 'main' of https://github.com/allenai/pdelfin into main	2024-11-21 08:45:19 -08:00
Jake Poznanski	9112d81bd1	No keep alive connection to try to resolve sglang livelock	2024-11-21 08:45:17 -08:00
Jake Poznanski	2443c22fde	Projected output tokens	2024-11-20 23:57:10 +00:00
Jake Poznanski	67d11ec0e6	TODOs and client fix	2024-11-20 14:45:12 -08:00
Jake Poznanski	9b8d58b59e	Better stats and metadata	2024-11-20 10:42:26 -08:00
Jake Poznanski	273a8b0d0a	Logging fallback pages	2024-11-19 15:11:02 -08:00
Jake Poznanski	b0acfa870e	Adding support for fallback pages	2024-11-19 14:59:20 -08:00
Jake Poznanski	204a4a8e5b	Better stats	2024-11-19 13:41:32 -08:00
Jake Poznanski	3ef4609bdd	Fixing args	2024-11-19 11:48:45 -08:00
Jake Poznanski	27d23525b7	Claude recommends httpx instead of aiohttp, seeing if that will help with straggler timeouts	2024-11-19 10:41:58 -08:00
Jake Poznanski	4469f4b2ce	Version patch	2024-11-18 19:55:26 -08:00
Jake Poznanski	9e2e09bd06	More fixes	2024-11-18 15:04:50 -08:00
Jake Poznanski	8793fc7d99	Adding more retries, and it was able to process more complicated books	2024-11-18 14:25:32 -08:00
Jake Poznanski	2f55a3ddb7	fix	2024-11-18 13:58:25 -08:00
Jake Poznanski	d4d47369cb	more gcs	2024-11-18 13:20:28 -08:00
Jake Poznanski	e48d4bef00	Fix	2024-11-18 13:16:19 -08:00
Jake Poznanski	8c3b5753c9	Gcs support better	2024-11-18 13:07:27 -08:00
Jake Poznanski	9381bf862a	docs	2024-11-18 12:44:34 -08:00
Jake Poznanski	f287f2451c	Fixing a few stats things	2024-11-18 11:50:22 -08:00
Jake Poznanski	e499413089	Better work queue	2024-11-18 11:04:51 -08:00
Jake Poznanski	995b1d15fc	Fixes, mocking out queue into separate file	2024-11-18 09:55:45 -08:00
Jake Poznanski	fcabb8e55a	Handling more error cases	2024-11-18 09:12:04 -08:00
Jake Poznanski	96984fcd77	Fix a reliability issue	2024-11-18 09:03:24 -08:00
Jake Poznanski	0af29f1f44	Adding page rotation	2024-11-18 08:29:32 -08:00
Jake Poznanski	e2303f28af	Running on l40s, fixing queue	2024-11-18 08:25:36 -08:00
Jake Poznanski	68543d40ce	Adding stats	2024-11-18 07:57:39 -08:00
Jake Poznanski	b4ca5636bc	Decent set of todos for monday	2024-11-18 04:54:12 +00:00
Jake Poznanski	2f1664f3d7	Stop everything on a Nan	2024-11-16 08:16:11 -08:00
Jake Poznanski	eefb045859	Single cluster fix	2024-11-15 13:30:27 -08:00
Jake Poznanski	2e1d0b6a90	Fix	2024-11-15 13:21:55 -08:00
Jake Poznanski	748b095e0f	Fix	2024-11-15 13:19:23 -08:00
Jake Poznanski	80ba562bb2	Fixing timeout situation	2024-11-15 13:18:13 -08:00
Jake Poznanski	2c52664301	Cleaner exit	2024-11-15 12:54:45 -08:00
Jake Poznanski	77c82fdb3a	New version with aiohttp fixes	2024-11-15 12:48:36 -08:00
Jake Poznanski	ae1e4bc07e	More realistic results	2024-11-15 11:35:10 -08:00
Jake Poznanski	770da2b7ae	Docker	2024-11-15 11:25:24 -08:00