307 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Jake Poznanski	a4d76206ff	Choosing proper page	2024-10-17 20:18:06 +00:00
Jake Poznanski	529d51d57d	Put LR back, need to save larger checkpoints to weka to prevent timeouts	2024-10-17 19:46:25 +00:00
Jake Poznanski	e141c91e5e	Try lora run higher LR	2024-10-17 17:12:35 +00:00
Jake Poznanski	2826bcad18	Yay all unit tests pass cleanly now too	2024-10-17 17:05:55 +00:00
Jake Poznanski	124aaf5fe0	Hmm, cant repro failing anchor case	2024-10-17 17:00:02 +00:00
Jake Poznanski	1c42a08d06	Fixes to prevent errors later in dataloading	2024-10-17 02:28:43 +00:00
Jake Poznanski	f13bcad943	Adding check that pdfs are valid in the new anchor text generation format	2024-10-16 23:31:40 +00:00
Jake Poznanski	5018d591f6	will try lower lr	2024-10-16 23:27:00 +00:00
Jake Poznanski	5c36c22bf7	Prepping for more training	2024-10-16 23:01:40 +00:00
Jake Poznanski	063be21287	New image	2024-10-16 14:46:28 -07:00
Jake Poznanski	90cb80fd65	Docker update	2024-10-16 21:40:39 +00:00
Jake Poznanski	277723fa2c	Adding cache	2024-10-16 21:18:52 +00:00
Jake Poznanski	87182ab573	Ensuring unique names	2024-10-16 20:44:23 +00:00
Jake Poznanski	4884b8288b	Full dataset	2024-10-16 13:30:25 -07:00
Jake Poznanski	51f1669451	fix	2024-10-16 13:30:06 -07:00
Jake Poznanski	d94713e73e	Truncation handled in a custom collator	2024-10-16 13:28:12 -07:00
Jake Poznanski	cbc667ce78	Prepping to train	2024-10-16 13:18:24 -07:00
Jake Poznanski	9d647b13b8	fix	2024-10-16 11:58:35 -07:00
Jake Poznanski	446773dbc8	First part of new dataloader	2024-10-16 11:54:06 -07:00
Jake Poznanski	202d81cece	Merge branch 'main' of https://github.com/allenai/pdelfin into main	2024-10-16 11:38:33 -07:00
Jake Poznanski	e2552b2f28	Adding test case	2024-10-16 11:38:31 -07:00
Jake Poznanski	d4f64ed82a	Config work	2024-10-16 18:37:52 +00:00
Jake Poznanski	3c1b7de293	Refactoring of train dataloaders	2024-10-16 18:26:25 +00:00
Jake Poznanski	23d129fd2c	Organizing around a new style of dataloader	2024-10-16 18:06:27 +00:00
Jake Poznanski	a2546e0b04	more stuff	2024-10-16 17:06:03 +00:00
Jake Poznanski	a7cd7467c3	mathjax	2024-10-16 16:45:07 +00:00
Jake Poznanski	baa82a4a9a	Fixing links, rendering tables	2024-10-16 16:37:08 +00:00
Jake Poznanski	19e56ec7ce	dolma viewer runs much faster now	2024-10-16 16:21:25 +00:00
Jake Poznanski	96682b2ecb	Refactoring	2024-10-16 16:18:27 +00:00
Jake Poznanski	2cd863ddce	Dolma viewer improvements	2024-10-16 16:05:44 +00:00
Jake Poznanski	35558dbddc	Make the prompt hint randomly select lines	2024-10-16 16:05:07 +00:00
Jake Poznanski	9eb252f8f6	Better tracking of completion_errors	2024-10-15 22:43:31 +00:00
Jake Poznanski	4ef14ec813	More stats	2024-10-15 22:26:31 +00:00
Jake Poznanski	4a280e55df	Nicer dolma viewer	2024-10-15 21:03:28 +00:00
Jake Poznanski	42cf6a639f	Dolma viewer	2024-10-15 18:37:31 +00:00
Jake Poznanski	b8cd414022	tiny fix	2024-10-15 16:54:19 +00:00
Jake Poznanski	a7fae0e659	fix	2024-10-15 16:36:54 +00:00
Jake Poznanski	4669eb7134	Adjusting workflow so I can do s2 pdfs	2024-10-15 16:22:55 +00:00
Jake Poznanski	6d61ae4aa8	Some pipeline cleanup stuff	2024-10-15 16:02:08 +00:00
Jake Poznanski	fc8fcfaeba	Fixing dataloader hopefully	2024-10-15 15:13:25 +00:00
Jake Poznanski	6d53683001	More stats hopefully running faster	2024-10-14 21:37:14 +00:00
Jake Poznanski	350061906e	Adding nicer output stats	2024-10-14 20:48:33 +00:00
Jake Poznanski	194af5ff52	Robustness	2024-10-14 20:31:37 +00:00
Jake Poznanski	1ed9e4c947	Runs to the end now	2024-10-14 20:28:54 +00:00
Jake Poznanski	879b974af2	More and more fixes	2024-10-14 20:06:07 +00:00
Jake Poznanski	77a850d7ef	Tracking rounds of inference better	2024-10-14 18:42:50 +00:00
Jake Poznanski	af992bd603	More refactoring	2024-10-14 18:23:22 +00:00
Jake Poznanski	cd8e28e459	Pipeline working hopefully soon	2024-10-14 18:19:17 +00:00
Jake Poznanski	f2f578cca9	More pipeline code	2024-10-14 17:23:09 +00:00
Jake Poznanski	39333f2c96	New pipeline stuff	2024-10-14 17:09:11 +00:00