2271 Commits

Author	SHA1	Message	Date
Nicolas	93e106d321	Update v0.ts	2024-11-20 16:43:02 -08:00
Nicolas	3eaa3b38ab	Nick: formatting	2024-11-20 16:42:42 -08:00
Nicolas	c78dae178b	Merge branch 'main' into nsc/new-extract	2024-11-20 16:41:13 -08:00
Nicolas	945183ffbd	Update extract.ts	2024-11-20 16:40:55 -08:00
Nicolas	d196b9d93d	Update extract.ts	2024-11-20 13:16:36 -08:00
Nicolas	9512d81e05	Update extract.ts	2024-11-20 13:15:52 -08:00
Nicolas	3de4997f4d	Loggin num tokens	2024-11-20 13:09:46 -08:00
Nicolas	769f08c10d	Billing and log for extract	2024-11-20 13:08:09 -08:00
Nicolas	0e4e9a3b37	Nick:	2024-11-20 13:01:36 -08:00
Nicolas	09dd5136b7	Update build-document.ts	2024-11-20 12:51:16 -08:00
Nicolas	67a2989874	Nick: fixes	2024-11-20 12:48:10 -08:00
Nicolas	28696da6b2	Nick: gpt-4o	2024-11-20 12:25:50 -08:00
Nicolas	d49f62fb56	Nick: extract fixes	2024-11-20 11:50:14 -08:00
Gergő Móricz	b1eaecfdb0	fix 2	2024-11-20 20:19:16 +01:00
Gergő Móricz	e2ddc6c65c	fix handling of badly formatted URLs	2024-11-20 20:18:40 +01:00
Gergő Móricz	ba6f29cdda	crawl fix, again	2024-11-20 19:55:35 +01:00
Gergő Móricz	b468bb4014	crawl fixes	2024-11-20 19:48:01 +01:00
Nicolas	c9b0a80522	Nick:	2024-11-20 10:23:44 -08:00
Nicolas	103c3f28e6	Update rate-limiter.ts	2024-11-19 17:51:31 -08:00
Gergő Móricz	79a75e088a	feat(crawl): allowSubdomain	2024-11-19 18:38:59 +01:00
rafaelmmiller	2fb8a3c8dc	fix schema	2024-11-19 10:04:42 -03:00
rafaelmmiller	53134b7c85	Rafa: removed throw error and added map to requests	2024-11-19 09:34:52 -03:00
rafaelmmiller	36cf49c959	Merge remote-tracking branch 'origin/main' into nsc/new-extract	2024-11-19 09:34:08 -03:00
rafaelmmiller	77e152cba8	added team_id to scrape-status endpoint	2024-11-18 15:02:00 -03:00
Gergő Móricz	31a0471bfa	fix(crawl-redis): ordered push to wrong side of list	2024-11-15 21:56:15 +01:00
Gergő Móricz	1a0f13c0eb	fix(webhook): add logging	2024-11-15 21:43:02 +01:00
Gergő Móricz	1b032b05fa	fix(map): make sitemapOnly simpler	2024-11-15 21:14:32 +01:00
Gergő Móricz	a4d3dba865	fix(map): ignore limit when using sitemapOnly	2024-11-15 21:03:20 +01:00
Gergő Móricz	63787bc504	fix(scrapeURL/fire-engine): wait longer if timeout is not specified	2024-11-15 20:25:16 +01:00
Gergő Móricz	4cddcd5206	fix(scrapeURL/fire-engine): timeout-less scrape support (initial)	2024-11-15 20:15:25 +01:00
Gergő Móricz	350d00d27a	fix(crawler): treat XML files as sitemaps (temporarily)	2024-11-15 20:09:20 +01:00
Gergő Móricz	ca2e33db0a	fix(log_job): add force option to retry on supabase failure	2024-11-15 19:55:23 +01:00
Gergő Móricz	7b02c45dd0	fix(v1/types): better timeout primitives	2024-11-15 19:35:54 +01:00
Gergő Móricz	c95a4a26c9	fix(v1/batch/scrape): raise default timeout	2024-11-15 18:58:03 +01:00
Móricz Gergő	3a342bfbf0	fix(scrapeURL/playwright): JSON body fix	2024-11-15 15:18:40 +01:00
Nicolas	3c1b1909f8	Update map.ts	2024-11-14 17:52:15 -05:00
Nicolas	7f084c6c43	Nick:	2024-11-14 17:44:32 -05:00
Nicolas	3fcdf57d2f	Update fireEngine.ts	2024-11-14 17:31:30 -05:00
Nicolas	d62f12c9d9	Nick: moved away from axios	2024-11-14 17:31:23 -05:00
Nicolas	f155449458	Nick: sitemap only	2024-11-14 17:29:53 -05:00
Móricz Gergő	431e64e752	fix(batch/scrape/webhook): add batch_scrape.started	2024-11-14 22:40:03 +01:00
Móricz Gergő	df05124ef5	feat(v1/batch/scrape): webhooks	2024-11-14 22:36:28 +01:00
Nicolas	91f4fd815f	Update extract.ts	2024-11-14 15:41:42 -05:00
Nicolas	22848af5ae	Nick:	2024-11-14 15:34:02 -05:00
Nicolas	ebe9de2ac5	Nick:	2024-11-14 15:26:15 -05:00
Nicolas	5056dcd8e9	Update index.test.ts	2024-11-14 15:06:22 -05:00
rafaelmmiller	be5e6da5ff	tests	2024-11-14 17:03:54 -03:00
Nicolas	796cd0746d	Update extract.ts	2024-11-14 15:03:06 -05:00
Nicolas	1b5f6a0959	Update extract.ts	2024-11-14 14:59:34 -05:00
Nicolas	d6749c211d	Nick: refactor and /* glob pattern support	2024-11-14 14:57:38 -05:00