1997 Commits

Author	SHA1	Message	Date
Gergő Móricz	31a0471bfa	fix(crawl-redis): ordered push to wrong side of list	2024-11-15 21:56:15 +01:00
Gergő Móricz	1a0f13c0eb	fix(webhook): add logging	2024-11-15 21:43:02 +01:00
Gergő Móricz	1b032b05fa	fix(map): make sitemapOnly simpler	2024-11-15 21:14:32 +01:00
Gergő Móricz	a4d3dba865	fix(map): ignore limit when using sitemapOnly	2024-11-15 21:03:20 +01:00
Gergő Móricz	63787bc504	fix(scrapeURL/fire-engine): wait longer if timeout is not specified	2024-11-15 20:25:16 +01:00
Gergő Móricz	4cddcd5206	fix(scrapeURL/fire-engine): timeout-less scrape support (initial)	2024-11-15 20:15:25 +01:00
Gergő Móricz	350d00d27a	fix(crawler): treat XML files as sitemaps (temporarily)	2024-11-15 20:09:20 +01:00
Gergő Móricz	ca2e33db0a	fix(log_job): add force option to retry on supabase failure	2024-11-15 19:55:23 +01:00
Gergő Móricz	7b02c45dd0	fix(v1/types): better timeout primitives	2024-11-15 19:35:54 +01:00
Gergő Móricz	c95a4a26c9	fix(v1/batch/scrape): raise default timeout	2024-11-15 18:58:03 +01:00
Móricz Gergő	3a342bfbf0	fix(scrapeURL/playwright): JSON body fix	2024-11-15 15:18:40 +01:00
Nicolas	3c1b1909f8	Update map.ts	2024-11-14 17:52:15 -05:00
Nicolas	7f084c6c43	Nick:	2024-11-14 17:44:32 -05:00
Nicolas	3fcdf57d2f	Update fireEngine.ts	2024-11-14 17:31:30 -05:00
Nicolas	d62f12c9d9	Nick: moved away from axios	2024-11-14 17:31:23 -05:00
Nicolas	f155449458	Nick: sitemap only	2024-11-14 17:29:53 -05:00
Móricz Gergő	431e64e752	fix(batch/scrape/webhook): add batch_scrape.started	2024-11-14 22:40:03 +01:00
Móricz Gergő	df05124ef5	feat(v1/batch/scrape): webhooks	2024-11-14 22:36:28 +01:00
Nicolas	91f4fd815f	Update extract.ts	2024-11-14 15:41:42 -05:00
Nicolas	22848af5ae	Nick:	2024-11-14 15:34:02 -05:00
Nicolas	ebe9de2ac5	Nick:	2024-11-14 15:26:15 -05:00
Nicolas	5056dcd8e9	Update index.test.ts	2024-11-14 15:06:22 -05:00
rafaelmmiller	be5e6da5ff	tests	2024-11-14 17:03:54 -03:00
Nicolas	796cd0746d	Update extract.ts	2024-11-14 15:03:06 -05:00
Nicolas	1b5f6a0959	Update extract.ts	2024-11-14 14:59:34 -05:00
Nicolas	d6749c211d	Nick: refactor and /* glob pattern support	2024-11-14 14:57:38 -05:00
Gergő Móricz	359c30fbda	fix(cache): don't cache on failure error code	2024-11-14 19:49:34 +01:00
Gergő Móricz	49ff37afb4	feat: cache	2024-11-14 19:47:12 +01:00
Gergő Móricz	86a78a03cb	fix(sitemap): scrape with tlsclient	2024-11-14 18:56:32 +01:00
Móricz Gergő	5519f077aa	fix(scrapeURL): adjust error message for clarity	2024-11-14 10:13:48 +01:00
Móricz Gergő	0a1c99074f	fix(html-to-markdown): make error reporting less intrusive	2024-11-14 08:58:00 +01:00
Nicolas	bd928b1512	Nick: changed email from hello to help	2024-11-13 20:27:20 -05:00
Nicolas	a1c018fdb0	Merge branch 'main' into nsc/new-extract	2024-11-13 17:14:43 -05:00
rafaelmmiller	904c904971	wip	2024-11-13 18:06:20 -03:00
Gergő Móricz	0310cd2afa	fix(crawl): redirect rebase	2024-11-13 21:38:44 +01:00
Gergő Móricz	32be2cf786	feat(v1/webhook): complex webhook object w/ headers (#899) ... * feat(v1/webhook): complex webhook object w/ headers * feat(js-sdk/crawl): add complex webhook support	2024-11-13 19:36:44 +01:00
rafaelmmiller	25f32000db	mvp done?	2024-11-13 13:05:29 -03:00
rafaelmmiller	a175c1513a	wip	2024-11-13 08:09:51 -03:00
Nicolas	1a636b4e59	Update email_notification.ts	2024-11-12 20:09:01 -05:00
Gergő Móricz	5ce4aaf0ec	fix(crawl): initialURL setting is unnecessary	2024-11-12 23:35:07 +01:00
Gergő Móricz	93ac20f930	fix(queue-worker): do not kill crawl on one-page error	2024-11-12 22:53:29 +01:00
Gergő Móricz	16e850288c	fix(scrapeURL/pdf,docx): ignore SSL when downloading PDF	2024-11-12 22:46:58 +01:00
rafaelmmiller	807703d94c	wip	2024-11-12 18:44:14 -03:00
Gergő Móricz	7081beff1f	fix(scrapeURL/pdf): retry	2024-11-12 22:26:36 +01:00
Gergő Móricz	9ace2ad071	fix(scrapeURL/pdf): fix llamaparse upload	2024-11-12 20:55:14 +01:00
Gergő Móricz	687ea69621	fix(requests.http): default to localhost baseUrl	2024-11-12 19:59:09 +01:00
Gergő Móricz	3a5eee6e3f	feat: improve requests.http using format features	2024-11-12 19:58:07 +01:00
Gergő Móricz	f2ecf0cc36	fix(v0): crawl timeout errors	2024-11-12 19:46:00 +01:00
Nicolas	464b41a5d2	Merge branch 'main' into nsc/new-extract	2024-11-12 12:24:47 -05:00
Nicolas	a23364e5da	Update extract.ts	2024-11-12 12:23:44 -05:00