3537 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nicolas	98894641c1	Update package.json	2024-11-20 12:27:56 -08:00
Nicolas	7b610354d9	Merge pull request #914 from ad-angelo/node-mobile-support ... Node SDK : Add Mobile Scraping	2024-11-20 12:26:57 -08:00
Nicolas	c873ee4680	Update index.ts	2024-11-20 12:26:04 -08:00
Nicolas	28696da6b2	Nick: gpt-4o	2024-11-20 12:25:50 -08:00
ad-angelo	4248c68f5a	Add Mobile Scraping ... https://www.firecrawl.dev/blog/launch-week-ii-day-6-introducing-mobile-scraping	2024-11-20 21:14:49 +01:00
Nicolas	d49f62fb56	Nick: extract fixes	2024-11-20 11:50:14 -08:00
Gergő Móricz	b1eaecfdb0	fix 2	2024-11-20 20:19:16 +01:00
Gergő Móricz	e2ddc6c65c	fix handling of badly formatted URLs	2024-11-20 20:18:40 +01:00
Gergő Móricz	ba6f29cdda	crawl fix, again	2024-11-20 19:55:35 +01:00
Gergő Móricz	b468bb4014	crawl fixes	2024-11-20 19:48:01 +01:00
Nicolas	c9b0a80522	Nick:	2024-11-20 10:23:44 -08:00
Nicolas	103c3f28e6	Update rate-limiter.ts	2024-11-19 17:51:31 -08:00
Nicolas	d02a8bcb82	Nick: extract urls to extract	2024-11-19 13:49:23 -08:00
Eric Ciarla	aa01c0b684	Create mastering-the-crawl-endpoint.ipynb	2024-11-19 12:50:31 -05:00
Gergő Móricz	79a75e088a	feat(crawl): allowSubdomain	2024-11-19 18:38:59 +01:00
rafaelmmiller	2fb8a3c8dc	fix schema	2024-11-19 10:04:42 -03:00
rafaelmmiller	53134b7c85	Rafa: removed throw error and added map to requests	2024-11-19 09:34:52 -03:00
rafaelmmiller	36cf49c959	Merge remote-tracking branch 'origin/main' into nsc/new-extract	2024-11-19 09:34:08 -03:00
Nicolas	91caa01c5e	Update CONTRIBUTING.md	2024-11-18 16:13:24 -08:00
Nicolas	1328ae0fa3	Update README.md	2024-11-18 15:49:19 -08:00
Eric Ciarla	a31336752c	Create README.md	2024-11-18 14:04:29 -05:00
rafaelmmiller	77e152cba8	added team_id to scrape-status endpoint	2024-11-18 15:02:00 -03:00
Gergő Móricz	31a0471bfa	fix(crawl-redis): ordered push to wrong side of list	2024-11-15 21:56:15 +01:00
Gergő Móricz	1a0f13c0eb	fix(webhook): add logging	2024-11-15 21:43:02 +01:00
Gergő Móricz	1b032b05fa	fix(map): make sitemapOnly simpler	2024-11-15 21:14:32 +01:00
Gergő Móricz	a4d3dba865	fix(map): ignore limit when using sitemapOnly	2024-11-15 21:03:20 +01:00
Gergő Móricz	63787bc504	fix(scrapeURL/fire-engine): wait longer if timeout is not specified	2024-11-15 20:25:16 +01:00
Gergő Móricz	4cddcd5206	fix(scrapeURL/fire-engine): timeout-less scrape support (initial)	2024-11-15 20:15:25 +01:00
Gergő Móricz	350d00d27a	fix(crawler): treat XML files as sitemaps (temporarily)	2024-11-15 20:09:20 +01:00
Gergő Móricz	ca2e33db0a	fix(log_job): add force option to retry on supabase failure	2024-11-15 19:55:23 +01:00
Gergő Móricz	7b02c45dd0	fix(v1/types): better timeout primitives	2024-11-15 19:35:54 +01:00
Gergő Móricz	c95a4a26c9	fix(v1/batch/scrape): raise default timeout	2024-11-15 18:58:03 +01:00
Móricz Gergő	3a342bfbf0	fix(scrapeURL/playwright): JSON body fix	2024-11-15 15:18:40 +01:00
Nicolas	3c1b1909f8	Update map.ts	2024-11-14 17:52:15 -05:00
Nicolas	9519897102	Merge branch 'nsc/sitemap-only'	2024-11-14 17:44:39 -05:00
Nicolas	7f084c6c43	Nick:	2024-11-14 17:44:32 -05:00
Nicolas	e8bd089c8a	Merge pull request #901 from mendableai/nsc/sitemap-only ... Allows `/map` to only return links present in the sitemap	2024-11-14 17:32:37 -05:00
Nicolas	3fcdf57d2f	Update fireEngine.ts	2024-11-14 17:31:30 -05:00
Nicolas	d62f12c9d9	Nick: moved away from axios	2024-11-14 17:31:23 -05:00
Nicolas	f155449458	Nick: sitemap only	2024-11-14 17:29:53 -05:00
Móricz Gergő	431e64e752	fix(batch/scrape/webhook): add batch_scrape.started	2024-11-14 22:40:03 +01:00
Nicolas	7bca4486b4	Update package.json	2024-11-14 16:37:53 -05:00
Móricz Gergő	df05124ef5	feat(v1/batch/scrape): webhooks	2024-11-14 22:36:28 +01:00
Nicolas	91f4fd815f	Update extract.ts	2024-11-14 15:41:42 -05:00
Nicolas	22848af5ae	Nick:	2024-11-14 15:34:02 -05:00
Nicolas	ebe9de2ac5	Nick:	2024-11-14 15:26:15 -05:00
Nicolas	5056dcd8e9	Update index.test.ts	2024-11-14 15:06:22 -05:00
rafaelmmiller	be5e6da5ff	tests	2024-11-14 17:03:54 -03:00
rafaelmmiller	45091430ab	Merge branch 'nsc/new-extract' of https://github.com/mendableai/firecrawl into nsc/new-extract	2024-11-14 17:03:15 -03:00
Nicolas	796cd0746d	Update extract.ts	2024-11-14 15:03:06 -05:00