362 Commits

Author	SHA1	Message	Date
Nicolas	18b024c238	Update single_url.ts	2024-09-17 01:41:46 -04:00
Nicolas	a4039bd008	Revert "Update single_url.ts" ... This reverts commit 0f8c0a570dca877d14d590e6002eaffd345a3927.	2024-09-16 23:36:38 -04:00
Nicolas	0f8c0a570d	Update single_url.ts	2024-09-16 21:44:56 -04:00
Gergo Moricz	000a316cc3	fix(fire-engine): poll more frequently	2024-09-13 16:41:27 +02:00
Nicolas	17e419a7fb	Nick:	2024-09-09 21:06:23 -03:00
Nicolas	32097fa29d	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-09-07 13:12:52 -03:00
Nicolas	5758af3291	Update website_params.ts	2024-09-07 13:12:46 -03:00
Nicolas	79870e7305	Update excludeTags.ts	2024-09-06 20:15:26 -03:00
Nicolas	cb630bfc34	Update fireEngine.ts	2024-09-05 14:24:10 -03:00
rafaelsideguide	8c1097e9e1	fix: pageOptions	2024-09-05 14:16:31 -03:00
rafaelsideguide	b301ffc922	added missing variables	2024-09-05 13:57:26 -03:00
Nicolas	08a9cb8db4	Merge branch 'main' into pr/516	2024-09-02 23:32:23 -03:00
Nicolas	6d1da2e22a	Update index.ts	2024-08-30 18:45:02 -03:00
Nicolas	49e1cb7ca0	Nick:	2024-08-29 20:08:06 -03:00
Nicolas	1174bc3c82	Nick: blocked sw/ra	2024-08-29 13:55:54 -03:00
rafaelsideguide	ef2d8d012b	Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper	2024-08-28 14:07:31 -03:00
rafaelsideguide	5cbf0dcaf5	fix(v1): includeTags	2024-08-28 14:07:28 -03:00
Nicolas	9a13034095	Nick:	2024-08-28 13:46:39 -03:00
Nicolas	ecd07be49e	Nick: fixed issues	2024-08-28 13:17:22 -03:00
Nicolas	4d0acc9722	Merge branch 'main' into v1-webscraper	2024-08-26 16:22:05 -03:00
Gergo Moricz	d591e0f51c	block corterix.com for performance issues	2024-08-25 20:06:12 +02:00
Nicolas	173f4ee1bf	Nick: chrome cdp main | simple autoscaler	2024-08-23 20:09:59 -03:00
Gergő Móricz	05c250d3b8	Merge branch 'main' into v1-webscraper	2024-08-23 19:38:57 +02:00
Nicolas	3d53f4e213	Nick: unblocking pin	2024-08-23 13:56:05 -03:00
Gergő Móricz	e7f267b6fe	Merge branch 'main' into v1-webscraper	2024-08-23 17:21:54 +02:00
Gergő Móricz	8d9ff90bcb	feat(fire-engine): propagate sentry trace	2024-08-22 23:38:04 +02:00
Gergő Móricz	8e3c2b2855	fix(crawler): verify URL	2024-08-22 23:30:19 +02:00
rafaelsideguide	7473b74021	fix: html and rawlhtmls for pdfs	2024-08-22 15:15:45 -03:00
rafaelsideguide	b1d61d8557	Merge remote-tracking branch 'origin/v1-webscraper' into v1/python-sdk	2024-08-22 13:39:09 -03:00
Gergő Móricz	6d48dbcd38	feat(sentry): add trace continuity for queue	2024-08-22 16:47:38 +02:00
Gergő Móricz	fbbc3878f1	fix(crawler): make sure includes/excludes is an array	2024-08-22 13:18:26 +02:00
rafaelsideguide	fe2e8c0b7a	includehtml fix	2024-08-21 15:54:00 -03:00
Gergő Móricz	55009e51f5	fix: filter out invalid URLs from crawl links	2024-08-21 20:49:25 +02:00
rafaelsideguide	52abec41c2	fixing delete	2024-08-21 10:35:50 -03:00
rafaelsideguide	b66553867e	reverting delete, fixed express bug on checkCredits	2024-08-21 09:28:20 -03:00
rafaelsideguide	138437d616	commenting out delete, crashing on fire-engine	2024-08-21 08:11:24 -03:00
rafaelsideguide	5e48bec1fd	commenting out delete, crashing on fire-engine	2024-08-21 08:10:46 -03:00
Nicolas	90b32f16c8	Nick: fixes	2024-08-20 21:38:11 -03:00
Nicolas	819ad50af3	Update fireEngine.ts	2024-08-20 21:16:33 -03:00
rafaelsideguide	e9d6ca197e	tests passing now	2024-08-20 20:00:41 -03:00
Nicolas	1b3ad60a2c	Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit ffe11a5bf73e3c57657972cd36c3af1d0b9a432c.	2024-08-20 19:22:09 -03:00
Nicolas	441628998f	Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit ffe11a5bf73e3c57657972cd36c3af1d0b9a432c.	2024-08-20 19:16:48 -03:00
Nicolas	ffe11a5bf7	Revert "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit 2030ec603109d6ce8786a011d431bc5c83917f1b, reversing changes made to f494d2b707d40b690ae41611d17f77f683570fc2.	2024-08-20 18:16:11 -03:00
Gergő Móricz	1368f9a87f	fix: treat existing screenshot as a scraper success condition	2024-08-20 22:24:18 +02:00
rafaelsideguide	f98be7d94e	Update fireEngine.ts	2024-08-20 16:53:01 -03:00
rafaelsideguide	1f27182a13	added try catch	2024-08-20 15:42:39 -03:00
rafaelsideguide	e326249a57	added check job and cancel to fire-engine requests	2024-08-20 14:26:42 -03:00
rafaelsideguide	e1c9cbf709	bug fixed. crawl should not stop if sitemap url is invalid	2024-08-20 09:11:58 -03:00
rafaelsideguide	ecd472356b	added variables to beta customers	2024-08-19 16:41:54 -03:00
rafaelsideguide	b8170aaa47	Update blocklist.ts	2024-08-19 08:51:48 -03:00