474 Commits

Author	SHA1	Message	Date
Gergő Móricz	e1a34b0a99	Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" ... This reverts commit 815bfc8f07b3493ed91ff90afda123e7d84f44f9.	2024-09-20 21:43:22 +02:00
Gergő Móricz	815bfc8f07	feat(scrape): scroll down/up with actions if fullpagescreenshot ... revert this if unneeded	2024-09-20 21:42:09 +02:00
Gergő Móricz	3dd912ec91	feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor	2024-09-20 21:02:53 +02:00
Gergő Móricz	01f42b980d	feat(scrape): add error tallying instead of empty response	2024-09-19 19:21:13 +02:00
Nicolas	ef71940429	Merge pull request #674 from mendableai/nsc/map-pagination ... v1/map improvements + higher limits	2024-09-19 11:31:23 -04:00
skeptrune	26825fc8fb	bugfix: using onlyIncludeTags and removeTags together	2024-09-19 00:29:13 -07:00
Gergő Móricz	712ca31615	minor fixes	2024-09-18 21:34:09 +02:00
Gergő Móricz	093c064bff	feat(v1): add public actions api	2024-09-18 20:39:25 +02:00
Gergő Móricz	42d677fe3c	feat(fire-engine): port waitFor and screenshot to use actions	2024-09-18 20:04:54 +02:00
Nicolas	3c2bfe2da2	Update single_url.ts	2024-09-17 01:58:47 -04:00
Nicolas	18b024c238	Update single_url.ts	2024-09-17 01:41:46 -04:00
Nicolas	a4039bd008	Revert "Update single_url.ts" ... This reverts commit 0f8c0a570dca877d14d590e6002eaffd345a3927.	2024-09-16 23:36:38 -04:00
Nicolas	0f8c0a570d	Update single_url.ts	2024-09-16 21:44:56 -04:00
Nicolas	af4804e13b	Merge branch 'main' into nsc/map-pagination	2024-09-16 11:04:44 -04:00
Gergo Moricz	000a316cc3	fix(fire-engine): poll more frequently	2024-09-13 16:41:27 +02:00
Nicolas	17e419a7fb	Nick:	2024-09-09 21:06:23 -03:00
Nicolas	32097fa29d	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-09-07 13:12:52 -03:00
Nicolas	5758af3291	Update website_params.ts	2024-09-07 13:12:46 -03:00
Nicolas	79870e7305	Update excludeTags.ts	2024-09-06 20:15:26 -03:00
Nicolas	f5b84e15e1	Update sitemap.ts	2024-09-05 17:52:27 -03:00
Nicolas	cb630bfc34	Update fireEngine.ts	2024-09-05 14:24:10 -03:00
rafaelsideguide	8c1097e9e1	fix: pageOptions	2024-09-05 14:16:31 -03:00
rafaelsideguide	b301ffc922	added missing variables	2024-09-05 13:57:26 -03:00
Nicolas	08a9cb8db4	Merge branch 'main' into pr/516	2024-09-02 23:32:23 -03:00
Nicolas	6d1da2e22a	Update index.ts	2024-08-30 18:45:02 -03:00
Nicolas	49e1cb7ca0	Nick:	2024-08-29 20:08:06 -03:00
Nicolas	1174bc3c82	Nick: blocked sw/ra	2024-08-29 13:55:54 -03:00
rafaelsideguide	ef2d8d012b	Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper	2024-08-28 14:07:31 -03:00
rafaelsideguide	5cbf0dcaf5	fix(v1): includeTags	2024-08-28 14:07:28 -03:00
Nicolas	9a13034095	Nick:	2024-08-28 13:46:39 -03:00
Nicolas	ecd07be49e	Nick: fixed issues	2024-08-28 13:17:22 -03:00
Nicolas	4d0acc9722	Merge branch 'main' into v1-webscraper	2024-08-26 16:22:05 -03:00
Gergo Moricz	d591e0f51c	block corterix.com for performance issues	2024-08-25 20:06:12 +02:00
Nicolas	173f4ee1bf	Nick: chrome cdp main | simple autoscaler	2024-08-23 20:09:59 -03:00
Gergő Móricz	05c250d3b8	Merge branch 'main' into v1-webscraper	2024-08-23 19:38:57 +02:00
Nicolas	3d53f4e213	Nick: unblocking pin	2024-08-23 13:56:05 -03:00
Gergő Móricz	e7f267b6fe	Merge branch 'main' into v1-webscraper	2024-08-23 17:21:54 +02:00
Gergő Móricz	8d9ff90bcb	feat(fire-engine): propagate sentry trace	2024-08-22 23:38:04 +02:00
Gergő Móricz	8e3c2b2855	fix(crawler): verify URL	2024-08-22 23:30:19 +02:00
rafaelsideguide	7473b74021	fix: html and rawlhtmls for pdfs	2024-08-22 15:15:45 -03:00
rafaelsideguide	b1d61d8557	Merge remote-tracking branch 'origin/v1-webscraper' into v1/python-sdk	2024-08-22 13:39:09 -03:00
Gergő Móricz	6d48dbcd38	feat(sentry): add trace continuity for queue	2024-08-22 16:47:38 +02:00
Gergő Móricz	fbbc3878f1	fix(crawler): make sure includes/excludes is an array	2024-08-22 13:18:26 +02:00
rafaelsideguide	fe2e8c0b7a	includehtml fix	2024-08-21 15:54:00 -03:00
Gergő Móricz	55009e51f5	fix: filter out invalid URLs from crawl links	2024-08-21 20:49:25 +02:00
rafaelsideguide	52abec41c2	fixing delete	2024-08-21 10:35:50 -03:00
rafaelsideguide	b66553867e	reverting delete, fixed express bug on checkCredits	2024-08-21 09:28:20 -03:00
rafaelsideguide	138437d616	commenting out delete, crashing on fire-engine	2024-08-21 08:11:24 -03:00
rafaelsideguide	5e48bec1fd	commenting out delete, crashing on fire-engine	2024-08-21 08:10:46 -03:00
Nicolas	90b32f16c8	Nick: fixes	2024-08-20 21:38:11 -03:00