144 Commits

Author	SHA1	Message	Date
Nicolas	446acfccde	Nick: support for the new actions	2024-10-31 20:01:52 -03:00
Nicolas	fa8875d64d	Update single_url.ts	2024-10-28 15:09:50 -03:00
Thomas Kosmas	bd55464b52	skipTlsVerification	2024-10-22 22:28:02 +03:00
Nicolas	b4f6a0f919	Nick: geolocation	2024-10-15 21:12:33 -03:00
Nicolas	25dd16bf2a	Nick: removed 401	2024-10-03 18:52:17 -03:00
Nicolas	ddd774ed68	Nick:	2024-10-03 17:20:57 -03:00
Nicolas	a150aa820c	Nick: shouldnt fallback on a 400 + error code should be correct on page status code	2024-10-03 15:21:42 -03:00
Nicolas	975f0575b4	Nick: max retries with axios-retry	2024-09-27 12:58:57 -04:00
Nicolas	1fdff87b3e	Update single_url.ts	2024-09-27 12:23:44 -04:00
Nicolas	1da026b26e	Update single_url.ts	2024-09-24 23:29:48 -04:00
Gergő Móricz	43730b5db6	feat(WebScraper): always report error of last scraper in order	2024-09-24 20:03:49 +02:00
Gergő Móricz	a59b5836d5	Revert error tallying	2024-09-24 10:27:49 +02:00
Gergő Móricz	677faa27f3	fix(WebScraper): explicitly ignore 404s	2024-09-23 18:47:07 +02:00
Gergő Móricz	d2f7031069	fix(WebScraper): fatal error handler triggering for 404s	2024-09-23 18:33:10 +02:00
Gergő Móricz	e1a34b0a99	Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" ... This reverts commit 815bfc8f07b3493ed91ff90afda123e7d84f44f9.	2024-09-20 21:43:22 +02:00
Gergő Móricz	815bfc8f07	feat(scrape): scroll down/up with actions if fullpagescreenshot ... revert this if unneeded	2024-09-20 21:42:09 +02:00
Gergő Móricz	3dd912ec91	feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor	2024-09-20 21:02:53 +02:00
Gergő Móricz	01f42b980d	feat(scrape): add error tallying instead of empty response	2024-09-19 19:21:13 +02:00
Gergő Móricz	712ca31615	minor fixes	2024-09-18 21:34:09 +02:00
Gergő Móricz	093c064bff	feat(v1): add public actions api	2024-09-18 20:39:25 +02:00
Gergő Móricz	42d677fe3c	feat(fire-engine): port waitFor and screenshot to use actions	2024-09-18 20:04:54 +02:00
Nicolas	3c2bfe2da2	Update single_url.ts	2024-09-17 01:58:47 -04:00
Nicolas	18b024c238	Update single_url.ts	2024-09-17 01:41:46 -04:00
Nicolas	a4039bd008	Revert "Update single_url.ts" ... This reverts commit 0f8c0a570dca877d14d590e6002eaffd345a3927.	2024-09-16 23:36:38 -04:00
Nicolas	0f8c0a570d	Update single_url.ts	2024-09-16 21:44:56 -04:00
Nicolas	17e419a7fb	Nick:	2024-09-09 21:06:23 -03:00
rafaelsideguide	8c1097e9e1	fix: pageOptions	2024-09-05 14:16:31 -03:00
rafaelsideguide	b301ffc922	added missing variables	2024-09-05 13:57:26 -03:00
Nicolas	08a9cb8db4	Merge branch 'main' into pr/516	2024-09-02 23:32:23 -03:00
Nicolas	49e1cb7ca0	Nick:	2024-08-29 20:08:06 -03:00
rafaelsideguide	ef2d8d012b	Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper	2024-08-28 14:07:31 -03:00
rafaelsideguide	5cbf0dcaf5	fix(v1): includeTags	2024-08-28 14:07:28 -03:00
Nicolas	ecd07be49e	Nick: fixed issues	2024-08-28 13:17:22 -03:00
Nicolas	4d0acc9722	Merge branch 'main' into v1-webscraper	2024-08-26 16:22:05 -03:00
Nicolas	173f4ee1bf	Nick: chrome cdp main | simple autoscaler	2024-08-23 20:09:59 -03:00
Gergő Móricz	e7f267b6fe	Merge branch 'main' into v1-webscraper	2024-08-23 17:21:54 +02:00
rafaelsideguide	7473b74021	fix: html and rawlhtmls for pdfs	2024-08-22 15:15:45 -03:00
rafaelsideguide	fe2e8c0b7a	includehtml fix	2024-08-21 15:54:00 -03:00
Gergő Móricz	1368f9a87f	fix: treat existing screenshot as a scraper success condition	2024-08-20 22:24:18 +02:00
rafaelsideguide	ecd472356b	added variables to beta customers	2024-08-19 16:41:54 -03:00
rafaelsideguide	7a61325500	map + search + scrape markdown bug	2024-08-16 17:57:11 -03:00
rafaelsideguide	3f998b688d	scrape ready	2024-08-16 15:14:37 -03:00
Gergő Móricz	29f0d9ec94	propagate priority to fire-engine	2024-08-15 19:04:46 +02:00
Rafael Miller	76160a38db	Update single_url.ts	2024-08-12 17:57:00 -03:00
Rafael Miller	7c339ea125	Update single_url.ts	2024-08-12 17:55:10 -03:00
rafaelsideguide	c3aeed510b	Update single_url.ts	2024-08-12 16:40:31 -03:00
Kevin Swiber	ba2af74adf	Ensuring USE_DB_AUTHENTICATION is true in single URL scraper.	2024-08-09 15:29:18 -07:00
Gergő Móricz	5fc7fcb77c	Merge branch 'main' into feat/queue-scrapes	2024-08-07 16:35:44 +02:00
Gergo Moricz	b60ee30dba	fix(single_url): accept 500	2024-08-06 18:00:56 +02:00
rafaelsideguide	4d24a99d50	fix params	2024-08-06 09:34:43 -03:00