1128 Commits

Author	SHA1	Message	Date
Gergő Móricz	29815e084b	feat(v1/Document): add warning field	2024-09-26 21:19:05 +02:00
Gergő Móricz	095babe70b	fix(queue-jobs): jobs with concurrency fails may vanish	2024-09-26 21:18:56 +02:00
Gergő Móricz	b696bfc854	fix(crawl-status): avoid race conditions where crawl may be deemed failed	2024-09-26 21:00:27 +02:00
Gergő Móricz	dec4171937	fix(queue-worker, queue-jobs): logic fixes	2024-09-26 20:39:19 +02:00
Gergő Móricz	d2881927c1	fix(queue-worker): remove concurrency entries when done in sentry-less branch	2024-09-26 20:29:17 +02:00
Gergő Móricz	53fce67ca1	feat(queue-worker): PoC of concurrency limits	2024-09-26 20:24:34 +02:00
Nicolas	30058b1da0	Nick: increased timeout for chrome-cdp due to smart wait	2024-09-26 20:24:34 +02:00
Gergő Móricz	953d4fb197	fix(redlock): use redlock.using for stability	2024-09-25 22:47:42 +02:00
Gergő Móricz	eef116bef8	fix(auth): move redlock settings	2024-09-25 22:27:51 +02:00
Gergő Móricz	2c96d2eef6	fix(auth/redlock): retry cached ACUC lock for 20 seconds	2024-09-25 22:25:13 +02:00
Gergő Móricz	1cca9b8ae6	fix(billTeam): cache update race condition	2024-09-25 22:15:02 +02:00
Gergő Móricz	eb7317c08a	fix(credit_billing): allow spending of exact credits	2024-09-25 21:44:05 +02:00
Gergő Móricz	e67cbc2ca1	fix(billTeam): update cached ACUC after billing	2024-09-25 21:37:01 +02:00
Gergő Móricz	5a8eb17a82	feat(db): implement bill_team RPC	2024-09-25 20:57:45 +02:00
Gergő Móricz	415fd9f333	fix(credit_billing): return chunk.remaining_credits	2024-09-25 20:37:35 +02:00
Gergő Móricz	417adf8e96	fix(db): fix caching and rpc error	2024-09-25 19:42:45 +02:00
Gergő Móricz	331e826bca	feat(db): implement auth_credit_usage_chunk RPC	2024-09-25 19:25:18 +02:00
Nicolas	1da026b26e	Update single_url.ts	2024-09-24 23:29:48 -04:00
Nicolas	b8266cc329	Update website_params.ts	2024-09-24 23:28:58 -04:00
Gergő Móricz	f00c0b82f9	fix(v1/scrape): add total wait specified in request to timeout	2024-09-24 21:56:22 +02:00
Nicolas	3f138e559e	Update website_params.ts	2024-09-24 15:14:26 -04:00
Gergő Móricz	43730b5db6	feat(WebScraper): always report error of last scraper in order	2024-09-24 20:03:49 +02:00
Gergő Móricz	3e661a2087	fix(v1/crawl-cancel): avoid double authing	2024-09-24 20:01:34 +02:00
Gergő Móricz	4194525640	fix(blocklist): unblock TikTok Business page ... This is just a regular business site, not social media.	2024-09-24 16:55:19 +02:00
Gergő Móricz	4a623c084a	fix(fly): don't use Depot builders (doesn't work)	2024-09-24 10:50:30 +02:00
Gergő Móricz	a59b5836d5	Revert error tallying	2024-09-24 10:27:49 +02:00
Gergő Móricz	483f97d21b	fix(v0/search): don't sent scrape fail errors to Sentry	2024-09-23 18:49:27 +02:00
Gergő Móricz	d927cafeea	fix(queue-worker): don't send scraping errors to sentry	2024-09-23 18:48:01 +02:00
Gergő Móricz	677faa27f3	fix(WebScraper): explicitly ignore 404s	2024-09-23 18:47:07 +02:00
Gergő Móricz	83d8287c14	fix(v0, sentry): don't send all scraping methods failed errors to Sentry	2024-09-23 18:40:21 +02:00
Gergő Móricz	d2f7031069	fix(WebScraper): fatal error handler triggering for 404s	2024-09-23 18:33:10 +02:00
Nicolas	dfdbae74c6	Update fireEngine.ts	2024-09-21 21:10:05 -04:00
Nicolas	db161ac55a	Nick: press + write	2024-09-20 19:45:23 -04:00
Nicolas	3fc5ce17d2	Nick: fixed error handling for v0 scrape	2024-09-20 18:35:30 -04:00
Nicolas	0690cfeaad	Merge branch 'main' into feat/actions	2024-09-20 18:24:13 -04:00
Gergő Móricz	e1a34b0a99	Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot" ... This reverts commit 815bfc8f07b3493ed91ff90afda123e7d84f44f9.	2024-09-20 21:43:22 +02:00
Gergő Móricz	815bfc8f07	feat(scrape): scroll down/up with actions if fullpagescreenshot ... revert this if unneeded	2024-09-20 21:42:09 +02:00
Gergő Móricz	d663bbf0ca	feat(actions): add scroll	2024-09-20 21:41:53 +02:00
Gergő Móricz	3dd912ec91	feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor	2024-09-20 21:02:53 +02:00
Gergő Móricz	01f42b980d	feat(scrape): add error tallying instead of empty response	2024-09-19 19:21:13 +02:00
Nicolas	b8b8522b33	Nick: fixed map exception	2024-09-19 12:49:33 -04:00
Nicolas	ef71940429	Merge pull request #674 from mendableai/nsc/map-pagination ... v1/map improvements + higher limits	2024-09-19 11:31:23 -04:00
Nicolas	00bb958b32	Update v1-openapi.json	2024-09-18 15:59:11 -04:00
Nicolas	03eb47d426	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-09-18 15:58:54 -04:00
Nicolas	3a4dd8fc7e	Nick: v1 openapi spec	2024-09-18 15:58:53 -04:00
Gergő Móricz	712ca31615	minor fixes	2024-09-18 21:34:09 +02:00
Gergő Móricz	093c064bff	feat(v1): add public actions api	2024-09-18 20:39:25 +02:00
Gergő Móricz	42d677fe3c	feat(fire-engine): port waitFor and screenshot to use actions	2024-09-18 20:04:54 +02:00
Gergő Móricz	c28e1e2959	fix(v1/zod): formats: add error message if both sc and sc@fP	2024-09-18 19:35:38 +02:00
Nicolas	3c2bfe2da2	Update single_url.ts	2024-09-17 01:58:47 -04:00