1997 Commits

Author	SHA1	Message	Date
Nicolas	0ea0a5db46	Nick: wip	2024-08-21 20:54:39 -03:00
rafaelsideguide	fe2e8c0b7a	includehtml fix	2024-08-21 15:54:00 -03:00
Gergő Móricz	629da74a5c	fix(sentry): decrease tracesSampleRate	2024-08-21 20:51:35 +02:00
Gergő Móricz	55009e51f5	fix: filter out invalid URLs from crawl links	2024-08-21 20:49:25 +02:00
Gergő Móricz	dae1408e66	fix(Dockerfile): retain sentry auth token properly	2024-08-21 20:40:42 +02:00
Gergő Móricz	ac9783ed2f	fix(sentry): adjust profiles sample rate to be even lower	2024-08-21 20:21:16 +02:00
Gergő Móricz	9579f03c4b	fix: import resolution	2024-08-21 20:16:06 +02:00
Gergő Móricz	6104d74213	fix(sentry): drop profiling sample rate	2024-08-21 20:12:47 +02:00
Gergő Móricz	3d5dc9d90a	feat(sentry): add log + server name	2024-08-21 19:39:10 +02:00
Gergő Móricz	85ff0c311e	Add worker ID to job attribute	2024-08-21 19:21:29 +02:00
Gergő Móricz	920702cdde	Update builder to handle uploading sourcemaps	2024-08-21 19:08:03 +02:00
Gergő Móricz	86942728e3	Add metadata for queue-worker and Express	2024-08-21 17:58:27 +02:00
Nicolas	35decb1af2	Nick:	2024-08-21 12:35:03 -03:00
rafaelsideguide	52abec41c2	fixing delete	2024-08-21 10:35:50 -03:00
Nicolas	db8c84ff0f	Update requests.http	2024-08-21 10:19:37 -03:00
rafaelsideguide	b66553867e	reverting delete, fixed express bug on checkCredits	2024-08-21 09:28:20 -03:00
rafaelsideguide	138437d616	commenting out delete, crashing on fire-engine	2024-08-21 08:11:24 -03:00
rafaelsideguide	5e48bec1fd	commenting out delete, crashing on fire-engine	2024-08-21 08:10:46 -03:00
Nicolas	90b32f16c8	Nick: fixes	2024-08-20 21:38:11 -03:00
Nicolas	819ad50af3	Update fireEngine.ts	2024-08-20 21:16:33 -03:00
rafaelsideguide	e9d6ca197e	tests passing now	2024-08-20 20:00:41 -03:00
Nicolas	1b3ad60a2c	Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit ffe11a5bf73e3c57657972cd36c3af1d0b9a432c.	2024-08-20 19:22:09 -03:00
Nicolas	441628998f	Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit ffe11a5bf73e3c57657972cd36c3af1d0b9a432c.	2024-08-20 19:16:48 -03:00
Nicolas	ffe11a5bf7	Revert "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" ... This reverts commit 2030ec603109d6ce8786a011d431bc5c83917f1b, reversing changes made to f494d2b707d40b690ae41611d17f77f683570fc2.	2024-08-20 18:16:11 -03:00
Gergő Móricz	5818236659	fix: remove rawHtml properly	2024-08-20 22:51:12 +02:00
Gergő Móricz	1368f9a87f	fix: treat existing screenshot as a scraper success condition	2024-08-20 22:24:18 +02:00
rafaelsideguide	f98be7d94e	Update fireEngine.ts	2024-08-20 16:53:01 -03:00
Nicolas	0c48c8a436	Nick: billing for map	2024-08-20 16:43:46 -03:00
Gergő Móricz	f494d2b707	prioritize search lower	2024-08-20 21:18:31 +02:00
Nicolas	59eb552df5	Merge pull request #560 from mendableai/feat/cancel-fire-engine ... [Feat] Added check job and cancel to fire-engine requests	2024-08-20 15:43:31 -03:00
rafaelsideguide	1f27182a13	added try catch	2024-08-20 15:42:39 -03:00
Gergő Móricz	efb91f9ca5	fix(search): delete jobs after done	2024-08-20 20:29:08 +02:00
Nicolas	39388cdc35	Update crawl.ts	2024-08-20 14:41:43 -03:00
Nicolas	674adee144	Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper	2024-08-20 14:41:05 -03:00
Nicolas	b36faeaf54	Nick:	2024-08-20 14:39:52 -03:00
Gergő Móricz	cf32893c2e	add strict enforcement + move crawlerOptions to top-level in /crawl	2024-08-20 19:31:26 +02:00
rafaelsideguide	e326249a57	added check job and cancel to fire-engine requests	2024-08-20 14:26:42 -03:00
Gergő Móricz	70d50b3640	fix(queue-worker): move dotenv config up	2024-08-20 19:25:19 +02:00
Nicolas	c5ad4dedeb	Update crawl.ts	2024-08-20 14:19:20 -03:00
Nicolas	de0dc20a02	Update credit_billing.ts	2024-08-20 14:18:14 -03:00
Nicolas	5abd26a267	Nick: set the crawl limit to the remaining credits	2024-08-20 14:16:54 -03:00
Nicolas	e200ec9e12	Nick:	2024-08-20 12:24:14 -03:00
Nicolas	55dad82df1	Nick: fixed map search	2024-08-20 12:17:53 -03:00
Nicolas	27903247b6	Nick: map tests and fixes	2024-08-20 12:04:08 -03:00
rafaelsideguide	e1c9cbf709	bug fixed. crawl should not stop if sitemap url is invalid	2024-08-20 09:11:58 -03:00
rafaelsideguide	ecd472356b	added variables to beta customers	2024-08-19 16:41:54 -03:00
Nicolas	3dc298be54	Nick: 2x rate limits for standard and growth for /scrape	2024-08-19 13:52:54 -03:00
rafaelsideguide	72461ce9a6	Update index.test.ts	2024-08-19 13:29:52 -03:00
rafaelsideguide	fd7fdc1d52	added blocklist middleware	2024-08-19 13:28:54 -03:00
Nicolas	8e4ca86463	Update crawl.ts	2024-08-19 11:02:24 -03:00