2271 Commits

Author	SHA1	Message	Date
Gergo Moricz	7c5cda7b45	fix(queue-worker): concurrency	2024-08-06 16:57:00 +02:00
Gergo Moricz	d7d63790e5	fix(crawl-status): isCancelled should be status failed	2024-08-06 16:35:55 +02:00
Gergo Moricz	03c84a9372	cleanup and fix cancelling	2024-08-06 16:26:46 +02:00
rafaelsideguide	4d24a99d50	fix params	2024-08-06 09:34:43 -03:00
Nicolas	e195ddbef4	Merge branch 'main' into nsc/hyper-v81	2024-08-05 20:47:39 -04:00
rafaelsideguide	3edc3a3d15	added fullpagescreenshot capabilities, wip on fire-engine side	2024-08-05 18:17:37 -03:00
rafaelsideguide	f32e8de156	fixes the empty excludes.filter undefined bug	2024-08-05 18:13:31 -03:00
tak-s	af9bc5c8bb	Suppressed repetitive logs	2024-08-04 15:09:36 +09:00
Nicolas	1742e4ceae	Nick:	2024-08-02 19:25:15 -04:00
Nicolas	39aecd974b	Update redis-health.ts	2024-08-02 17:43:45 -04:00
Nicolas	b448e3c3ad	Update website_params.ts	2024-08-02 14:26:35 -04:00
rafaelsideguide	4051630632	Update sitemap.ts	2024-08-02 11:32:48 -03:00
rafaelsideguide	8568b61015	bugfix for sitemaps	2024-08-02 11:03:01 -03:00
Nicolas	af68b7a785	Merge pull request #475 from mendableai/bugfix/issue-466 ... [Bug] pdfs and logging pdf events, also added trycatchs for docx	2024-08-01 22:05:26 -04:00
rafaelsideguide	f48ff36b32	added .inc files and forced lower case comparison	2024-07-31 09:28:43 -03:00
Nicolas	ad6f6eff4b	Update fireEngine.ts	2024-07-30 19:15:54 -04:00
Nicolas	f9827b2151	Update credit_billing.ts	2024-07-30 19:13:17 -04:00
Nicolas	6d99dedd3c	Nick: fixed tests	2024-07-30 19:11:01 -04:00
Nicolas	a28ecc1f61	Nick: caching	2024-07-30 18:59:35 -04:00
Nicolas	52198f2991	Nick:	2024-07-30 16:15:08 -04:00
Nicolas	f43d5e7895	Nick: scrape queue	2024-07-30 14:44:13 -04:00
Nicolas	7e002a8b06	Nick: bull mq	2024-07-30 13:27:23 -04:00
Nicolas	46bcbd931f	Merge branch 'main' into feat/queue-scrapes	2024-07-30 12:44:07 -04:00
Nicolas	fd2452ec9c	Update scrape.ts	2024-07-30 12:42:12 -04:00
rafaelsideguide	8f5174ffc7	Update auth.ts	2024-07-30 10:37:33 -03:00
rafaelsideguide	d25d7e7244	special case: developer.apple.com	2024-07-30 10:13:09 -03:00
Nicolas	c446942306	Nick:	2024-07-29 21:28:29 -04:00
Nicolas	5e8ffcf505	Update website_params.ts	2024-07-29 20:43:47 -04:00
Nicolas	7b813883ef	Nick: first layer	2024-07-29 20:31:51 -04:00
Nicolas	e99c2568f4	Update auth.ts	2024-07-29 18:44:18 -04:00
Nicolas	968a2dc753	Nick:	2024-07-29 18:37:09 -04:00
Nicolas	04942bb9de	Nick:	2024-07-29 18:31:43 -04:00
Nicolas	267d4681bf	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-07-29 17:21:15 -04:00
Nicolas	b4833c1694	Nick: increasing default timeout to 45s	2024-07-29 17:21:11 -04:00
Nicolas	7fa08100bf	Merge pull request #414 from NiuBlibing/support_model_name ... support custom models	2024-07-29 13:21:29 -04:00
rafaelsideguide	49e3e64787	bugfix for pdfs and logging pdf events, also added trycatchs for docx	2024-07-29 14:13:46 -03:00
Nicolas	4c9d62f6d3	Nick: fixing sitemap fallback	2024-07-26 18:25:44 -04:00
Nicolas	091924a636	Nick: moving machines from mia to virginia	2024-07-26 17:37:46 -04:00
Nicolas	cb97871ff9	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-07-26 17:21:11 -04:00
Nicolas	ff4266f09e	Update pdfProcessor.ts	2024-07-26 17:21:09 -04:00
Nicolas	0c2e3a72cc	Merge pull request #460 from mendableai/nsc/admin-router ... Admin router + Improve redis notifications	2024-07-26 12:16:14 -04:00
rafaelsideguide	96cec2a673	fix checking scrape log success content length	2024-07-26 12:00:52 -03:00
Nicolas	542270f4c2	Merge pull request #461 from mendableai/nsc/small-handle-for-client-side-errors ... Client side error handling	2024-07-25 20:53:10 -04:00
Nicolas	dc6f825270	Update email_notification.ts	2024-07-25 20:43:50 -04:00
Nicolas	f82ca3be17	Nick:	2024-07-25 19:53:29 -04:00
Nicolas	01fab6e036	Update single_url.ts	2024-07-25 17:51:41 -04:00
Nicolas	56042d090c	Update single_url.ts	2024-07-25 17:48:44 -04:00
Nicolas	88f5efce8f	Merge branch 'feat/scrape-monitoring'	2024-07-25 17:44:21 -04:00
Nicolas	3242872503	Update single_url.ts	2024-07-25 17:43:55 -04:00
Nicolas	ffd430f198	Merge pull request #457 from JakobStadlhuber/Readiness-Liveness-Probes ... Readiness liveness probes	2024-07-25 17:20:31 -04:00