1997 Commits

Author	SHA1	Message	Date
Gergő Móricz	a1efe33c8a	fix(scrapeQueue): change expiry to 1 hour	2025-01-23 20:30:20 +01:00
Gergő Móricz	a7b56ab87c	feat(crawl-status): same for v0	2025-01-23 19:39:33 +01:00
Gergő Móricz	95ce3c3b71	feat(crawl-status): allow for jobs to expire out of the redis	2025-01-23 19:33:43 +01:00
Gergő Móricz	6f696d32ae	feat(extract): add log on 0 links	2025-01-23 19:25:12 +01:00
Gergő Móricz	5d56627bfa	feat(extraction-service): highlight req schema generation	2025-01-23 19:24:24 +01:00
Móricz Gergő	9da51a7514	feat(extract): add original schema to logs	2025-01-23 14:59:54 +01:00
Móricz Gergő	561f0186ef	fix build error	2025-01-23 12:07:37 +01:00
Móricz Gergő	6557365149	feat(sitemap): change sitemap logging	2025-01-23 12:06:50 +01:00
Móricz Gergő	d3518e85a8	feat(extract): add logging	2025-01-23 12:05:15 +01:00
Móricz Gergő	434a435a4b	fix(sitemap): increase limit to 20	2025-01-23 11:29:49 +01:00
Móricz Gergő	1e28ba291e	fix(sitemap): increase limit	2025-01-23 09:21:38 +01:00
Móricz Gergő	bee2b2873e	fix(sitemap): better ordering	2025-01-23 08:58:18 +01:00
Móricz Gergő	3761eb17a7	feat(sitemap): reenable fallback to tlsclient	2025-01-23 08:43:13 +01:00
Móricz Gergő	72198123cb	fix(crawler): move sitemap deduplication to deeper in the process	2025-01-23 08:10:46 +01:00
Móricz Gergő	aa2c369060	feat(sitemap): propagate crawlid	2025-01-23 07:19:00 +01:00
Móricz Gergő	a922aac805	fix(crawler): dumb sitemap limit	2025-01-23 07:10:07 +01:00
Móricz Gergő	51a0e233e3	fix(sitemap): temporarily disable tlsclient	2025-01-23 06:56:15 +01:00
Nicolas	d162247703	Update cache.ts	2025-01-23 02:37:04 -03:00
Nicolas	ccb74a2b43	Nick: increased timeouts on extract + reduced extract redis usage	2025-01-23 01:28:26 -03:00
Nicolas	498558d358	Nick: formatting done	2025-01-22 18:47:44 -03:00
Nicolas	994e1eb502	Nick: rm logs	2025-01-22 17:27:48 -03:00
Nicolas	56f048aeff	Reapply "Nick:" ... This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.	2025-01-22 17:26:32 -03:00
Nicolas	4b4385c520	Revert "Nick:" ... This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.	2025-01-22 17:26:09 -03:00
Nicolas	e1ef826ac6	Merge branch 'main' of https://github.com/mendableai/firecrawl	2025-01-22 17:25:49 -03:00
Nicolas	6718ce8908	Nick:	2025-01-22 17:25:48 -03:00
Gergő Móricz	208bd4ca0c	fix(extraction-service): marginally improve logging	2025-01-22 19:38:09 +01:00
Gergő Móricz	ed929221ab	feat(sitemap): switch around engine order	2025-01-22 19:10:27 +01:00
Gergő Móricz	5a039e7b64	fix(v1/map): add wrapper around tryGetSitemap	2025-01-22 19:00:46 +01:00
Nicolas	5aad21b35a	Update extract.ts	2025-01-22 11:01:10 -03:00
Nicolas	04916f17e2	Nick: bug fixes + acuc fixes + cache fixes	2025-01-21 19:17:06 -03:00
Nicolas	3604f2a3ae	Nick: misc improvements	2025-01-21 16:57:45 -03:00
Nicolas	ac0d10c451	Nick: sitemap fetch only below threshold for /map	2025-01-21 16:28:57 -03:00
Nicolas	c7b219169b	Nick: fixed crawl maps index dedup	2025-01-21 16:22:27 -03:00
Nicolas	720a429115	Nick: temp fix	2025-01-21 13:23:34 -03:00
Nicolas	2b9f63cf10	Nick: more permissive re-ranker	2025-01-21 11:30:54 -03:00
Gergő Móricz	dcbe0b319c	fix(v1/crawl-status-ws): wait to send catchup before closing	2025-01-20 20:01:27 +01:00
Nicolas	ef69b1ac88	Nick: allowExternalLinks is now enableWebSearch	2025-01-20 13:41:30 -03:00
Nicolas	5030fea634	Update document-scraper.ts	2025-01-20 13:28:59 -03:00
Móricz Gergő	2d4f4de0ab	fix(credit_billing): logs	2025-01-20 10:16:47 +01:00
Móricz Gergő	ae0d705f5d	fix(v0/crawl): force kickoff	2025-01-20 09:55:00 +01:00
Móricz Gergő	2cf7a4f57a	fix(batch-scrape): auto finish "kickoff" (no kickoff)	2025-01-20 09:40:59 +01:00
Nicolas	f385b250be	Update html-to-markdown.ts	2025-01-20 00:20:20 -03:00
Nicolas	240e4e4702	Update auth.ts	2025-01-19 23:17:12 -03:00
Nicolas	1ca50e6e8f	Update llmExtract.ts	2025-01-19 22:18:51 -03:00
Nicolas	d786949639	Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract" ... This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.	2025-01-19 22:04:12 -03:00
Nicolas	8b17af4001	Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract" ... This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.	2025-01-19 22:00:28 -03:00
Nicolas	406f28c04a	Merge pull request #1068 from mendableai/nsc/llm-usage-extract ... (feat/extract) - LLMs usage analysis + billing	2025-01-19 21:36:33 -03:00
Nicolas	02dea23892	Update auth.ts	2025-01-19 21:35:32 -03:00
Nicolas	34ad9ec25d	Merge pull request #1073 from mendableai/nsc/index-queue ... (feat/index) Index/Insertion queue	2025-01-19 17:45:57 -03:00
Gergő Móricz	6637dce626	fix: status	2025-01-19 17:34:09 +01:00