1997 Commits

Author	SHA1	Message	Date
rafaelsideguide	ae464ada60	tests: teamIds	2024-10-09 15:06:29 -03:00
Nicolas	1cd49a0a95	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-10-09 14:41:25 -03:00
Nicolas	064ce482c2	Update blocklist.ts	2024-10-09 14:41:23 -03:00
rafaelsideguide	4020a7d781	test: added test suite tokens	2024-10-08 15:11:08 -03:00
Harsh Master	aa3d4b8d6c	Fixed Issue #734	2024-10-08 11:36:12 +05:30
Nicolas	5c0c952a27	Update website_params.ts	2024-10-07 14:51:05 -03:00
Nicolas	1f1afeaac4	Update system-monitor.ts	2024-10-04 15:15:04 -03:00
Nicolas	dba96998e3	Update fetch.ts	2024-10-03 18:56:51 -03:00
Nicolas	668ff3c71b	Update fetch.ts	2024-10-03 18:55:39 -03:00
Nicolas	25dd16bf2a	Nick: removed 401	2024-10-03 18:52:17 -03:00
Nicolas	93657f6a44	Update queue-worker.ts	2024-10-03 18:44:40 -03:00
Thomas Kosmas	28b64fc704	Change the gracefull shutdown signal	2024-10-04 00:40:09 +03:00
Nicolas	497ac3328b	Merge pull request #732 from mendableai/fix/url-validation-params ... [BUG] Fixed URLs with params	2024-10-03 17:43:37 -03:00
rafaelsideguide	cfd776a5de	fix: now urls with params are passing validation ... example: https://www.granitecreek.com?asljhda=akjshd	2024-10-03 17:37:04 -03:00
Nicolas	c6a29efbed	Update crawl-status.ts	2024-10-03 17:33:38 -03:00
Nicolas	ddd774ed68	Nick:	2024-10-03 17:20:57 -03:00
Nicolas	82551bb6bc	Update index.test.ts	2024-10-03 17:13:30 -03:00
Nicolas	49bd95327e	Update types.ts	2024-10-03 17:00:33 -03:00
Nicolas	1a1ac9fd60	Nick:	2024-10-03 16:37:58 -03:00
Nicolas	a150aa820c	Nick: shouldnt fallback on a 400 + error code should be correct on page status code	2024-10-03 15:21:42 -03:00
Gergő Móricz	26771e2e71	debug(zod): log unsupported protocol errors	2024-10-01 22:13:28 +02:00
Nicolas	d1b838322d	Merge pull request #721 from mendableai/feat/concurrency-limit ... Concurrency limits	2024-10-01 16:15:05 -03:00
Nicolas	ac5e1fc194	Update sitemap.ts	2024-10-01 16:14:43 -03:00
Nicolas	c6717fecaa	Nick: got rid of job interval sleep and math.min	2024-10-01 16:11:12 -03:00
Nicolas	18f9cd09e1	Nick: fixed more stuff	2024-10-01 16:04:39 -03:00
Gergő Móricz	fe721fffbe	fix(crawl-redis): normalize URL before locking	2024-10-01 20:59:50 +02:00
Nicolas	c0541cc990	Update queue-worker.ts	2024-10-01 15:38:24 -03:00
Nicolas	37299fc035	Update types.ts	2024-10-01 15:18:11 -03:00
Nicolas	8aa07afb6d	Nick: fixes	2024-10-01 15:15:49 -03:00
Nicolas	92dbd33e57	Update queue-worker.ts	2024-10-01 14:53:26 -03:00
Nicolas	4d5477f357	Nick: resolved conflicts	2024-10-01 14:39:57 -03:00
Nicolas	96245e387d	Update crawl.ts	2024-10-01 14:29:53 -03:00
Nicolas	258c67ce67	Revert "feat(queue-worker): always crawl links from content even if sitemapped" ... This reverts commit 3c045c43a446bb7895892338c881cd7bc4f77cbf.	2024-10-01 14:20:23 -03:00
Nicolas	445fc432e9	Reapply "fix(v1/crawl): always use sitemap" ... This reverts commit 339b19ce9d57fd15b11820e1cfbe4d7b5f44cf30.	2024-10-01 14:03:07 -03:00
Nicolas	339b19ce9d	Revert "fix(v1/crawl): always use sitemap" ... This reverts commit 5dc0fcf644bfc64b2b30dd345b2a61b64a4c1262.	2024-10-01 13:59:49 -03:00
Gergő Móricz	5dc0fcf644	fix(v1/crawl): always use sitemap	2024-10-01 18:49:44 +02:00
Gergő Móricz	3c045c43a4	feat(queue-worker): always crawl links from content even if sitemapped	2024-10-01 18:32:53 +02:00
Nicolas	1af26fe1b4	Nick: sitemap fix	2024-10-01 12:38:48 -03:00
Nicolas	ff4b7a835b	Merge pull request #685 from devflowinc/main ... bugfix: using onlyIncludeTags and removeTags together	2024-09-30 17:18:30 -03:00
Nicolas	986262e1d4	Update search.ts	2024-09-30 15:23:43 -03:00
Gergő Móricz	0dd06d33ef	fix(v0/search): pass job priority	2024-09-30 19:20:24 +02:00
Gergő Móricz	20ffdbd15c	hotfix	2024-09-30 19:17:52 +02:00
Gergő Móricz	a8df85fd9b	fix(acuc): remove sentry capture	2024-09-30 19:10:24 +02:00
Gergő Móricz	3621e191bd	feat(concurrency-limit): set limit based on plan	2024-09-28 00:19:54 +02:00
Gergő Móricz	c6a83ab92c	fix(api): entrypoint	2024-09-27 22:16:27 +02:00
Gergő Móricz	e44bdf7a54	bad dockerfile	2024-09-27 21:07:11 +02:00
Gergő Móricz	f0a1a2e45b	fix: increase ulimit -n in docker	2024-09-27 20:44:52 +02:00
Gergő Móricz	d5e2a80e4a	fix(crawl-status): keep 10 megabyte pages if they're the only thing in the output	2024-09-27 20:41:41 +02:00
Nicolas	975f0575b4	Nick: max retries with axios-retry	2024-09-27 12:58:57 -04:00
Nicolas	92961cf74f	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-09-27 12:23:45 -04:00