374 Commits

Author	SHA1	Message	Date
busaud	c6ebbc6f6a	bugfix: self-host crawling doesnt respect limit	2024-10-09 22:52:49 +00:00
Nicolas	497ac3328b	Merge pull request #732 from mendableai/fix/url-validation-params ... [BUG] Fixed URLs with params	2024-10-03 17:43:37 -03:00
rafaelsideguide	cfd776a5de	fix: now urls with params are passing validation ... example: https://www.granitecreek.com?asljhda=akjshd	2024-10-03 17:37:04 -03:00
Nicolas	c6a29efbed	Update crawl-status.ts	2024-10-03 17:33:38 -03:00
Nicolas	49bd95327e	Update types.ts	2024-10-03 17:00:33 -03:00
Nicolas	1a1ac9fd60	Nick:	2024-10-03 16:37:58 -03:00
Nicolas	c6717fecaa	Nick: got rid of job interval sleep and math.min	2024-10-01 16:11:12 -03:00
Nicolas	18f9cd09e1	Nick: fixed more stuff	2024-10-01 16:04:39 -03:00
Nicolas	37299fc035	Update types.ts	2024-10-01 15:18:11 -03:00
Nicolas	8aa07afb6d	Nick: fixes	2024-10-01 15:15:49 -03:00
Nicolas	4d5477f357	Nick: resolved conflicts	2024-10-01 14:39:57 -03:00
Nicolas	96245e387d	Update crawl.ts	2024-10-01 14:29:53 -03:00
Nicolas	445fc432e9	Reapply "fix(v1/crawl): always use sitemap" ... This reverts commit 339b19ce9d57fd15b11820e1cfbe4d7b5f44cf30.	2024-10-01 14:03:07 -03:00
Nicolas	339b19ce9d	Revert "fix(v1/crawl): always use sitemap" ... This reverts commit 5dc0fcf644bfc64b2b30dd345b2a61b64a4c1262.	2024-10-01 13:59:49 -03:00
Gergő Móricz	5dc0fcf644	fix(v1/crawl): always use sitemap	2024-10-01 18:49:44 +02:00
Nicolas	1af26fe1b4	Nick: sitemap fix	2024-10-01 12:38:48 -03:00
Nicolas	986262e1d4	Update search.ts	2024-09-30 15:23:43 -03:00
Gergő Móricz	0dd06d33ef	fix(v0/search): pass job priority	2024-09-30 19:20:24 +02:00
Gergő Móricz	a8df85fd9b	fix(acuc): remove sentry capture	2024-09-30 19:10:24 +02:00
Gergő Móricz	3621e191bd	feat(concurrency-limit): set limit based on plan	2024-09-28 00:19:54 +02:00
Gergő Móricz	d5e2a80e4a	fix(crawl-status): keep 10 megabyte pages if they're the only thing in the output	2024-09-27 20:41:41 +02:00
Gergő Móricz	5e8ef4954e	feat(auth): log cache key in acuc update error	2024-09-27 17:13:10 +02:00
Gergő Móricz	e98f858eb6	fix(api): playground scrape errors	2024-09-26 22:28:14 +02:00
Gergő Móricz	2cb493321a	fix(ACUC): do not refresh cache every set	2024-09-26 22:15:15 +02:00
Gergő Móricz	9bdd344b36	fix(redlock): use redlock.using for stability	2024-09-26 22:15:15 +02:00
Gergő Móricz	250c3bb5c6	fix(auth): move redlock settings	2024-09-26 22:15:15 +02:00
Gergő Móricz	81245e68fa	fix(auth/redlock): retry cached ACUC lock for 20 seconds	2024-09-26 22:15:15 +02:00
Gergő Móricz	0f89f5e7cb	fix(billTeam): cache update race condition	2024-09-26 22:15:15 +02:00
Gergő Móricz	84bff8add8	fix(billTeam): update cached ACUC after billing	2024-09-26 22:15:15 +02:00
Gergő Móricz	f22ab5ffaf	feat(db): implement bill_team RPC	2024-09-26 22:15:15 +02:00
Gergő Móricz	2073063fb7	fix(db): fix caching and rpc error	2024-09-26 22:15:15 +02:00
Gergő Móricz	f8c70fe5dd	feat(db): implement auth_credit_usage_chunk RPC	2024-09-26 22:15:15 +02:00
Gergő Móricz	29815e084b	feat(v1/Document): add warning field	2024-09-26 21:19:05 +02:00
Gergő Móricz	b696bfc854	fix(crawl-status): avoid race conditions where crawl may be deemed failed	2024-09-26 21:00:27 +02:00
Gergő Móricz	953d4fb197	fix(redlock): use redlock.using for stability	2024-09-25 22:47:42 +02:00
Gergő Móricz	eef116bef8	fix(auth): move redlock settings	2024-09-25 22:27:51 +02:00
Gergő Móricz	2c96d2eef6	fix(auth/redlock): retry cached ACUC lock for 20 seconds	2024-09-25 22:25:13 +02:00
Gergő Móricz	1cca9b8ae6	fix(billTeam): cache update race condition	2024-09-25 22:15:02 +02:00
Gergő Móricz	e67cbc2ca1	fix(billTeam): update cached ACUC after billing	2024-09-25 21:37:01 +02:00
Gergő Móricz	5a8eb17a82	feat(db): implement bill_team RPC	2024-09-25 20:57:45 +02:00
Gergő Móricz	417adf8e96	fix(db): fix caching and rpc error	2024-09-25 19:42:45 +02:00
Gergő Móricz	331e826bca	feat(db): implement auth_credit_usage_chunk RPC	2024-09-25 19:25:18 +02:00
Gergő Móricz	f00c0b82f9	fix(v1/scrape): add total wait specified in request to timeout	2024-09-24 21:56:22 +02:00
Gergő Móricz	3e661a2087	fix(v1/crawl-cancel): avoid double authing	2024-09-24 20:01:34 +02:00
Gergő Móricz	a59b5836d5	Revert error tallying	2024-09-24 10:27:49 +02:00
Gergő Móricz	483f97d21b	fix(v0/search): don't sent scrape fail errors to Sentry	2024-09-23 18:49:27 +02:00
Gergő Móricz	83d8287c14	fix(v0, sentry): don't send all scraping methods failed errors to Sentry	2024-09-23 18:40:21 +02:00
Nicolas	db161ac55a	Nick: press + write	2024-09-20 19:45:23 -04:00
Nicolas	3fc5ce17d2	Nick: fixed error handling for v0 scrape	2024-09-20 18:35:30 -04:00
Nicolas	0690cfeaad	Merge branch 'main' into feat/actions	2024-09-20 18:24:13 -04:00