17 Commits

Author	SHA1	Message	Date
Thomas Kosmas	acde353e56	skipTlsVerification on robots.txt scraping	2024-10-23 01:07:03 +03:00
busaud	c6ebbc6f6a	bugfix: self-host crawling doesnt respect limit	2024-10-09 22:52:49 +00:00
Gergő Móricz	3621e191bd	feat(concurrency-limit): set limit based on plan	2024-09-28 00:19:54 +02:00
Nicolas	c6f1d80992	Update crawl.ts	2024-09-05 13:03:43 -03:00
Nicolas	95b9dc915d	Nick: webhooks v1 working great	2024-09-01 13:44:36 -03:00
Nicolas	d7dbc2536d	Update crawl.ts	2024-08-30 15:21:22 -03:00
Nicolas	4003d37fbc	Nick:	2024-08-28 12:46:59 -03:00
Nicolas	558acffb33	Nick: @rafaelsideguide isarray for includes/excludes	2024-08-26 19:07:14 -03:00
Gergo Moricz	96e91ab9ec	convert webhook call to v1	2024-08-25 14:05:46 +02:00
Nicolas	39388cdc35	Update crawl.ts	2024-08-20 14:41:43 -03:00
Nicolas	b36faeaf54	Nick:	2024-08-20 14:39:52 -03:00
Nicolas	c5ad4dedeb	Update crawl.ts	2024-08-20 14:19:20 -03:00
Gergő Móricz	803577eeba	feat(crawl): webhook	2024-08-16 23:42:48 +02:00
Gergő Móricz	f20328bdbb	crawl status and document stuff	2024-08-16 22:48:05 +02:00
Gergő Móricz	8b7569f8f3	add zod, create middleware, update openapi declaration, add crawl logic	2024-08-15 23:30:33 +02:00
Gergő Móricz	4165de1773	v1 restructure	2024-08-15 21:51:59 +02:00
rafaelsideguide	6cdf4c68ec	wip: map, crawl, scrape mockups	2024-08-06 15:24:45 -03:00