18 Commits

Author	SHA1	Message	Date
Gergő Móricz	c281fe62c0	fix(crawl): propagate db fix to preview endpoint	2024-08-16 23:43:54 +02:00
Gergő Móricz	e6738abf96	fix(crawl-status): retrieve from DB in bulk	2024-08-16 23:39:39 +02:00
Gergo Moricz	9252940b52	fix(crawl-status): sort data	2024-08-13 21:55:13 +02:00
Gergo Moricz	8dbac0268c	feat: offload crawl results to the DB	2024-08-13 21:40:59 +02:00
Gergo Moricz	d7549d4dc5	feat: remove webScraperQueue	2024-08-13 21:03:24 +02:00
Gergo Moricz	86e136beca	feat: crawl to scrape conversion	2024-08-13 20:51:43 +02:00
Gergo Moricz	06751a8e21	fix(crawl-status): missing partial data after cancel	2024-08-06 17:31:20 +02:00
Gergo Moricz	8e0aa69603	fix(crawl-status): partial_data	2024-08-06 17:06:21 +02:00
Gergo Moricz	d7d63790e5	fix(crawl-status): isCancelled should be status failed	2024-08-06 16:35:55 +02:00
Gergo Moricz	03c84a9372	cleanup and fix cancelling	2024-08-06 16:26:46 +02:00
Nicolas	7e002a8b06	Nick: bull mq	2024-07-30 13:27:23 -04:00
rafaelsideguide	309728a482	updated logs	2024-07-25 09:48:06 -03:00
Kuniaki Shimizu	bd986a453c	fix USE_DB_AUTHENTICATION checks	2024-07-13 03:50:46 +09:00
rafaelsideguide	d66e1f7846	looking good	2024-06-27 16:00:45 -03:00
rafaelsideguide	c40da77be0	Added implementation for saving docs on supabase ... - TODO: remove the comments on `log_job.ts` before deploying to prod	2024-06-26 18:23:28 -03:00
Nicolas	67f135a5b6	Update crawl-status.ts	2024-05-04 12:31:28 -07:00
Nicolas	2aa09a3000	Nick: partial docs working, cleaner	2024-05-04 12:30:12 -07:00
Nicolas	23b2190e5d	Nick:	2024-04-20 16:38:05 -07:00