38 Commits

Author	SHA1	Message	Date
Nicolas	e28c415cf4	Nick:	2024-08-09 14:07:46 -04:00
Gergo Moricz	920b7f2f44	fix(runWebScraper): don't filter empty docs	2024-08-07 21:00:22 +02:00
Gergo Moricz	55ec96c23f	fix(queue-worker): bad job lock extension time	2024-08-07 20:24:16 +02:00
Gergo Moricz	ab7a35c581	fix(queue-worker): log lock extensions	2024-08-07 19:49:48 +02:00
Gergo Moricz	a1c2ee5aa9	fix: always complete job, no try	2024-08-07 19:39:09 +02:00
Gergo Moricz	191dfbd9ca	fix: move to completed in one place	2024-08-07 18:49:58 +02:00
Gergő Móricz	5fc7fcb77c	Merge branch 'main' into feat/queue-scrapes	2024-08-07 16:35:44 +02:00
Gergo Moricz	fe9fdb578b	revert bad hotfixes	2024-08-07 16:34:25 +02:00
Gergo Moricz	cdf7bad5b4	fix(runWebScraper): don't move to completed	2024-08-07 15:20:56 +02:00
Gergo Moricz	9df8719efa	fix(queue-worker): raise queue log level to info	2024-08-07 14:56:04 +02:00
Gergo Moricz	7bb922071c	fix(queue-worker): manually renew lock (testing)	2024-08-07 14:35:20 +02:00
Gergo Moricz	7c5cda7b45	fix(queue-worker): concurrency	2024-08-06 16:57:00 +02:00
Gergo Moricz	03c84a9372	cleanup and fix cancelling	2024-08-06 16:26:46 +02:00
Nicolas	52198f2991	Nick:	2024-07-30 16:15:08 -04:00
Nicolas	f43d5e7895	Nick: scrape queue	2024-07-30 14:44:13 -04:00
Nicolas	7e002a8b06	Nick: bull mq	2024-07-30 13:27:23 -04:00
Nicolas	46bcbd931f	Merge branch 'main' into feat/queue-scrapes	2024-07-30 12:44:07 -04:00
Gergő Móricz	6798695ee4	feat: move scraper to queue	2024-07-25 00:14:25 +02:00
Gergo Moricz	60c74357df	feat(ScrapeEvents): log queue events	2024-07-24 18:44:14 +02:00
rafaelsideguide	6208ecdbc0	added logger	2024-07-23 17:30:46 -03:00
Gergo Moricz	8160c311c0	fix queue stuck bug via lock setting changes	2024-07-17 21:31:25 +02:00
Gergo Moricz	09bca05b20	feat: fix iteration 3 (actually works)	2024-07-11 23:14:15 +02:00
Gergo Moricz	9cd7d79b64	feat: avoid double SIGINT crashing	2024-07-11 20:35:15 +02:00
Gergo Moricz	bffb9f8fd0	feat: stuck job restoration iteration 2	2024-07-11 20:08:21 +02:00
rafaelsideguide	86d0e88a91	removed hyperdx (they also have graceful shutdown) and tried to change the process for running on server. It didn't work.	2024-07-10 18:29:55 -03:00
rafaelsideguide	c40da77be0	Added implementation for saving docs on supabase ... - TODO: remove the comments on `log_job.ts` before deploying to prod	2024-06-26 18:23:28 -03:00
Nicolas	08c1fa799b	Update queue-worker.ts	2024-06-24 16:51:32 -03:00
rafaelsideguide	21d29de819	testing crawl with new.abb.com case ... many unnecessary console.logs for tracing the code execution	2024-06-24 16:25:07 -03:00
rafaelsideguide	d20af257ba	Added jobId to webhook data	2024-06-12 15:38:41 -03:00
Nicolas	229b9908d2	Nick: only enable hyper dx in prod	2024-05-21 18:52:46 -07:00
Nicolas	9e61d431f0	Nick: hyper dx integration init	2024-05-20 13:36:34 -07:00
Nicolas	a96fc5b96d	Nick: 4x speed	2024-05-13 20:45:11 -07:00
Nicolas	d2f808a5fd	Update queue-worker.ts	2024-04-20 19:54:37 -07:00
Nicolas	9b31e68a7e	Update queue-worker.ts	2024-04-20 19:38:44 -07:00
Nicolas	0db0874b00	Nick:	2024-04-20 19:37:45 -07:00
Nicolas	6aa3cc3ce8	Nick:	2024-04-20 13:53:11 -07:00
rafaelsideguide	72e1dadccd	adding option to replace all relative paths with absolute paths	2024-04-19 11:47:20 -03:00
Nicolas	a6c2a87811	Initial commit	2024-04-15 17:01:47 -04:00