60 Commits

Author	SHA1	Message	Date
Gergo Moricz	d7549d4dc5	feat: remove webScraperQueue	2024-08-13 21:03:24 +02:00
Gergő Móricz	4a2c37dcf5	Merge branch 'main' into feat/queue-scrapes	2024-08-13 20:53:49 +02:00
Nicolas	74a5125185	Nick: removed redlock	2024-08-12 15:07:30 -04:00
Nicolas	f43d5e7895	Nick: scrape queue	2024-07-30 14:44:13 -04:00
Nicolas	7e002a8b06	Nick: bull mq	2024-07-30 13:27:23 -04:00
Nicolas	04942bb9de	Nick:	2024-07-29 18:31:43 -04:00
Nicolas	10e80f00cf	Merge branch 'main' into nsc/admin-router	2024-07-25 16:46:38 -04:00
Nicolas	e5b797549e	Merge branch 'main' into feat/scrape-monitoring	2024-07-25 16:21:02 -04:00
Nicolas	2014d9dd2e	Nick: admin router	2024-07-25 14:54:20 -04:00
rafaelsideguide	309728a482	updated logs	2024-07-25 09:48:06 -03:00
Gergo Moricz	60c74357df	feat(ScrapeEvents): log queue events	2024-07-24 18:44:14 +02:00
rafaelsideguide	6208ecdbc0	added logger	2024-07-23 17:30:46 -03:00
Gergo Moricz	8160c311c0	fix queue stuck bug via lock setting changes	2024-07-17 21:31:25 +02:00
Nicolas	ffc3b7c5fb	Update index.ts	2024-07-16 22:42:40 -04:00
Nicolas	c9073a747c	Nick:	2024-07-16 22:41:13 -04:00
Nicolas	436e8922a7	Nick: doing on the ci instead	2024-07-12 19:49:38 -04:00
Nicolas	fd18f2269b	Nick: slack alerts	2024-07-12 19:07:59 -04:00
Nicolas	b5b75086c1	Update index.ts	2024-07-12 10:44:14 -04:00
Gergo Moricz	0d3e09e798	fix: try-catch job removal	2024-07-12 16:35:50 +02:00
Gergő Móricz	69d724714f	Merge branch 'main' into mog/job-stuck-fix	2024-07-12 16:33:34 +02:00
Nicolas	c3eecf7b9f	Update index.ts	2024-07-12 10:22:06 -04:00
Gergo Moricz	10957b748b	fix(bull): requeue jobs after restart	2024-07-12 13:55:53 +02:00
Gergo Moricz	7e3a368684	fix: unpause globally	2024-07-12 00:05:35 +02:00
Gergo Moricz	ee1d41406e	feat: unpause by http request	2024-07-11 23:56:36 +02:00
Gergo Moricz	bd84290b9e	fix: reenable hyperdx	2024-07-11 23:20:51 +02:00
Gergo Moricz	09bca05b20	feat: fix iteration 3 (actually works)	2024-07-11 23:14:15 +02:00
Gergo Moricz	bffb9f8fd0	feat: stuck job restoration iteration 2	2024-07-11 20:08:21 +02:00
rafaelsideguide	86d0e88a91	removed hyperdx (they also have graceful shutdown) and tried to change the process for running on server. It didn't work.	2024-07-10 18:29:55 -03:00
Gergo Moricz	1a07e9d23b	feat: pick up and commit interrupted jobs from/to DB	2024-07-09 15:57:38 +02:00
Gergo Moricz	77aa46588f	feat: graceful exit handler	2024-07-09 14:29:32 +02:00
rafaelsideguide	d02829d335	fixed clean jobs	2024-06-25 17:49:29 -03:00
Nicolas	182f8d4d6c	Update index.ts	2024-06-12 18:07:05 -07:00
Nicolas	67dc46b454	Nick: clusters	2024-06-12 17:53:04 -07:00
rafaelsideguide	157fbe4a1e	added bull auth key	2024-06-11 17:52:01 -03:00
rafaelsideguide	a9f93c2f1e	Added route to clean completed jobs and a github action cron that triggers every 24h	2024-06-11 14:18:05 -03:00
Nicolas	ace46f340b	Nick: new limits, new pricing	2024-05-30 14:31:36 -07:00
Nicolas	229b9908d2	Nick: only enable hyper dx in prod	2024-05-21 18:52:46 -07:00
Nicolas	9e61d431f0	Nick: hyper dx integration init	2024-05-20 13:36:34 -07:00
rafaelsideguide	9b01dc6281	Changed from active to waiting jobs	2024-04-23 16:07:22 -03:00
rafaelsideguide	a680c7ce84	[Feat] Server health check + slack message	2024-04-23 15:46:29 -03:00
Nicolas	898d729a84	Nick: tests	2024-04-21 11:27:31 -07:00
Nicolas	4543c57e4e	Nick:	2024-04-20 19:04:27 -07:00
Nicolas	23b2190e5d	Nick:	2024-04-20 16:38:05 -07:00
Nicolas	5b3c75b06e	Nick:	2024-04-20 14:10:29 -07:00
Nicolas	43c2e877e7	Update index.ts	2024-04-20 14:05:01 -07:00
Nicolas	408c7a479f	Nick: rate limit fixes	2024-04-20 14:02:22 -07:00
rafaelsideguide	ddb3b25171	adding ci-cd workflow	2024-04-18 16:28:01 -03:00
Nicolas	3e9e24aaf1	Update index.ts	2024-04-18 11:01:24 -07:00
Nicolas	0f7ab4107f	Update index.ts	2024-04-18 10:41:06 -07:00
Nicolas	6112cc1c2c	Update index.ts	2024-04-18 10:34:41 -07:00