1738 Commits

Author	SHA1	Message	Date
rafaelsideguide	f453bcf17c	bugfix docker self hosting	2024-07-12 16:51:20 -03:00
Nicolas	0ddaac6ae0	Nick: fixed the other instances as well	2024-07-12 15:39:10 -04:00
Nicolas	5da03a8fbd	Update fireEngine.ts	2024-07-12 14:59:49 -04:00
Kuniaki Shimizu	bd986a453c	fix USE_DB_AUTHENTICATION checks	2024-07-13 03:50:46 +09:00
Nicolas	b5b75086c1	Update index.ts	2024-07-12 10:44:14 -04:00
Gergo Moricz	0d3e09e798	fix: try-catch job removal	2024-07-12 16:35:50 +02:00
Gergő Móricz	69d724714f	Merge branch 'main' into mog/job-stuck-fix	2024-07-12 16:33:34 +02:00
Nicolas	c3eecf7b9f	Update index.ts	2024-07-12 10:22:06 -04:00
Gergo Moricz	10957b748b	fix(bull): requeue jobs after restart	2024-07-12 13:55:53 +02:00
Nicolas	961b27811d	Merge pull request #386 from mendableai/feat/fire-engine-fallback-for-sitemap ... [Feat] Added fire-engine fallback for getting sitemaps	2024-07-11 20:38:01 -04:00
Nicolas	84de63dbeb	Merge pull request #375 from StefanTerdell/self-host-qol ... Self-hosting quality of life fixes	2024-07-11 20:37:39 -04:00
Nicolas	30c1118713	Merge pull request #326 from mendableai/feat/save-docs-on-supabase ... [Feat] Added implementation for saving docs on supabase	2024-07-11 20:27:41 -04:00
Gergo Moricz	7e3a368684	fix: unpause globally	2024-07-12 00:05:35 +02:00
Gergo Moricz	ee1d41406e	feat: unpause by http request	2024-07-11 23:56:36 +02:00
Gergo Moricz	bd84290b9e	fix: reenable hyperdx	2024-07-11 23:20:51 +02:00
Gergo Moricz	09bca05b20	feat: fix iteration 3 (actually works)	2024-07-11 23:14:15 +02:00
Gergo Moricz	9cd7d79b64	feat: avoid double SIGINT crashing	2024-07-11 20:35:15 +02:00
Gergo Moricz	bffb9f8fd0	feat: stuck job restoration iteration 2	2024-07-11 20:08:21 +02:00
rafaelsideguide	86d0e88a91	removed hyperdx (they also have graceful shutdown) and tried to change the process for running on server. It didn't work.	2024-07-10 18:29:55 -03:00
rafaelsideguide	9ad06fdf56	added fire-engine fallback for getting sitemaps	2024-07-09 16:07:53 -03:00
Gergo Moricz	1a07e9d23b	feat: pick up and commit interrupted jobs from/to DB	2024-07-09 15:57:38 +02:00
Gergo Moricz	77aa46588f	feat: graceful exit handler	2024-07-09 14:29:32 +02:00
Stefan Terdell	188fe56203	Optional jobId webhook URL templating	2024-07-07 15:11:45 +02:00
Stefan Terdell	a2ae5f81d9	Only check Supabase if configured to	2024-07-07 15:06:31 +02:00
rafaelsideguide	c2bba54b4f	Added veeva to special case params	2024-07-05 16:58:07 -03:00
rafaelsideguide	0ab6cef471	Merge remote-tracking branch 'origin/main' into dependabot/npm_and_yarn/apps/api/prod-deps-5b38a50718	2024-07-05 14:00:10 -03:00
Nicolas	914897c9d2	Merge branch 'main' into feat/save-docs-on-supabase	2024-07-05 12:27:22 -03:00
rafaelsideguide	538dc63035	Fixing rate-limiter-flexible package version ... Redis version <3.0.2 throws TS bug: https://github.com/animir/node-rate-limiter-flexible/issues/228	2024-07-05 12:12:00 -03:00
Nicolas	32849b017f	Nick:	2024-07-03 20:18:11 -03:00
Nicolas	066d92f643	Update single_url.ts	2024-07-03 18:38:17 -03:00
Nicolas	f5b2fbd7e8	Nick: revision	2024-07-03 18:06:53 -03:00
Nicolas	2d30cc6117	Nick: comments	2024-07-03 18:01:54 -03:00
Nicolas	90c54c32fd	Nick: refactor	2024-07-03 18:01:17 -03:00
Nicolas	90cf799a3c	Update single_url.ts	2024-07-03 17:56:21 -03:00
Nicolas	b36406e465	Nick: log scrpaers	2024-07-03 17:28:53 -03:00
rafaelsideguide	0175152577	Fixed PDF match custom scraping ... Now it's working for both `https://getgc.ai/privacy` and `https://prairie.cards/products/wood-designs` usecases.	2024-07-02 11:25:17 -03:00
rafaelsideguide	96de948d6b	Update index.test.ts	2024-07-02 11:04:09 -03:00
rafaelsideguide	7b7154ba1e	bugfixed pageStatusCode	2024-07-02 10:51:35 -03:00
Rafael Miller	f0f449fe51	Merge pull request #336 from snippet/allow-external-content-links ... [Proposal] new feature allowExternalContentLinks	2024-07-02 09:45:21 -03:00
rafaelsideguide	db4a743365	Added e2e test	2024-07-02 09:44:08 -03:00
Nicolas	42cd58a679	Merge pull request #332 from mendableai/feat/rawHtmlExtraction ... Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html"	2024-07-01 18:23:26 -03:00
rafaelsideguide	16aac7f8c5	Update single_url.ts	2024-07-01 18:21:15 -03:00
Nicolas	6d0c7a9ccd	Merge pull request #323 from mendableai/tests/crawl-limit-unit-tests ... [Tests] Added crawl limit unit test	2024-07-01 17:56:04 -03:00
rafaelsideguide	4d6e25619b	minor spacing and comment stuff	2024-07-01 16:05:34 -03:00
Eric Ciarla	e1af815f8c	Update scrape.ts	2024-07-01 08:48:21 -04:00
Eric Ciarla	7ae195bacc	Update index.test.ts	2024-06-29 10:13:12 -04:00
Eric Ciarla	837b446390	Update index.test.ts	2024-06-29 08:48:42 -04:00
Eric Ciarla	fe6e3aeadc	Update index.test.ts	2024-06-29 08:44:21 -04:00
Eric Ciarla	6c9f0dfc91	Add tests	2024-06-29 08:32:20 -04:00
Jeff Pereira	a5fb45988c	new feature allowExternalContentLinks	2024-06-28 17:23:40 -07:00