339 Commits

Author	SHA1	Message	Date
Nicolas	99059814a8	Nick:	2024-06-03 21:32:48 -07:00
Nicolas	918059ee9e	Merge branch 'main' into nsc/improvemnts-fixes-misc	2024-06-03 16:46:02 -07:00
Nicolas	38e583f66c	Update socialBlockList.test.ts	2024-06-03 16:44:23 -07:00
Nicolas	c69c89f838	Nick:	2024-06-03 16:42:42 -07:00
Nicolas	48d1ec05b2	Merge branch 'main' into nsc/improved-blocklist	2024-06-03 16:38:03 -07:00
Nicolas	d30ced4394	Merge pull request #221 from mendableai/nsc/fwd-header-auth ... feat: Ability to forward headers to reliable providers for auth etc...	2024-06-03 16:33:40 -07:00
rafaelsideguide	1fc3a15149	Update single_url.ts	2024-06-03 15:24:40 -03:00
Nicolas	fde522c3e1	Update single_url.ts	2024-06-02 20:23:45 -07:00
Matt Joyce	deefe65cbe	Change the way the playwright response is parsed ... Was failing with a Type Error, but actually looked ok. This fixes the type error, and stop scraper fallback.	2024-06-01 19:16:56 +10:00
Nicolas	8cb62dde92	Update website_params.ts	2024-05-31 16:09:39 -07:00
Nicolas	3b8059edb6	Update single_url.ts	2024-05-31 15:43:06 -07:00
Nicolas	6bea803120	Nick:	2024-05-31 15:39:54 -07:00
Nicolas	6c939d534d	Nick: small refactor	2024-05-29 19:43:51 -07:00
Eric Ciarla	37915e11e8	Final push	2024-05-29 21:18:24 -04:00
Eric Ciarla	a0e404f94e	init commit	2024-05-29 18:56:57 -04:00
rafaelsideguide	ee9a2184e2	Added custom scraping conditions for readme docs	2024-05-29 13:39:43 -03:00
Nicolas	1b3547dcf2	Nick:	2024-05-28 12:56:24 -07:00
Nicolas	e98434606d	Update blocklist.ts	2024-05-24 15:04:15 -07:00
Nicolas	e5c8719554	Update blocklist.ts	2024-05-24 14:53:04 -07:00
Nicolas	53a214cefb	Merge pull request #168 from mendableai/nsc/allowed-keywords-in-blocklist ... feat: Allow privacy/legal/ other pages in social media websites	2024-05-24 09:43:15 -07:00
rafaelsideguide	aa6df4305e	crawl load tests 6 and 7	2024-05-22 18:20:24 -03:00
rafaelsideguide	73f1d09d39	Update website_params.ts	2024-05-22 15:07:12 -03:00
rafaelsideguide	f4a3469b9e	Merge branch 'main' into bug/crawl-limit	2024-05-22 14:27:28 -03:00
Nicolas	0d187f0425	Merge pull request #77 from tractorjuice/patch-1 ... Add additional file extensions to crawler.ts	2024-05-22 10:16:49 -07:00
rafaelsideguide	04a0bef0fb	Merge branch 'main' into test/load-testing	2024-05-22 11:26:19 -03:00
rafaelsideguide	e4573c08ca	Update website_params.ts	2024-05-22 11:24:48 -03:00
Nicolas	a8ff295977	Update single_url.ts	2024-05-21 18:50:42 -07:00
Nicolas	a5e718b084	Nick: improvements	2024-05-21 18:34:23 -07:00
Nicolas	6285f12cd1	Merge pull request #167 from mendableai/nsc/hyper-dx-integration ... feat: HyperDX Integration	2024-05-21 13:19:38 -07:00
rafaelsideguide	75f4e34d8e	Merge branch 'main' into test/load-testing	2024-05-21 10:28:02 -03:00
Nicolas	7f64fe884a	Update blocklist.ts	2024-05-20 17:26:01 -07:00
Nicolas	756f54466d	Nick: allowed keywords for now	2024-05-20 17:24:21 -07:00
Nicolas	77a79b5a79	Nick: max num tokens for llm extract (for now) + slice the max	2024-05-20 17:07:38 -07:00
Nicolas	9e61d431f0	Nick: hyper dx integration init	2024-05-20 13:36:34 -07:00
rafaelsideguide	d667e1417b	added fly staging load test ... - being rate limited. Need to add the token to the rate-limit functions	2024-05-17 19:09:19 -03:00
Nicolas	6feb21cc35	Update website_params.ts	2024-05-17 11:21:26 -07:00
Nicolas	5be208f595	Nick: fixed	2024-05-17 10:40:44 -07:00
Nicolas	df6c3d1e7d	Merge branch 'main' into detect-pdfs	2024-05-17 09:55:51 -07:00
Nicolas	9d635cb2a3	Nick: docx support	2024-05-16 11:48:02 -07:00
Nicolas	098db17913	Update index.ts	2024-05-15 17:37:09 -07:00
Nicolas	6ca368327f	Merge branch 'main' into test/crawl-options	2024-05-15 17:18:25 -07:00
Nicolas	24be4866c5	Nick:	2024-05-15 17:16:20 -07:00
Nicolas	ade4e05cff	Nick: working	2024-05-15 17:13:04 -07:00
Nicolas	bfccaf670d	Nick: fixes most of it	2024-05-15 15:30:37 -07:00
rafaelsideguide	d91043376c	not working yet	2024-05-15 18:54:40 -03:00
rafaelsideguide	fa014defc7	Fixing child links only bug	2024-05-15 18:35:09 -03:00
Nicolas	2ba743fb1a	Merge pull request #27 from eltociear/patch-1 ... refactor: fix typo in WebScraper/index.ts	2024-05-15 13:28:38 -07:00
Nicolas	1b0d6341d3	Update index.ts	2024-05-15 11:48:12 -07:00
Nicolas	d10f81e7fe	Nick: fixes	2024-05-15 11:28:20 -07:00
Nicolas	87570bdfa1	Update index.ts	2024-05-15 11:06:03 -07:00