339 Commits

Author	SHA1	Message	Date
Ikko Eltociear Ashimine	e91c122c69	Merge branch 'main' into patch-1	2024-05-15 12:14:52 +09:00
Nicolas	a0fdc6f7c6	Nick:	2024-05-14 12:12:40 -07:00
Nicolas	7f31959be7	Nick:	2024-05-14 12:04:36 -07:00
Nicolas	8a72cf556b	Nick:	2024-05-13 21:10:58 -07:00
Nicolas	26a092f780	Update index.ts	2024-05-13 21:04:49 -07:00
Nicolas	8101cbee37	Update index.ts	2024-05-13 21:02:47 -07:00
Nicolas	86b8439844	Nick:	2024-05-13 20:51:42 -07:00
Nicolas	a96fc5b96d	Nick: 4x speed	2024-05-13 20:45:11 -07:00
rafaelsideguide	8eb2e95f19	Cleaned up	2024-05-13 16:13:10 -03:00
Nicolas	2ce045912f	Nick: disable vision right now	2024-05-13 10:56:08 -07:00
rafaelsideguide	f4348024c6	Added check during scraping to deal with pdfs ... Checks if the URL is a PDF during the scraping process (single_url.ts). TODO: Run integration tests - Does this strat affect the running time? ps. Some comments need to be removed if we decide to proceed with this strategy.	2024-05-13 09:13:42 -03:00
Rafael Miller	5a2712fa5a	Merge branch 'main' into detect-pdfs	2024-05-10 15:53:13 -03:00
rafaelsideguide	bc6b929b43	[Bug] Fixing /crawl limit	2024-05-10 12:15:54 -03:00
Nicolas	66bd1e4020	Update website_params.ts	2024-05-09 18:41:15 -07:00
Nicolas	d21091bb06	Update single_url.ts	2024-05-09 17:52:46 -07:00
Nicolas	be85008622	Nick: better	2024-05-09 17:48:11 -07:00
Nicolas	be5661a768	Nick: a lot better	2024-05-09 17:45:16 -07:00
Nicolas	dcedb8d798	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:49 -07:00
Nicolas	6505bf6bf2	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:44 -07:00
Nicolas	bdbee963f7	Merge branch 'main' into nsc/cancel-job	2024-05-07 10:13:43 -07:00
rafaelsideguide	61d615c04b	Added tests	2024-05-07 14:03:00 -03:00
rafaelsideguide	e1f52c538f	nested includeHtml inside pageOptions	2024-05-07 13:40:24 -03:00
Nicolas	f46bf19fa5	Nick:	2024-05-07 09:26:52 -07:00
rafaelsideguide	83f3408634	Added max depth option	2024-05-07 11:06:26 -03:00
Nicolas	6d5da358cc	Nick: cancel job	2024-05-06 17:16:43 -07:00
rafaelsideguide	509250c4ef	changed to includeHtml	2024-05-06 19:45:56 -03:00
rafaelsideguide	538355f1af	Added toMarkdown option	2024-05-06 11:36:44 -03:00
Nicolas	15b774e974	Update index.ts	2024-05-04 12:44:30 -07:00
Nicolas	2aa09a3000	Nick: partial docs working, cleaner	2024-05-04 12:30:12 -07:00
Nicolas	00373228fa	Update index.ts	2024-05-04 11:53:16 -07:00
Nicolas	768166b066	Update single_url.ts	2024-04-30 16:57:44 -07:00
Nicolas	cbd9e88b77	Merge branch 'main' into llm-extraction	2024-04-30 14:49:20 -07:00
Nicolas	4f526cff92	Nick: cleanup	2024-04-30 12:19:43 -07:00
Caleb Peffer	3ca9e5153f	Caleb: trying to get loggin workng	2024-04-30 09:20:15 -07:00
rafaelsideguide	a095e1b63d	Resolve merge conflicts with main	2024-04-30 10:54:18 -03:00
rafaelsideguide	d3c36adaa7	Update index.ts	2024-04-29 17:58:47 -03:00
rafaelsideguide	f8b207793f	changed the request to do a HEAD to check for a PDF instead	2024-04-29 15:15:32 -03:00
Nicolas	b69feab916	Merge branch 'main' into llm-extraction	2024-04-29 08:40:44 -07:00
Caleb Peffer	2ad7a58eb7	Caleb: first test passing	2024-04-28 17:38:20 -07:00
Caleb Peffer	06497729e2	Caleb: got it to a testable state I believe	2024-04-28 15:52:09 -07:00
Caleb Peffer	6ee1f2d3bc	Caleb: initially pulled inspiration code from https://github.com/mishushakov/llm-scraper	2024-04-28 13:59:35 -07:00
Nicolas	68838c9e0d	Update single_url.ts	2024-04-28 12:44:00 -07:00
Nicolas	d8ee4e90d6	Update website_params.ts	2024-04-28 11:47:25 -07:00
Nicolas	8e44696c4d	Nick:	2024-04-28 11:34:25 -07:00
Nicolas	1dc6458c6a	Update crawler.ts	2024-04-27 11:17:10 -07:00
Nicolas	0f694e0608	Update crawler.ts	2024-04-27 11:14:52 -07:00
tractorjuice	a5d38039f2	Add additional file extensions to crawler.ts ... Add additional file extensions.	2024-04-27 11:03:27 +01:00
rafaelsideguide	75597f72a1	[Feat] Added allowed urls ... FireCrawl should be able to scrape LinkedIn Articles (/pulse/*)	2024-04-25 08:39:45 -03:00
Rafael Miller	f189589da4	Merge pull request #34 from mendableai/nsc/returnOnlyUrls ... Implements the ability for the crawler to output all the links it found, without scraping	2024-04-24 10:34:42 -03:00
rafaelsideguide	942ac3b41c	Resolved merge conflicts between feat/added-anthropic-vision-api and main	2024-04-24 09:57:45 -03:00