474 Commits

Author	SHA1	Message	Date
Nicolas	be85008622	Nick: better	2024-05-09 17:48:11 -07:00
Nicolas	be5661a768	Nick: a lot better	2024-05-09 17:45:16 -07:00
Nicolas	dcedb8d798	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:49 -07:00
Nicolas	6505bf6bf2	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:44 -07:00
Nicolas	bdbee963f7	Merge branch 'main' into nsc/cancel-job	2024-05-07 10:13:43 -07:00
rafaelsideguide	61d615c04b	Added tests	2024-05-07 14:03:00 -03:00
rafaelsideguide	e1f52c538f	nested includeHtml inside pageOptions	2024-05-07 13:40:24 -03:00
Nicolas	f46bf19fa5	Nick:	2024-05-07 09:26:52 -07:00
rafaelsideguide	83f3408634	Added max depth option	2024-05-07 11:06:26 -03:00
Nicolas	6d5da358cc	Nick: cancel job	2024-05-06 17:16:43 -07:00
rafaelsideguide	509250c4ef	changed to includeHtml	2024-05-06 19:45:56 -03:00
rafaelsideguide	538355f1af	Added toMarkdown option	2024-05-06 11:36:44 -03:00
Nicolas	15b774e974	Update index.ts	2024-05-04 12:44:30 -07:00
Nicolas	2aa09a3000	Nick: partial docs working, cleaner	2024-05-04 12:30:12 -07:00
Nicolas	00373228fa	Update index.ts	2024-05-04 11:53:16 -07:00
Nicolas	768166b066	Update single_url.ts	2024-04-30 16:57:44 -07:00
Nicolas	cbd9e88b77	Merge branch 'main' into llm-extraction	2024-04-30 14:49:20 -07:00
Nicolas	4f526cff92	Nick: cleanup	2024-04-30 12:19:43 -07:00
Caleb Peffer	3ca9e5153f	Caleb: trying to get loggin workng	2024-04-30 09:20:15 -07:00
rafaelsideguide	a095e1b63d	Resolve merge conflicts with main	2024-04-30 10:54:18 -03:00
rafaelsideguide	d3c36adaa7	Update index.ts	2024-04-29 17:58:47 -03:00
rafaelsideguide	f8b207793f	changed the request to do a HEAD to check for a PDF instead	2024-04-29 15:15:32 -03:00
Nicolas	b69feab916	Merge branch 'main' into llm-extraction	2024-04-29 08:40:44 -07:00
Caleb Peffer	2ad7a58eb7	Caleb: first test passing	2024-04-28 17:38:20 -07:00
Caleb Peffer	06497729e2	Caleb: got it to a testable state I believe	2024-04-28 15:52:09 -07:00
Caleb Peffer	6ee1f2d3bc	Caleb: initially pulled inspiration code from https://github.com/mishushakov/llm-scraper	2024-04-28 13:59:35 -07:00
Nicolas	68838c9e0d	Update single_url.ts	2024-04-28 12:44:00 -07:00
Nicolas	d8ee4e90d6	Update website_params.ts	2024-04-28 11:47:25 -07:00
Nicolas	8e44696c4d	Nick:	2024-04-28 11:34:25 -07:00
Nicolas	1dc6458c6a	Update crawler.ts	2024-04-27 11:17:10 -07:00
Nicolas	0f694e0608	Update crawler.ts	2024-04-27 11:14:52 -07:00
tractorjuice	a5d38039f2	Add additional file extensions to crawler.ts ... Add additional file extensions.	2024-04-27 11:03:27 +01:00
rafaelsideguide	75597f72a1	[Feat] Added allowed urls ... FireCrawl should be able to scrape LinkedIn Articles (/pulse/*)	2024-04-25 08:39:45 -03:00
Rafael Miller	f189589da4	Merge pull request #34 from mendableai/nsc/returnOnlyUrls ... Implements the ability for the crawler to output all the links it found, without scraping	2024-04-24 10:34:42 -03:00
rafaelsideguide	942ac3b41c	Resolved merge conflicts between feat/added-anthropic-vision-api and main	2024-04-24 09:57:45 -03:00
Nicolas	8939ca570b	Merge branch 'main' into nsc/returnOnlyUrls	2024-04-23 18:05:48 -07:00
Nicolas	fdb2789eaa	Nick: added url as return param	2024-04-23 17:14:34 -07:00
Nicolas	734c76fc56	Merge branch 'main' into nsc/mvp-search	2024-04-23 17:04:31 -07:00
Nicolas	f0695c7123	Update single_url.ts	2024-04-23 17:04:10 -07:00
Nicolas	0146157876	Nick: mvp	2024-04-23 15:28:32 -07:00
rafaelsideguide	849c0b6ebf	[Feat] Added blocklist for social media urls	2024-04-23 18:50:35 -03:00
Nicolas	306cfe4ce1	Nick:	2024-04-23 11:15:11 -07:00
Nicolas	ddf9ff9c9a	Nick:	2024-04-20 11:46:06 -07:00
Nicolas	f1dd97af0f	Update index.ts	2024-04-19 15:37:27 -07:00
Nicolas	84cebf618b	Nick:	2024-04-19 15:36:00 -07:00
Nicolas	5b93799149	Nick: a bit faster	2024-04-19 15:13:17 -07:00
Nicolas	c5cb268b61	Update pdfProcessor.ts	2024-04-19 13:13:42 -07:00
Nicolas	43cfcec326	Nick: disabling in crawl and sitemap for now	2024-04-19 13:12:08 -07:00
Nicolas	140529c609	Nick: fixes pdfs not found	2024-04-19 13:05:21 -07:00
Ikko Eltociear Ashimine	9e9d66f7a3	refactor: fix typo in WebScraper/index.ts ... breakign -> breaking	2024-04-20 02:27:53 +09:00