46 Commits

Author	SHA1	Message	Date
rafaelsideguide	f4348024c6	Added check during scraping to deal with pdfs ... Checks if the URL is a PDF during the scraping process (single_url.ts). TODO: Run integration tests - Does this strat affect the running time? ps. Some comments need to be removed if we decide to proceed with this strategy.	2024-05-13 09:13:42 -03:00
Rafael Miller	5a2712fa5a	Merge branch 'main' into detect-pdfs	2024-05-10 15:53:13 -03:00
Nicolas	dcedb8d798	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:49 -07:00
Nicolas	6505bf6bf2	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:44 -07:00
Nicolas	bdbee963f7	Merge branch 'main' into nsc/cancel-job	2024-05-07 10:13:43 -07:00
rafaelsideguide	61d615c04b	Added tests	2024-05-07 14:03:00 -03:00
rafaelsideguide	e1f52c538f	nested includeHtml inside pageOptions	2024-05-07 13:40:24 -03:00
Nicolas	f46bf19fa5	Nick:	2024-05-07 09:26:52 -07:00
rafaelsideguide	83f3408634	Added max depth option	2024-05-07 11:06:26 -03:00
Nicolas	6d5da358cc	Nick: cancel job	2024-05-06 17:16:43 -07:00
rafaelsideguide	509250c4ef	changed to includeHtml	2024-05-06 19:45:56 -03:00
rafaelsideguide	538355f1af	Added toMarkdown option	2024-05-06 11:36:44 -03:00
Nicolas	15b774e974	Update index.ts	2024-05-04 12:44:30 -07:00
Nicolas	2aa09a3000	Nick: partial docs working, cleaner	2024-05-04 12:30:12 -07:00
Nicolas	00373228fa	Update index.ts	2024-05-04 11:53:16 -07:00
Nicolas	cbd9e88b77	Merge branch 'main' into llm-extraction	2024-04-30 14:49:20 -07:00
Nicolas	4f526cff92	Nick: cleanup	2024-04-30 12:19:43 -07:00
Caleb Peffer	3ca9e5153f	Caleb: trying to get loggin workng	2024-04-30 09:20:15 -07:00
rafaelsideguide	a095e1b63d	Resolve merge conflicts with main	2024-04-30 10:54:18 -03:00
rafaelsideguide	d3c36adaa7	Update index.ts	2024-04-29 17:58:47 -03:00
rafaelsideguide	f8b207793f	changed the request to do a HEAD to check for a PDF instead	2024-04-29 15:15:32 -03:00
Caleb Peffer	2ad7a58eb7	Caleb: first test passing	2024-04-28 17:38:20 -07:00
Caleb Peffer	06497729e2	Caleb: got it to a testable state I believe	2024-04-28 15:52:09 -07:00
Rafael Miller	f189589da4	Merge pull request #34 from mendableai/nsc/returnOnlyUrls ... Implements the ability for the crawler to output all the links it found, without scraping	2024-04-24 10:34:42 -03:00
rafaelsideguide	942ac3b41c	Resolved merge conflicts between feat/added-anthropic-vision-api and main	2024-04-24 09:57:45 -03:00
Nicolas	ddf9ff9c9a	Nick:	2024-04-20 11:46:06 -07:00
Nicolas	f1dd97af0f	Update index.ts	2024-04-19 15:37:27 -07:00
Nicolas	84cebf618b	Nick:	2024-04-19 15:36:00 -07:00
Nicolas	5b93799149	Nick: a bit faster	2024-04-19 15:13:17 -07:00
Nicolas	43cfcec326	Nick: disabling in crawl and sitemap for now	2024-04-19 13:12:08 -07:00
Nicolas	140529c609	Nick: fixes pdfs not found	2024-04-19 13:05:21 -07:00
rafaelsideguide	72e1dadccd	adding option to replace all relative paths with absolute paths	2024-04-19 11:47:20 -03:00
rafaelsideguide	c4cc4b9262	fixing document response	2024-04-18 14:12:39 -03:00
Rafael Miller	704a059448	Update index.ts	2024-04-18 13:53:11 -03:00
rafaelsideguide	57e5b36014	[Feat] Adding pdf parser	2024-04-18 11:43:57 -03:00
Nicolas	36abe0f7f9	Nick:	2024-04-17 18:24:46 -07:00
Nicolas	52fb28bc1a	Update index.ts	2024-04-17 12:52:15 -07:00
Nicolas	de439f6529	Update index.ts	2024-04-17 12:51:29 -07:00
Nicolas	871d5d91b0	Update index.ts	2024-04-17 12:51:12 -07:00
rafaelsideguide	b375ce3e39	adding unit tests and bugfixing	2024-04-17 14:54:54 -03:00
Nicolas	27674a624d	Update index.ts	2024-04-17 10:39:00 -07:00
rafaelsideguide	00941d94a4	Added anthropic vision to getImageDescription function	2024-04-16 18:03:48 -03:00
rafaelsideguide	d23a7ae591	improving relative paths	2024-04-16 16:34:01 -03:00
rafaelsideguide	a04610302a	Spliting relative paths for images	2024-04-16 16:31:33 -03:00
Nicolas	4c4775e0b8	Nick:	2024-04-16 12:49:14 -04:00
Nicolas	a6c2a87811	Initial commit	2024-04-15 17:01:47 -04:00