54 Commits

Author	SHA1	Message	Date
Nicolas	098db17913	Update index.ts	2024-05-15 17:37:09 -07:00
Nicolas	6ca368327f	Merge branch 'main' into test/crawl-options	2024-05-15 17:18:25 -07:00
Nicolas	ade4e05cff	Nick: working	2024-05-15 17:13:04 -07:00
Nicolas	bfccaf670d	Nick: fixes most of it	2024-05-15 15:30:37 -07:00
rafaelsideguide	d91043376c	not working yet	2024-05-15 18:54:40 -03:00
rafaelsideguide	fa014defc7	Fixing child links only bug	2024-05-15 18:35:09 -03:00
Nicolas	2ba743fb1a	Merge pull request #27 from eltociear/patch-1 ... refactor: fix typo in WebScraper/index.ts	2024-05-15 13:28:38 -07:00
Nicolas	1b0d6341d3	Update index.ts	2024-05-15 11:48:12 -07:00
Nicolas	d10f81e7fe	Nick: fixes	2024-05-15 11:28:20 -07:00
Nicolas	87570bdfa1	Update index.ts	2024-05-15 11:06:03 -07:00
Ikko Eltociear Ashimine	e91c122c69	Merge branch 'main' into patch-1	2024-05-15 12:14:52 +09:00
Nicolas	a0fdc6f7c6	Nick:	2024-05-14 12:12:40 -07:00
Nicolas	7f31959be7	Nick:	2024-05-14 12:04:36 -07:00
Nicolas	8a72cf556b	Nick:	2024-05-13 21:10:58 -07:00
Nicolas	a96fc5b96d	Nick: 4x speed	2024-05-13 20:45:11 -07:00
Nicolas	2ce045912f	Nick: disable vision right now	2024-05-13 10:56:08 -07:00
Nicolas	dcedb8d798	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:49 -07:00
Nicolas	6505bf6bf2	Merge branch 'main' into feat/max-depth	2024-05-07 10:20:44 -07:00
Nicolas	bdbee963f7	Merge branch 'main' into nsc/cancel-job	2024-05-07 10:13:43 -07:00
rafaelsideguide	61d615c04b	Added tests	2024-05-07 14:03:00 -03:00
rafaelsideguide	e1f52c538f	nested includeHtml inside pageOptions	2024-05-07 13:40:24 -03:00
Nicolas	f46bf19fa5	Nick:	2024-05-07 09:26:52 -07:00
rafaelsideguide	83f3408634	Added max depth option	2024-05-07 11:06:26 -03:00
Nicolas	6d5da358cc	Nick: cancel job	2024-05-06 17:16:43 -07:00
rafaelsideguide	509250c4ef	changed to includeHtml	2024-05-06 19:45:56 -03:00
rafaelsideguide	538355f1af	Added toMarkdown option	2024-05-06 11:36:44 -03:00
Nicolas	15b774e974	Update index.ts	2024-05-04 12:44:30 -07:00
Nicolas	2aa09a3000	Nick: partial docs working, cleaner	2024-05-04 12:30:12 -07:00
Nicolas	00373228fa	Update index.ts	2024-05-04 11:53:16 -07:00
Nicolas	cbd9e88b77	Merge branch 'main' into llm-extraction	2024-04-30 14:49:20 -07:00
Nicolas	4f526cff92	Nick: cleanup	2024-04-30 12:19:43 -07:00
Caleb Peffer	3ca9e5153f	Caleb: trying to get loggin workng	2024-04-30 09:20:15 -07:00
rafaelsideguide	d3c36adaa7	Update index.ts	2024-04-29 17:58:47 -03:00
Caleb Peffer	2ad7a58eb7	Caleb: first test passing	2024-04-28 17:38:20 -07:00
Caleb Peffer	06497729e2	Caleb: got it to a testable state I believe	2024-04-28 15:52:09 -07:00
Rafael Miller	f189589da4	Merge pull request #34 from mendableai/nsc/returnOnlyUrls ... Implements the ability for the crawler to output all the links it found, without scraping	2024-04-24 10:34:42 -03:00
rafaelsideguide	942ac3b41c	Resolved merge conflicts between feat/added-anthropic-vision-api and main	2024-04-24 09:57:45 -03:00
Nicolas	ddf9ff9c9a	Nick:	2024-04-20 11:46:06 -07:00
Ikko Eltociear Ashimine	9e9d66f7a3	refactor: fix typo in WebScraper/index.ts ... breakign -> breaking	2024-04-20 02:27:53 +09:00
rafaelsideguide	72e1dadccd	adding option to replace all relative paths with absolute paths	2024-04-19 11:47:20 -03:00
rafaelsideguide	c4cc4b9262	fixing document response	2024-04-18 14:12:39 -03:00
Rafael Miller	704a059448	Update index.ts	2024-04-18 13:53:11 -03:00
rafaelsideguide	57e5b36014	[Feat] Adding pdf parser	2024-04-18 11:43:57 -03:00
Nicolas	36abe0f7f9	Nick:	2024-04-17 18:24:46 -07:00
Nicolas	52fb28bc1a	Update index.ts	2024-04-17 12:52:15 -07:00
Nicolas	de439f6529	Update index.ts	2024-04-17 12:51:29 -07:00
Nicolas	871d5d91b0	Update index.ts	2024-04-17 12:51:12 -07:00
rafaelsideguide	b375ce3e39	adding unit tests and bugfixing	2024-04-17 14:54:54 -03:00
Nicolas	27674a624d	Update index.ts	2024-04-17 10:39:00 -07:00
rafaelsideguide	00941d94a4	Added anthropic vision to getImageDescription function	2024-04-16 18:03:48 -03:00