17 Commits

Author SHA1 Message Date
Nicolas
2c391b0105 Nick: 2025-01-24 18:09:25 -03:00
rafaelmmiller
64d116540f rerank with lower threshold + back to map if lenght = 0 2025-01-24 09:08:16 -03:00
Nicolas
498558d358 Nick: formatting done 2025-01-22 18:47:44 -03:00
Nicolas
56f048aeff Reapply "Nick:"
This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.
2025-01-22 17:26:32 -03:00
Nicolas
4b4385c520 Revert "Nick:"
This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.
2025-01-22 17:26:09 -03:00
Nicolas
6718ce8908 Nick: 2025-01-22 17:25:48 -03:00
Nicolas
2b9f63cf10 Nick: more permissive re-ranker 2025-01-21 11:30:54 -03:00
Nicolas
d786949639 Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
2025-01-19 22:04:12 -03:00
Nicolas
8b17af4001 Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
2025-01-19 22:00:28 -03:00
Nicolas
4db023280d Nick: introduce llm-usage cost analysis 2025-01-15 21:01:29 -03:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction (#1061)
* agent that decides if splits schema or not

* split and merge properties done

* wip

* wip

* changes

* ch

* array merge working!

* comment

* wip

* dereferentiate schema

* dereference schemas

* Nick: new re-ranker

* Create llm-links.txt

* Nick: format

* Update extraction-service.ts

* wip: cooking schema mix and spread functions

* wip

* wip getting there!!!

* nick:

* moved functions to helpers

* nick:

* cant reproduce the error anymore

* error handling all scrapes failed

* fix

* Nick: added the sitemap index

* Update sitemap-index.ts

* Update map.ts

* deduplicate and merge arrays

* added error handler for object transformations

* Update url-processor.ts

* Nick:

* Nick: fixes

* Nick: big improvements to rerank of multi-entity

* Nick: working

* Update reranker.ts

* fixed transformations for nested objs

* fix merge nulls

* Nick: fixed error piping

* Update queue-worker.ts

* Update extraction-service.ts

* Nick: format

* Update queue-worker.ts

* Update pnpm-lock.yaml

* Update queue-worker.ts

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
2025-01-13 22:30:15 -03:00
Nicolas
f4d10c5031 Nick: formatting fixes 2025-01-10 18:35:10 -03:00
Nicolas
c3fd13a82b Nick: fixed re-ranker and enabled url cache of 2hrs 2024-12-31 18:06:07 -03:00
Nicolas
233f347f5e Nick: refactor 2024-12-26 12:41:37 -03:00
Nicolas
8a1c404918 Nick: revert trailing comma 2024-12-11 19:51:08 -03:00
Nicolas
00335e2ba9 Nick: fixed prettier 2024-12-11 19:46:11 -03:00
Nicolas
78badf8f72 Nick: wip 2024-10-28 16:02:07 -03:00