Gergő Móricz
|
8546bcacc0
|
new cost tracking
|
2025-04-17 09:23:53 -07:00 |
|
Nicolas
|
6634d236bf
|
(feat/fire-1) FIRE-1 (#1462)
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459)
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
|
2025-04-15 00:19:45 -07:00 |
|
Nicolas
|
25d9bdb1f6
|
(feat/ai-sdk) Migrate to AI-SDK (#1220)
* Nick: init
* Update llmExtract.ts
* Update llmExtract.ts
* Nick rename
* fix(v1/types): extract json schema validation
* Update url-processor.ts
* feat(ai-sdk): ollama support
* feat(ai-sdk): further ollama support
* Nick: it is broken btw
* feat(ai-sdk): abstract model adapter
* Update pnpm-lock.yaml
* Update analyzeSchemaAndPrompt.ts
* Nick:
* feat(ai-sdk): ollama support
* doc(SELF_HOST): update with embedding param
* Nick:
* Update ranker.ts
* Nick:
* feat(ai-sdk): fixes
* Update llmExtract.ts
* feat: remove zod-to-json-schema
* fix
* Update llmExtract.ts
* use openai
* fixes
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
|
2025-02-20 22:48:58 +01:00 |
|
Nicolas
|
10133adcc6
|
Update reranker.ts
|
2025-01-24 18:35:36 -03:00 |
|
Nicolas
|
2c391b0105
|
Nick:
|
2025-01-24 18:09:25 -03:00 |
|
rafaelmmiller
|
64d116540f
|
rerank with lower threshold + back to map if lenght = 0
|
2025-01-24 09:08:16 -03:00 |
|
Nicolas
|
498558d358
|
Nick: formatting done
|
2025-01-22 18:47:44 -03:00 |
|
Nicolas
|
56f048aeff
|
Reapply "Nick:"
This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.
|
2025-01-22 17:26:32 -03:00 |
|
Nicolas
|
4b4385c520
|
Revert "Nick:"
This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.
|
2025-01-22 17:26:09 -03:00 |
|
Nicolas
|
6718ce8908
|
Nick:
|
2025-01-22 17:25:48 -03:00 |
|
Nicolas
|
2b9f63cf10
|
Nick: more permissive re-ranker
|
2025-01-21 11:30:54 -03:00 |
|
Nicolas
|
d786949639
|
Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
|
2025-01-19 22:04:12 -03:00 |
|
Nicolas
|
8b17af4001
|
Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
|
2025-01-19 22:00:28 -03:00 |
|
Nicolas
|
4db023280d
|
Nick: introduce llm-usage cost analysis
|
2025-01-15 21:01:29 -03:00 |
|
Nicolas
|
5e5b5ee0e2
|
(feat/extract) New re-ranker + multi entity extraction (#1061)
* agent that decides if splits schema or not
* split and merge properties done
* wip
* wip
* changes
* ch
* array merge working!
* comment
* wip
* dereferentiate schema
* dereference schemas
* Nick: new re-ranker
* Create llm-links.txt
* Nick: format
* Update extraction-service.ts
* wip: cooking schema mix and spread functions
* wip
* wip getting there!!!
* nick:
* moved functions to helpers
* nick:
* cant reproduce the error anymore
* error handling all scrapes failed
* fix
* Nick: added the sitemap index
* Update sitemap-index.ts
* Update map.ts
* deduplicate and merge arrays
* added error handler for object transformations
* Update url-processor.ts
* Nick:
* Nick: fixes
* Nick: big improvements to rerank of multi-entity
* Nick: working
* Update reranker.ts
* fixed transformations for nested objs
* fix merge nulls
* Nick: fixed error piping
* Update queue-worker.ts
* Update extraction-service.ts
* Nick: format
* Update queue-worker.ts
* Update pnpm-lock.yaml
* Update queue-worker.ts
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
|
2025-01-13 22:30:15 -03:00 |
|
Nicolas
|
f4d10c5031
|
Nick: formatting fixes
|
2025-01-10 18:35:10 -03:00 |
|
Nicolas
|
c3fd13a82b
|
Nick: fixed re-ranker and enabled url cache of 2hrs
|
2024-12-31 18:06:07 -03:00 |
|
Nicolas
|
233f347f5e
|
Nick: refactor
|
2024-12-26 12:41:37 -03:00 |
|
Nicolas
|
8a1c404918
|
Nick: revert trailing comma
|
2024-12-11 19:51:08 -03:00 |
|
Nicolas
|
00335e2ba9
|
Nick: fixed prettier
|
2024-12-11 19:46:11 -03:00 |
|
Nicolas
|
78badf8f72
|
Nick: wip
|
2024-10-28 16:02:07 -03:00 |
|