Gergő Móricz
46048bc94d
feat(scrapeURL): return js returns from f-e (FIR-1535) ( #1385 )
...
* feat(scrapeURL): return js returns from f-e
* feat(js-sdk): handle new results
2025-03-28 12:42:25 +01:00
Gergő Móricz
c3ebfafba7
fix(llmExtract): remove unsupported JSON schema properties ( #1335 )
2025-03-13 20:10:39 +01:00
Nicolas
c6cad942ab
Nick: errors -> warn
2025-03-13 13:07:51 -04:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block ( #1280 )
2025-03-02 15:25:24 -03:00
Nicolas
289e351c14
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes ( #1271 )
...
* Nick: fixes
* Nick:
* Update deep-research-status.ts
2025-02-27 13:24:40 -03:00
Grass Huang
7bf04d409a
fix(scraper): improve charset detection regex to accurately parse meta tags ( #1265 )
2025-02-26 17:31:06 +01:00
Nicolas
5ab86b8b43
(fix/token-slicer) Fixes extract token limit issues ( #1236 )
...
* Nick: fixes extract token limit errors
* Update llmExtract.ts
* Update llmExtract.ts
2025-02-21 20:44:42 +01:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK ( #1220 )
...
* Nick: init
* Update llmExtract.ts
* Update llmExtract.ts
* Nick rename
* fix(v1/types): extract json schema validation
* Update url-processor.ts
* feat(ai-sdk): ollama support
* feat(ai-sdk): further ollama support
* Nick: it is broken btw
* feat(ai-sdk): abstract model adapter
* Update pnpm-lock.yaml
* Update analyzeSchemaAndPrompt.ts
* Nick:
* feat(ai-sdk): ollama support
* doc(SELF_HOST): update with embedding param
* Nick:
* Update ranker.ts
* Nick:
* feat(ai-sdk): fixes
* Update llmExtract.ts
* feat: remove zod-to-json-schema
* fix
* Update llmExtract.ts
* use openai
* fixes
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 22:48:58 +01:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode ( #1221 )
...
* fix(scrapeURL/engines/fetch): discover charset and re-decode
* fix(snips/scrape): allow more time for stealth proxy
2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support ( #1219 )
2025-02-20 16:59:19 +01:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) ( #1212 )
...
* feat(self-host): proxy support
* fix(playwright-service-ts): return untreated text/plain
2025-02-20 14:20:03 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) ( #1213 )
...
* feat(map,fetch): mock support
* feat(snips/map): mock out long-running test
* fix(snips/scrape): use more reliable site for adblock testing
2025-02-20 10:41:43 +01:00
Gergő Móricz
11ed679274
feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off
2025-02-20 09:28:13 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot ( #1198 )
2025-02-20 04:11:30 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController ( #1205 )
2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES ( #1207 )
...
* fix(extract): construct OpenAI on demand
Fixes hard-crash if api key not specified in a self-hosting environment.
* fix(ci): try sleeping
* fix(ci): override host
* fix(ci): wait for server to start
* Support /extract and /crawl for self-hosted (FIR-1097) (#1137 )
* Support /extract for self-hosted
This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)
* Use getJob for extract and use correct types
* fix(v1/crawl-status): only poll DB for total count if DB is enabled
* feat(snips): TEST_SUITE_SELF_HOSTED
* fix(ci/test-server-self-host): use pr trigger
* fix(scrapeURL): f-e mocking in selfhosted env
* fix(snips): do not try to eval json format on selfhost
* fix(scrapeURL): further f-e mocking
* fix(snips): don't timeout on hard fail polling
* fix(v1/extract-status): fix-up the db-agnostic impl
unfortunately had to separate the functions since the schema
was too divergent :(
* fix(snips): boost screenshot delay
* feat(ci): test with openai
* feat(ci): extract, search testing
* fix(ci): matrix
* fix(ci): bleh
* Update: fix default google search (#1174 )
* fix log title
* search should always work
* asd
* fix ci
---------
Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research ( #1202 )
...
* Nick:
* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195 )"
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
* Update deep-research-service.ts
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Update deep-research-service.ts
* Nick:
* Update deep-research-service.ts
* Apply suggestions from code review
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-19 12:44:21 -03:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) ( #1196 )
...
* feat(v1): proxy option / stealthProxy flag
* feat(js-sdk): add proxy option
2025-02-18 18:03:10 +01:00
Gergő Móricz
5c4021e8cf
fix(scrapeURL/fire-engine): screenshot broken hotfix
2025-02-17 18:59:16 +01:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions ( #1192 )
2025-02-17 12:35:55 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) ( #1183 )
...
* fix(scrapeURL/scrapingbee): enforce timeout
* fix(scrapeURL/sb): types
* fix the test
* fixup: remove nix files
2025-02-16 15:55:03 +01:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
6bfd24d903
Nick: waitFor fixes
2025-01-30 23:23:03 -03:00
Nicolas
fa99c62f64
(feat/extract) Improved completions to use model's limits ( #1109 )
...
* Update analyzeSchemaAndPrompt.ts
* Nick: fixes FIR-663
* Update llmExtract.ts
* Update llmExtract.ts
2025-01-29 12:37:14 -03:00
Gergő Móricz
d09e0603f8
feat(scrapeUrl/fire-engine): add blockAds flag (FIR-692) ( #1106 )
...
* feat(scrapeUrl/fire-engine): add blockAds flag
* feat(v1/scrape): blockAds test
2025-01-29 15:03:37 +01:00
Gergő Móricz
5733b82e9d
fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) ( #1104 )
...
* feat(location/country): default to us-generic
* add tests + fix mock
2025-01-29 08:23:36 +01:00
Móricz Gergő
a2d94b525f
feat: rewrite html transformer in rust
2025-01-25 09:41:33 +01:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust ( #1089 )
2025-01-24 22:04:54 +01:00
Móricz Gergő
bee2b2873e
fix(sitemap): better ordering
2025-01-23 08:58:18 +01:00
Nicolas
498558d358
Nick: formatting done
2025-01-22 18:47:44 -03:00
Nicolas
994e1eb502
Nick: rm logs
2025-01-22 17:27:48 -03:00
Nicolas
56f048aeff
Reapply "Nick:"
...
This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.
2025-01-22 17:26:32 -03:00
Nicolas
4b4385c520
Revert "Nick:"
...
This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.
2025-01-22 17:26:09 -03:00
Nicolas
6718ce8908
Nick:
2025-01-22 17:25:48 -03:00
Nicolas
04916f17e2
Nick: bug fixes + acuc fixes + cache fixes
2025-01-21 19:17:06 -03:00
Nicolas
1ca50e6e8f
Update llmExtract.ts
2025-01-19 22:18:51 -03:00
Nicolas
d786949639
Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
...
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
2025-01-19 22:04:12 -03:00
Nicolas
8b17af4001
Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
...
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
2025-01-19 22:00:28 -03:00
Nicolas
406f28c04a
Merge pull request #1068 from mendableai/nsc/llm-usage-extract
...
(feat/extract) - LLMs usage analysis + billing
2025-01-19 21:36:33 -03:00
Nicolas
81c347f538
Update llmExtract.ts
2025-01-18 22:49:03 -03:00
Nicolas
34b40f6a23
Nick:
2025-01-18 17:17:42 -03:00
Nicolas
260a726f37
Merge branch 'main' into nsc/llm-usage-extract
2025-01-17 23:02:12 -03:00
Móricz Gergő
faf58dfca7
fix(removeUnwantedElements): post-includeTags excludeTags
...
Fixes #700
2025-01-17 12:41:00 +01:00
Gergő Móricz
6c94db7ed0
fix(html,markdown): always get absolute links
2025-01-16 16:56:13 +01:00
Gergő Móricz
e824303d87
feat(html): always pick largest image from srcset
2025-01-16 16:51:33 +01:00
Nicolas
4db023280d
Nick: introduce llm-usage cost analysis
2025-01-15 21:01:29 -03:00
Nicolas
957eea4113
Nick: extract without a schema should work as expected
2025-01-14 11:37:00 -03:00
Nicolas
033e9bbf29
Nick: __experimental_streamSteps
2025-01-14 01:45:50 -03:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction ( #1061 )
...
* agent that decides if splits schema or not
* split and merge properties done
* wip
* wip
* changes
* ch
* array merge working!
* comment
* wip
* dereferentiate schema
* dereference schemas
* Nick: new re-ranker
* Create llm-links.txt
* Nick: format
* Update extraction-service.ts
* wip: cooking schema mix and spread functions
* wip
* wip getting there!!!
* nick:
* moved functions to helpers
* nick:
* cant reproduce the error anymore
* error handling all scrapes failed
* fix
* Nick: added the sitemap index
* Update sitemap-index.ts
* Update map.ts
* deduplicate and merge arrays
* added error handler for object transformations
* Update url-processor.ts
* Nick:
* Nick: fixes
* Nick: big improvements to rerank of multi-entity
* Nick: working
* Update reranker.ts
* fixed transformations for nested objs
* fix merge nulls
* Nick: fixed error piping
* Update queue-worker.ts
* Update extraction-service.ts
* Nick: format
* Update queue-worker.ts
* Update pnpm-lock.yaml
* Update queue-worker.ts
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
2025-01-13 22:30:15 -03:00
Gergő Móricz
5c62bb1195
feat: new snips test framework (FIR-414) ( #1033 )
...
* feat: new snips test framework
* Update mock.ts
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-01-13 20:50:47 +01:00
