159 Commits

Author SHA1 Message Date
Gergő Móricz
2857496356 feat(diff): better rpc (revert this if broken) 2025-04-13 22:45:58 -07:00
Nicolas
723e7b7c60 Update diff.ts 2025-04-13 11:26:30 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes (#1445)
* Add git-diff support to change tracking format

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix type issues with parse-diff library

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Fix parse-diff type definitions to match actual library structure

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Add structured output/prompt support to change tracking

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* (feat/change-tracking) Change Tracking Modes (#1447)

* Refactor change tracking to use modes array instead of separate formats

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Implement schema-based change tracking with old/new value comparison

Co-Authored-By: Nicolas Camara <nick@sideguide.dev>

* Nick:

* Nick: .json

* Update diff.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>

* Update index.ts

* Update types.ts

* Update diff.ts

* Update scrape.ts

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
415603acb0 fixes 2025-04-10 12:50:03 +02:00
Gergő Móricz
4294face78
feat(scrapeURL): reintroduce default timeout for simple queries (#1440)
* feat(scrapeURL): reintroduce default timeout

* fix

* adjust timeouts
2025-04-10 12:29:54 +02:00
devin-ai-integration[bot]
8c801ed956
Rename 'compare' format and property to 'changeTracking' (#1423) 2025-04-08 21:09:31 +02:00
Gergő Móricz
f5e5bdb710 fix(llmExtract): arbitrary objects caused error to be thrown 2025-04-05 15:48:47 +02:00
Gergő Móricz
2e2c3d52ce feat: add swoogo classes to force include main tags 2025-04-03 09:57:19 +02:00
Gergő Móricz
24f5199359
compare format (FIR-1560) (#1405) 2025-04-02 19:52:43 +02:00
Gergő Móricz
46048bc94d
feat(scrapeURL): return js returns from f-e (FIR-1535) (#1385)
* feat(scrapeURL): return js returns from f-e

* feat(js-sdk): handle new results
2025-03-28 12:42:25 +01:00
Gergő Móricz
c3ebfafba7
fix(llmExtract): remove unsupported JSON schema properties (#1335) 2025-03-13 20:10:39 +01:00
Nicolas
c6cad942ab Nick: errors -> warn 2025-03-13 13:07:51 -04:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block (#1280) 2025-03-02 15:25:24 -03:00
Nicolas
289e351c14
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes (#1271)
* Nick: fixes

* Nick:

* Update deep-research-status.ts
2025-02-27 13:24:40 -03:00
Grass Huang
7bf04d409a
fix(scraper): improve charset detection regex to accurately parse meta tags (#1265) 2025-02-26 17:31:06 +01:00
Nicolas
5ab86b8b43
(fix/token-slicer) Fixes extract token limit issues (#1236)
* Nick: fixes extract token limit errors

* Update llmExtract.ts

* Update llmExtract.ts
2025-02-21 20:44:42 +01:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK (#1220)
* Nick: init

* Update llmExtract.ts

* Update llmExtract.ts

* Nick rename

* fix(v1/types): extract json schema validation

* Update url-processor.ts

* feat(ai-sdk): ollama support

* feat(ai-sdk): further ollama support

* Nick: it is broken btw

* feat(ai-sdk): abstract model adapter

* Update pnpm-lock.yaml

* Update analyzeSchemaAndPrompt.ts

* Nick:

* feat(ai-sdk): ollama support

* doc(SELF_HOST): update with embedding param

* Nick:

* Update ranker.ts

* Nick:

* feat(ai-sdk): fixes

* Update llmExtract.ts

* feat: remove zod-to-json-schema

* fix

* Update llmExtract.ts

* use openai

* fixes

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 22:48:58 +01:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode (#1221)
* fix(scrapeURL/engines/fetch): discover charset and re-decode

* fix(snips/scrape): allow more time for stealth proxy
2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support (#1219) 2025-02-20 16:59:19 +01:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) (#1212)
* feat(self-host): proxy support

* fix(playwright-service-ts): return untreated text/plain
2025-02-20 14:20:03 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) (#1213)
* feat(map,fetch): mock support

* feat(snips/map): mock out long-running test

* fix(snips/scrape): use more reliable site for adblock testing
2025-02-20 10:41:43 +01:00
Gergő Móricz
11ed679274 feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off 2025-02-20 09:28:13 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot (#1198) 2025-02-20 04:11:30 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController (#1205) 2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES (#1207)
* fix(extract): construct OpenAI on demand

Fixes hard-crash if api key not specified in a self-hosting environment.

* fix(ci): try sleeping

* fix(ci): override host

* fix(ci): wait for server to start

* Support /extract and /crawl for self-hosted (FIR-1097) (#1137)

* Support /extract for self-hosted

This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)

* Use getJob for extract and use correct types

* fix(v1/crawl-status): only poll DB for total count if DB is enabled

* feat(snips): TEST_SUITE_SELF_HOSTED

* fix(ci/test-server-self-host): use pr trigger

* fix(scrapeURL): f-e mocking in selfhosted env

* fix(snips): do not try to eval json format on selfhost

* fix(scrapeURL): further f-e mocking

* fix(snips): don't timeout on hard fail polling

* fix(v1/extract-status): fix-up the db-agnostic impl

unfortunately had to separate the functions since the schema
was too divergent :(

* fix(snips): boost screenshot delay

* feat(ci): test with openai

* feat(ci): extract, search testing

* fix(ci): matrix

* fix(ci): bleh

* Update: fix default google search (#1174)

* fix log title

* search should always work

* asd

* fix ci

---------

Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research (#1202)
* Nick:

* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195)"

This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.

* Update deep-research-service.ts

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Nick:

* Update deep-research-service.ts

* Nick:

* Update deep-research-service.ts

* Apply suggestions from code review

---------

Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-19 12:44:21 -03:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) (#1196)
* feat(v1): proxy option / stealthProxy flag

* feat(js-sdk): add proxy option
2025-02-18 18:03:10 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183)
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
2025-02-16 15:55:03 +01:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138)
* fix(scrape): allow getting valid JSON via rawHtml

* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
6bfd24d903 Nick: waitFor fixes 2025-01-30 23:23:03 -03:00
Nicolas
fa99c62f64
(feat/extract) Improved completions to use model's limits (#1109)
* Update analyzeSchemaAndPrompt.ts

* Nick: fixes FIR-663

* Update llmExtract.ts

* Update llmExtract.ts
2025-01-29 12:37:14 -03:00
Gergő Móricz
d09e0603f8
feat(scrapeUrl/fire-engine): add blockAds flag (FIR-692) (#1106)
* feat(scrapeUrl/fire-engine): add blockAds flag

* feat(v1/scrape): blockAds test
2025-01-29 15:03:37 +01:00
Gergő Móricz
5733b82e9d
fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) (#1104)
* feat(location/country): default to us-generic

* add tests + fix mock
2025-01-29 08:23:36 +01:00
Móricz Gergő
a2d94b525f feat: rewrite html transformer in rust 2025-01-25 09:41:33 +01:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust (#1089) 2025-01-24 22:04:54 +01:00
Móricz Gergő
bee2b2873e fix(sitemap): better ordering 2025-01-23 08:58:18 +01:00
Nicolas
498558d358 Nick: formatting done 2025-01-22 18:47:44 -03:00
Nicolas
994e1eb502 Nick: rm logs 2025-01-22 17:27:48 -03:00
Nicolas
56f048aeff Reapply "Nick:"
This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.
2025-01-22 17:26:32 -03:00
Nicolas
4b4385c520 Revert "Nick:"
This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.
2025-01-22 17:26:09 -03:00
Nicolas
6718ce8908 Nick: 2025-01-22 17:25:48 -03:00
Nicolas
04916f17e2 Nick: bug fixes + acuc fixes + cache fixes 2025-01-21 19:17:06 -03:00
Nicolas
1ca50e6e8f Update llmExtract.ts 2025-01-19 22:18:51 -03:00
Nicolas
d786949639 Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
2025-01-19 22:04:12 -03:00
Nicolas
8b17af4001 Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract"
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
2025-01-19 22:00:28 -03:00
Nicolas
406f28c04a
Merge pull request #1068 from mendableai/nsc/llm-usage-extract
(feat/extract) - LLMs usage analysis + billing
2025-01-19 21:36:33 -03:00
Nicolas
81c347f538 Update llmExtract.ts 2025-01-18 22:49:03 -03:00
Nicolas
34b40f6a23 Nick: 2025-01-18 17:17:42 -03:00