Gergő Móricz
129b10e478
fix(llmExtract): cost calculation
2025-04-16 01:04:48 -07:00
Gergő Móricz
0d813b628b
feat: correlate smart scrape
2025-04-16 00:25:42 -07:00
Gergő Móricz
edd4c30908
FIX IT
2025-04-16 00:11:27 -07:00
Gergő Móricz
a06910115b
asd
2025-04-16 00:03:07 -07:00
Gergő Móricz
2193bee133
Improve logging
2025-04-15 23:52:20 -07:00
Gergő Móricz
0935ec210e
feat(smartScrape): better loggin
2025-04-15 23:34:00 -07:00
Gergő Móricz
b6abe4f26b
fix(smartScrape): pass extract id
2025-04-15 23:23:18 -07:00
Gergő Móricz
80b507e64e
correlate with eid
2025-04-15 23:06:13 -07:00
Gergő Móricz
13bd50ad2b
feat(fetch): don't time out (for smart scrape)
2025-04-15 21:57:05 -07:00
rafaelmmiller
252a9ccc89
Refactor robustFetch logging to exclude sensitive parameters and improve error handling.
2025-04-15 17:15:01 -07:00
Gergő Móricz
edb40d75c1
log session ID
2025-04-15 01:46:31 -07:00
Nicolas
6634d236bf
(feat/fire-1) FIRE-1 ( #1462 )
...
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459 )
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-15 00:19:45 -07:00
Gergő Móricz
2857496356
feat(diff): better rpc (revert this if broken)
2025-04-13 22:45:58 -07:00
Nicolas
723e7b7c60
Update diff.ts
2025-04-13 11:26:30 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes ( #1445 )
...
* Add git-diff support to change tracking format
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix type issues with parse-diff library
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix parse-diff type definitions to match actual library structure
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Add structured output/prompt support to change tracking
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* (feat/change-tracking) Change Tracking Modes (#1447 )
* Refactor change tracking to use modes array instead of separate formats
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Implement schema-based change tracking with old/new value comparison
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Nick:
* Nick: .json
* Update diff.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Update index.ts
* Update types.ts
* Update diff.ts
* Update scrape.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Gergő Móricz
415603acb0
fixes
2025-04-10 12:50:03 +02:00
Gergő Móricz
4294face78
feat(scrapeURL): reintroduce default timeout for simple queries ( #1440 )
...
* feat(scrapeURL): reintroduce default timeout
* fix
* adjust timeouts
2025-04-10 12:29:54 +02:00
devin-ai-integration[bot]
8c801ed956
Rename 'compare' format and property to 'changeTracking' ( #1423 )
2025-04-08 21:09:31 +02:00
Gergő Móricz
f5e5bdb710
fix(llmExtract): arbitrary objects caused error to be thrown
2025-04-05 15:48:47 +02:00
Gergő Móricz
2e2c3d52ce
feat: add swoogo classes to force include main tags
2025-04-03 09:57:19 +02:00
Gergő Móricz
24f5199359
compare format (FIR-1560) ( #1405 )
2025-04-02 19:52:43 +02:00
Gergő Móricz
46048bc94d
feat(scrapeURL): return js returns from f-e (FIR-1535) ( #1385 )
...
* feat(scrapeURL): return js returns from f-e
* feat(js-sdk): handle new results
2025-03-28 12:42:25 +01:00
Gergő Móricz
c3ebfafba7
fix(llmExtract): remove unsupported JSON schema properties ( #1335 )
2025-03-13 20:10:39 +01:00
Nicolas
c6cad942ab
Nick: errors -> warn
2025-03-13 13:07:51 -04:00
Gergő Móricz
4f25f12a12
fix(ai): handle if AI returns a JSON code block ( #1280 )
2025-03-02 15:25:24 -03:00
Nicolas
289e351c14
(feat/deep-research-alpha) Added Max Urls, Sources and Fixes ( #1271 )
...
* Nick: fixes
* Nick:
* Update deep-research-status.ts
2025-02-27 13:24:40 -03:00
Grass Huang
7bf04d409a
fix(scraper): improve charset detection regex to accurately parse meta tags ( #1265 )
2025-02-26 17:31:06 +01:00
Nicolas
5ab86b8b43
(fix/token-slicer) Fixes extract token limit issues ( #1236 )
...
* Nick: fixes extract token limit errors
* Update llmExtract.ts
* Update llmExtract.ts
2025-02-21 20:44:42 +01:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK ( #1220 )
...
* Nick: init
* Update llmExtract.ts
* Update llmExtract.ts
* Nick rename
* fix(v1/types): extract json schema validation
* Update url-processor.ts
* feat(ai-sdk): ollama support
* feat(ai-sdk): further ollama support
* Nick: it is broken btw
* feat(ai-sdk): abstract model adapter
* Update pnpm-lock.yaml
* Update analyzeSchemaAndPrompt.ts
* Nick:
* feat(ai-sdk): ollama support
* doc(SELF_HOST): update with embedding param
* Nick:
* Update ranker.ts
* Nick:
* feat(ai-sdk): fixes
* Update llmExtract.ts
* feat: remove zod-to-json-schema
* fix
* Update llmExtract.ts
* use openai
* fixes
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 22:48:58 +01:00
Gergő Móricz
283a3bfef3
fix(scrapeURL/engines/fetch): discover charset and re-decode ( #1221 )
...
* fix(scrapeURL/engines/fetch): discover charset and re-decode
* fix(snips/scrape): allow more time for stealth proxy
2025-02-20 18:56:15 +01:00
Gergő Móricz
e417f83c28
feat(self-host): ollama support ( #1219 )
2025-02-20 16:59:19 +01:00
Gergő Móricz
c38dcd0432
feat(self-host): proxy support (FIR-1111) ( #1212 )
...
* feat(self-host): proxy support
* fix(playwright-service-ts): return untreated text/plain
2025-02-20 14:20:03 +01:00
Gergő Móricz
da1670b78c
feat(map): mock support (FIR-1109) ( #1213 )
...
* feat(map,fetch): mock support
* feat(snips/map): mock out long-running test
* fix(snips/scrape): use more reliable site for adblock testing
2025-02-20 10:41:43 +01:00
Gergő Móricz
11ed679274
feat(scrapeURL/pdf): support PDF prefetch when parsePDF is off
2025-02-20 09:28:13 +01:00
Gergő Móricz
55d047b6b3
feat(scrapeURL): handle PDFs behind anti-bot ( #1198 )
2025-02-20 04:11:30 +01:00
Gergő Móricz
46b187bc64
feat(v1/map): stop mapping if timed out via AbortController ( #1205 )
2025-02-20 00:42:13 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES ( #1207 )
...
* fix(extract): construct OpenAI on demand
Fixes hard-crash if api key not specified in a self-hosting environment.
* fix(ci): try sleeping
* fix(ci): override host
* fix(ci): wait for server to start
* Support /extract and /crawl for self-hosted (FIR-1097) (#1137 )
* Support /extract for self-hosted
This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)
* Use getJob for extract and use correct types
* fix(v1/crawl-status): only poll DB for total count if DB is enabled
* feat(snips): TEST_SUITE_SELF_HOSTED
* fix(ci/test-server-self-host): use pr trigger
* fix(scrapeURL): f-e mocking in selfhosted env
* fix(snips): do not try to eval json format on selfhost
* fix(scrapeURL): further f-e mocking
* fix(snips): don't timeout on hard fail polling
* fix(v1/extract-status): fix-up the db-agnostic impl
unfortunately had to separate the functions since the schema
was too divergent :(
* fix(snips): boost screenshot delay
* feat(ci): test with openai
* feat(ci): extract, search testing
* fix(ci): matrix
* fix(ci): bleh
* Update: fix default google search (#1174 )
* fix log title
* search should always work
* asd
* fix ci
---------
Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Nicolas
5c47e97db2
(feat/deep-research) Alpha implementation of deep research ( #1202 )
...
* Nick:
* Revert "fix(v1/types): fix extract -> json rename (FIR-1072) (#1195 )"
This reverts commit 586a10f40d354a038afc2b67809f20a7a829f8cb.
* Update deep-research-service.ts
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Nick:
* Update deep-research-service.ts
* Nick:
* Update deep-research-service.ts
* Apply suggestions from code review
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-19 12:44:21 -03:00
Gergő Móricz
b136e42b53
feat(v1): proxy option / stealthProxy flag (FIR-1050) ( #1196 )
...
* feat(v1): proxy option / stealthProxy flag
* feat(js-sdk): add proxy option
2025-02-18 18:03:10 +01:00
Gergő Móricz
5c4021e8cf
fix(scrapeURL/fire-engine): screenshot broken hotfix
2025-02-17 18:59:16 +01:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions ( #1192 )
2025-02-17 12:35:55 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) ( #1183 )
...
* fix(scrapeURL/scrapingbee): enforce timeout
* fix(scrapeURL/sb): types
* fix the test
* fixup: remove nix files
2025-02-16 15:55:03 +01:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
6bfd24d903
Nick: waitFor fixes
2025-01-30 23:23:03 -03:00
Nicolas
fa99c62f64
(feat/extract) Improved completions to use model's limits ( #1109 )
...
* Update analyzeSchemaAndPrompt.ts
* Nick: fixes FIR-663
* Update llmExtract.ts
* Update llmExtract.ts
2025-01-29 12:37:14 -03:00
Gergő Móricz
d09e0603f8
feat(scrapeUrl/fire-engine): add blockAds flag (FIR-692) ( #1106 )
...
* feat(scrapeUrl/fire-engine): add blockAds flag
* feat(v1/scrape): blockAds test
2025-01-29 15:03:37 +01:00
Gergő Móricz
5733b82e9d
fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) ( #1104 )
...
* feat(location/country): default to us-generic
* add tests + fix mock
2025-01-29 08:23:36 +01:00
Móricz Gergő
a2d94b525f
feat: rewrite html transformer in rust
2025-01-25 09:41:33 +01:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust ( #1089 )
2025-01-24 22:04:54 +01:00
Móricz Gergő
bee2b2873e
fix(sitemap): better ordering
2025-01-23 08:58:18 +01:00
