Gergő Móricz
7a8be13220
remove indexes that are no longer used
2025-06-02 22:09:55 +02:00
Nicolas
1c421f2d74
Nick: ( #1492 )
2025-04-22 21:42:37 -04:00
Nicolas
6634d236bf
(feat/fire-1) FIRE-1 ( #1462 )
...
* wip
* integrating smart-scrape
* integrate smartscrape into llmExtract
* wip
* smart scrape multiple links
* fixes
* fix
* wip
* it worked!
* wip. there's a bug on the batchExtract TypeError: Converting circular structure to JSON
* wip
* retry model
* retry models
* feat/scrape+json+extract interfaces ready
* vertex -> googleapi
* fix/transformArrayToObject. required params on schema is still a bug
* change model
* o3-mini -> gemini
* Update extractSmartScrape.ts
* sessionId
* sessionId
* Nick: f-0 start
* Update extraction-service-f0.ts
* Update types.ts
* Nick:
* Update queue-worker.ts
* Nick: new interface
* rename analyzeSchemaAndPrompt -> F0
* refactor: rename agent ID to model in types and extract logic
* agent
* id->model
* id->model
* refactor: standardize agent model handling and validation across extraction logic
* livecast agent
* (feat/f1) sdks (#1459 )
* feat: add FIRE-1 agent support to Python and JavaScript SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add FIRE-1 agent support to scrape methods in both SDKs
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* feat: add prompt and sessionId to AgentOptions interface
Co-Authored-By: hello@sideguide.dev <hello@sideguide.dev>
* Update index.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* feat(v1): rate limits
* Update types.ts
* Update llmExtract.ts
* add cost tracking
* remove
* Update requests.http
* fix smart scrape cost calc
* log sm cost
* fix counts
* fix
* expose cost tracking
* models fix
* temp: skipLibcheck
* get rid of it
* fix ts
* dont skip lib check
* Update extractSmartScrape.ts
* Update queue-worker.ts
* Update smartScrape.ts
* Update requests.http
* fix(rate-limiter):
* types: fire-1 refine
* bill 150
* fix credits used on crawl
* ban from crawl
* route cost limit warning
* Update generic-ai.ts
* genres
* Update llmExtract.ts
* test server diff
* cletu
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
Co-authored-by: Ademílson F. Tonato <ademilsonft@outlook.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: hello@sideguide.dev <hello@sideguide.dev>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-04-15 00:19:45 -07:00
devin-ai-integration[bot]
138a9757ae
(feat/change-tracking) Change Tracking Modes ( #1445 )
...
* Add git-diff support to change tracking format
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix type issues with parse-diff library
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Fix parse-diff type definitions to match actual library structure
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Add structured output/prompt support to change tracking
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* (feat/change-tracking) Change Tracking Modes (#1447 )
* Refactor change tracking to use modes array instead of separate formats
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Implement schema-based change tracking with old/new value comparison
Co-Authored-By: Nicolas Camara <nick@sideguide.dev>
* Nick:
* Nick: .json
* Update diff.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
* Update index.ts
* Update types.ts
* Update diff.ts
* Update scrape.ts
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nicolas Camara <nick@sideguide.dev>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-04-12 16:38:56 -07:00
Nicolas
f52d6aab91
(feat/deep-research) Improvements to final analysis ( #1443 )
...
* Nick: fixes
* Update research-manager.ts
* Update research-manager.ts
2025-04-10 22:59:28 -07:00
Gergő Móricz
bd1c1b0012
feat(log_job): start saving jobs to GCS ( #1424 )
2025-04-08 19:28:21 +02:00
Nicolas
ee211132c8
Nick:
2025-04-01 21:06:27 +04:00
Gergő Móricz
76e1f29ae8
Update Dockerfile ( #1231 ) ( #1232 )
...
* Update Dockerfile (#1231 )
* Dockerfile: re-add prod-deps stage and fix copies
---------
Co-authored-by: Loris <loris.rion@gmail.com>
2025-02-21 17:44:01 +01:00
Nicolas
25d9bdb1f6
(feat/ai-sdk) Migrate to AI-SDK ( #1220 )
...
* Nick: init
* Update llmExtract.ts
* Update llmExtract.ts
* Nick rename
* fix(v1/types): extract json schema validation
* Update url-processor.ts
* feat(ai-sdk): ollama support
* feat(ai-sdk): further ollama support
* Nick: it is broken btw
* feat(ai-sdk): abstract model adapter
* Update pnpm-lock.yaml
* Update analyzeSchemaAndPrompt.ts
* Nick:
* feat(ai-sdk): ollama support
* doc(SELF_HOST): update with embedding param
* Nick:
* Update ranker.ts
* Nick:
* feat(ai-sdk): fixes
* Update llmExtract.ts
* feat: remove zod-to-json-schema
* fix
* Update llmExtract.ts
* use openai
* fixes
---------
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-20 22:48:58 +01:00
Gergő Móricz
5eb0235ccb
feat(apps/api): remove Sentry builds
2025-02-20 08:19:06 +01:00
Gergő Móricz
2200f084f3
SELFHOST FIXES ( #1207 )
...
* fix(extract): construct OpenAI on demand
Fixes hard-crash if api key not specified in a self-hosting environment.
* fix(ci): try sleeping
* fix(ci): override host
* fix(ci): wait for server to start
* Support /extract and /crawl for self-hosted (FIR-1097) (#1137 )
* Support /extract for self-hosted
This returns the job response from redis rather than supabase when db auth is disabled (self hosted mode)
* Use getJob for extract and use correct types
* fix(v1/crawl-status): only poll DB for total count if DB is enabled
* feat(snips): TEST_SUITE_SELF_HOSTED
* fix(ci/test-server-self-host): use pr trigger
* fix(scrapeURL): f-e mocking in selfhosted env
* fix(snips): do not try to eval json format on selfhost
* fix(scrapeURL): further f-e mocking
* fix(snips): don't timeout on hard fail polling
* fix(v1/extract-status): fix-up the db-agnostic impl
unfortunately had to separate the functions since the schema
was too divergent :(
* fix(snips): boost screenshot delay
* feat(ci): test with openai
* feat(ci): extract, search testing
* fix(ci): matrix
* fix(ci): bleh
* Update: fix default google search (#1174 )
* fix log title
* search should always work
* asd
* fix ci
---------
Co-authored-by: Nick Roth <nlr06886@gmail.com>
Co-authored-by: William <sdustusun@gmail.com>
2025-02-20 00:41:22 +01:00
Ademílson F. Tonato
24e8aaf6b5
feat(blocklist): Improve URL blocking with tldts parsing
2025-01-31 13:14:48 +00:00
Gergő Móricz
ce1fe6f06a
update bullmq
2025-01-24 18:56:03 +01:00
Nicolas
92b8d97be3
Nick:
2025-01-19 13:09:29 -03:00
Móricz Gergő
de08b37480
feat: adjust CI testing
2025-01-17 11:51:46 +01:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction ( #1061 )
...
* agent that decides if splits schema or not
* split and merge properties done
* wip
* wip
* changes
* ch
* array merge working!
* comment
* wip
* dereferentiate schema
* dereference schemas
* Nick: new re-ranker
* Create llm-links.txt
* Nick: format
* Update extraction-service.ts
* wip: cooking schema mix and spread functions
* wip
* wip getting there!!!
* nick:
* moved functions to helpers
* nick:
* cant reproduce the error anymore
* error handling all scrapes failed
* fix
* Nick: added the sitemap index
* Update sitemap-index.ts
* Update map.ts
* deduplicate and merge arrays
* added error handler for object transformations
* Update url-processor.ts
* Nick:
* Nick: fixes
* Nick: big improvements to rerank of multi-entity
* Nick: working
* Update reranker.ts
* fixed transformations for nested objs
* fix merge nulls
* Nick: fixed error piping
* Update queue-worker.ts
* Update extraction-service.ts
* Nick: format
* Update queue-worker.ts
* Update pnpm-lock.yaml
* Update queue-worker.ts
---------
Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
2025-01-13 22:30:15 -03:00
Gergő Móricz
5c62bb1195
feat: new snips test framework (FIR-414) ( #1033 )
...
* feat: new snips test framework
* Update mock.ts
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-01-13 20:50:47 +01:00
Nicolas
bf9d41d0b2
Nick: index exploration
2024-12-30 19:37:48 -03:00
Gergő Móricz
4d1f92f4c8
fix(scrapeURL/fetch): block loopback and link-local IPs
2024-12-29 17:35:14 +01:00
Nicolas
de57e7f4dd
Nick: from dependencies to dev-dependencies
2024-12-11 20:07:05 -03:00
Nicolas
52f2e733e2
Nick: fixes
2024-12-11 19:48:22 -03:00
Nicolas
00335e2ba9
Nick: fixed prettier
2024-12-11 19:46:11 -03:00
Nicolas
42922c68d6
Update package.json
2024-11-20 16:44:40 -08:00
Nicolas
a1c018fdb0
Merge branch 'main' into nsc/new-extract
2024-11-13 17:14:43 -05:00
Gergő Móricz
16e850288c
fix(scrapeURL/pdf,docx): ignore SSL when downloading PDF
2024-11-12 22:46:58 +01:00
Gergő Móricz
9ace2ad071
fix(scrapeURL/pdf): fix llamaparse upload
2024-11-12 20:55:14 +01:00
Nicolas
540d4e5b56
Nick:
2024-11-12 12:10:18 -05:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714 )
...
* feat: use strictNullChecking
* feat: switch logger to Winston
* feat(scrapeURL): first batch
* fix(scrapeURL): error swallow
* fix(scrapeURL): add timeout to EngineResultsTracker
* fix(scrapeURL): report unexpected error to sentry
* chore: remove unused modules
* feat(transfomers/coerce): warn when a format's response is missing
* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support
* (add note)
* feat(scrapeURL): wip readme
* feat(scrapeURL): LLM extract
* feat(scrapeURL): better warnings
* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot
* feat(scrapeURL): add forceEngine internal option
* feat(scrapeURL/engines): scrapingbee
* feat(scrapeURL/transformars): uploadScreenshot
* feat(scrapeURL): more intense tests
* bunch of stuff
* get rid of WebScraper (mostly)
* adapt batch scrape
* add staging deploy workflow
* fix yaml
* fix logger issues
* fix v1 test schema
* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions
* scrapeURL: v0 backwards compat
* logger fixes
* feat(scrapeurl): v0 returnOnlyUrls support
* fix(scrapeURL/v0): URL leniency
* fix(batch-scrape): ts non-nullable
* fix(scrapeURL/fire-engine/chromecdp): fix wait action
* fix(logger): remove error debug key
* feat(requests.http): use dotenv expression
* fix(scrapeURL/extractMetadata): extract custom metadata
* fix crawl option conversion
* feat(scrapeURL): Add retry logic to robustFetch
* fix(scrapeURL): crawl stuff
* fix(scrapeURL): LLM extract
* fix(scrapeURL/v0): search fix
* fix(tests/v0): grant larger response size to v0 crawl status
* feat(scrapeURL): basic fetch engine
* feat(scrapeURL): playwright engine
* feat(scrapeURL): add url-specific parameters
* Update readme and examples
* added e2e tests for most parameters. Still a few actions, location and iframes to be done.
* fixed type
* Nick:
* Update scrape.ts
* Update index.ts
* added actions and base64 check
* Nick: skipTls feature flag?
* 403
* todo
* todo
* fixes
* yeet headers from url specific params
* add warning when final engine has feature deficit
* expose engine results tracker for ScrapeEvents implementation
* ingest scrape events
* fixed some tests
* comment
* Update index.test.ts
* fixed rawHtml
* Update index.test.ts
* update comments
* move geolocation to global f-e option, fix removeBase64Images
* Nick:
* trim url-specific params
* Update index.ts
---------
Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2024-11-07 20:57:33 +01:00
Nicolas
78badf8f72
Nick: wip
2024-10-28 16:02:07 -03:00
Nicolas
975f0575b4
Nick: max retries with axios-retry
2024-09-27 12:58:57 -04:00
Gergő Móricz
4a623c084a
fix(fly): don't use Depot builders (doesn't work)
2024-09-24 10:50:30 +02:00
rafaelsideguide
291d9e375b
now using compiled go/C lib with koffi
2024-09-03 10:56:07 -03:00
Nicolas
41eb620959
Nick: prompt option, still need to convert to new structured outputs
2024-08-29 21:00:57 -03:00
Nicolas
f277a0e2bb
Update package.json
2024-08-26 18:17:09 -03:00
Gergő Móricz
e7f267b6fe
Merge branch 'main' into v1-webscraper
2024-08-23 17:21:54 +02:00
rafaelsideguide
5f60a55967
workflow and npm now running v1 tests
2024-08-22 15:28:49 -03:00
Gergő Móricz
920702cdde
Update builder to handle uploading sourcemaps
2024-08-21 19:08:03 +02:00
Gergő Móricz
86942728e3
Add metadata for queue-worker and Express
2024-08-21 17:58:27 +02:00
Nicolas
35decb1af2
Nick:
2024-08-21 12:35:03 -03:00
Gergő Móricz
eb84673b06
feat: crawl status websocket WIP
2024-08-17 01:04:14 +02:00
Nicolas
78ca94251c
Merge pull request #480 from mendableai/nsc/hyper-v81
...
Reduce metrics ingestion w/ HyperDX v0.8.1
2024-08-16 14:34:14 -04:00
Gergő Móricz
8b7569f8f3
add zod, create middleware, update openapi declaration, add crawl logic
2024-08-15 23:30:33 +02:00
Gergő Móricz
5fc7fcb77c
Merge branch 'main' into feat/queue-scrapes
2024-08-07 16:35:44 +02:00
Nicolas
e195ddbef4
Merge branch 'main' into nsc/hyper-v81
2024-08-05 20:47:39 -04:00
Nicolas
a28ecc1f61
Nick: caching
2024-07-30 18:59:35 -04:00
Nicolas
7e002a8b06
Nick: bull mq
2024-07-30 13:27:23 -04:00
Nicolas
c446942306
Nick:
2024-07-29 21:28:29 -04:00
Nicolas
968a2dc753
Nick:
2024-07-29 18:37:09 -04:00
Nicolas
04942bb9de
Nick:
2024-07-29 18:31:43 -04:00
Gergo Moricz
8160c311c0
fix queue stuck bug via lock setting changes
2024-07-17 21:31:25 +02:00
