3537 Commits

Author SHA1 Message Date
Thomas Kosmas
f043f5fd61 Enhance error handling in E2E tests and introduce CrawlWatcher tests
- Updated error messages in E2E tests to provide clearer feedback for blocked URLs and invalid API keys.
- Added new test suite for CrawlWatcher to ensure proper instantiation and error handling when WebSocket is unavailable.
- Improved test conditions for URL scraping and crawling to reflect updated error responses.
2024-12-21 02:27:22 +02:00
Nicolas
818f5544cb
Merge pull request #1001 from mendableai/nsc/block-list-string
Refactored Blocklist Error Messages
2024-12-20 18:11:01 -03:00
Nicolas
d1f3e26f9e Nick: blocklist string 2024-12-20 18:09:49 -03:00
Nicolas
8e947344ad
Merge pull request #999 from mendableai/nsc/credit-usage-endpoint
Credit usage endpoint
2024-12-20 15:45:57 -03:00
Nicolas
ba95df96b1 Update rate-limiter.ts 2024-12-20 15:45:44 -03:00
Nicolas
6222152249 Nick: credit usage endpoint 2024-12-20 15:44:17 -03:00
Nicolas
58b8064958
Merge pull request #998 from mendableai/feat-sdk/try-catch-message-handler
[SDK] Added try catch to ws message handler
2024-12-20 14:01:15 -03:00
rafaelmmiller
63bbeadbfc Added try catch to message handler 2024-12-20 08:05:13 -03:00
RutamBhagat
f47e3114d6 feat(rust-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-20 02:04:02 -08:00
RutamBhagat
2b488cac3d chore: remove pytest dependency from pyproject.toml 2024-12-20 01:54:29 -08:00
Nicolas
3e60f175bb Nick: prompt should be optional on /extract sdks 2024-12-19 16:14:49 -03:00
Nicolas
8063474c85 Update __init__.py 2024-12-19 16:14:17 -03:00
Nicolas
525a71d789 Update __init__.py 2024-12-19 16:10:42 -03:00
Nicolas
4fddc86e66 Update package.json 2024-12-19 16:09:08 -03:00
Nicolas
d67db99791
Merge pull request #994 from mendableai/feat/added-id-to-ws-sdks
feat-SDK/added crawl id to ws
2024-12-19 16:08:08 -03:00
Nicolas
ed24853ca6
Merge pull request #996 from mendableai/fix/title-extra-info
[BUG] fixed title extra info
2024-12-19 16:05:49 -03:00
Thomas Kosmas
c8cd0148dd refactor: remove error logging for 'isows' module import in WebSocket initialization 2024-12-19 20:39:30 +02:00
Gergő Móricz
071b9a01c3 fix(scrapeURL/fire-engine): pass geolocation 2024-12-19 18:23:21 +01:00
Thomas Kosmas
6002bf3228 feat: dynamically import WebSocket module with error handling 2024-12-19 14:52:43 +02:00
rafaelmmiller
cf2ec77131 fixed title extra info 2024-12-19 08:32:10 -03:00
Nicolas
066071cd54 Update llmExtract.ts 2024-12-18 23:45:43 -03:00
Nicolas
05605112bb Update extract.ts 2024-12-18 23:34:07 -03:00
Nicolas
2d37dca9dc Nick: introduced system prompt to /extract 2024-12-18 22:10:41 -03:00
Nicolas
75984b4582 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-12-18 21:45:12 -03:00
Nicolas
a759a7ab7a Nick: small improvements 2024-12-18 21:45:06 -03:00
Eric Ciarla
94267ff406 Create o1_web_extractor.py 2024-12-18 17:31:45 -05:00
rafaelmmiller
19246f6289 feat-SDK/added crawl id to ws 2024-12-18 18:36:04 -03:00
Nicolas
e899ecbe44 Update llmExtract.ts 2024-12-18 16:52:05 -03:00
RutamBhagat
e776847c71 feat(js-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-17 13:40:14 -08:00
RutamBhagat
29cea4c51d feat(python-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-17 13:31:35 -08:00
Móricz Gergő
bd36c441d3 feat(queue-worker): improve team-based logging 2024-12-17 22:06:36 +01:00
Móricz Gergő
780442d73b feat: improve billing logging 2024-12-17 22:02:31 +01:00
Nicolas
ac187452c3 Nick: better filtering for urls that should be scraped 2024-12-17 17:34:55 -03:00
Nicolas
3b6edef9fa chore: formatting 2024-12-17 16:58:57 -03:00
Nicolas
b9f621bed5 Nick: extract fixes 2024-12-17 16:58:35 -03:00
Nicolas
79e335636a Nick: fixed extract issues 2024-12-17 16:40:45 -03:00
Nicolas
6d77879d68 Update extract.ts 2024-12-17 15:22:25 -03:00
Nicolas
e26a0a65a7 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-12-17 15:19:53 -03:00
Nicolas
0f8b8a717d Update map.ts 2024-12-17 15:19:52 -03:00
Eric Ciarla
4ad6665d39
Merge pull request #987 from mendableai/default-to-pdf-parse
Revert to pdf parse
2024-12-17 12:13:46 -05:00
Eric Ciarla
a20a003c74 revert to pdf parse 2024-12-17 12:12:22 -05:00
Eric Ciarla
db8e9c36d6
Merge pull request #984 from mendableai/default-to-pdf-parse
Default to pdf2md, if under 500 chars (indicating failure) use LlamaParse
2024-12-17 10:06:40 -05:00
Eric Ciarla
194353af0d Remove pdf parse 2024-12-17 10:04:20 -05:00
Eric Ciarla
1402831a0a Replace pdf parse with pdf to md 2024-12-17 09:59:52 -05:00
Eric Ciarla
ed7d15d2af Update index.ts 2024-12-17 09:50:29 -05:00
Gergő Móricz
654d6c6e0b fix(scrapeURL): increase timeToRun 2024-12-17 13:21:24 +01:00
Gergő Móricz
47b968fede fix(scrapeURL/fire-engine): timeout calculation issues 2024-12-17 13:17:55 +01:00
Gergő Móricz
7f57c868be Revert "fix(scrapeURL): better timeToRun distribution"
This reverts commit 284a6ccedd1baede825571ee933eb7e4f773e2de.
2024-12-16 23:08:20 +01:00
Gergő Móricz
284a6ccedd fix(scrapeURL): better timeToRun distribution 2024-12-16 23:01:34 +01:00
Gergő Móricz
0013bdfcb4 feat(v1/scrape): add more context to timeout logs 2024-12-16 22:42:51 +01:00