yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-07-25 09:56:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,537 Commits 651 Branches 22 Tags
Commit Graph

3537 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Thomas Kosmas
f043f5fd61 Enhance error handling in E2E tests and introduce CrawlWatcher tests 
- Updated error messages in E2E tests to provide clearer feedback for blocked URLs and invalid API keys.
- Added new test suite for CrawlWatcher to ensure proper instantiation and error handling when WebSocket is unavailable.
- Improved test conditions for URL scraping and crawling to reflect updated error responses.
 2024-12-21 02:27:22 +02:00
Nicolas
818f5544cb
Merge pull request #1001 from mendableai/nsc/block-list-string 
Refactored Blocklist Error Messages
 2024-12-20 18:11:01 -03:00
Nicolas
d1f3e26f9e Nick: blocklist string 2024-12-20 18:09:49 -03:00
Nicolas
8e947344ad
Merge pull request #999 from mendableai/nsc/credit-usage-endpoint 
Credit usage endpoint
 2024-12-20 15:45:57 -03:00
Nicolas
ba95df96b1 Update rate-limiter.ts 2024-12-20 15:45:44 -03:00
Nicolas
6222152249 Nick: credit usage endpoint 2024-12-20 15:44:17 -03:00
Nicolas
58b8064958
Merge pull request #998 from mendableai/feat-sdk/try-catch-message-handler 
[SDK] Added try catch to ws message handler
 2024-12-20 14:01:15 -03:00
rafaelmmiller
63bbeadbfc Added try catch to message handler 2024-12-20 08:05:13 -03:00
RutamBhagat
f47e3114d6 feat(rust-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-20 02:04:02 -08:00
RutamBhagat
2b488cac3d chore: remove pytest dependency from pyproject.toml 2024-12-20 01:54:29 -08:00
Nicolas
3e60f175bb Nick: prompt should be optional on /extract sdks 2024-12-19 16:14:49 -03:00
Nicolas
8063474c85 Update __init__.py 2024-12-19 16:14:17 -03:00
Nicolas
525a71d789 Update __init__.py 2024-12-19 16:10:42 -03:00
Nicolas
4fddc86e66 Update package.json 2024-12-19 16:09:08 -03:00
Nicolas
d67db99791
Merge pull request #994 from mendableai/feat/added-id-to-ws-sdks 
feat-SDK/added crawl id to ws
 2024-12-19 16:08:08 -03:00
Nicolas
ed24853ca6
Merge pull request #996 from mendableai/fix/title-extra-info 
[BUG] fixed title extra info
 2024-12-19 16:05:49 -03:00
Thomas Kosmas
c8cd0148dd refactor: remove error logging for 'isows' module import in WebSocket initialization 2024-12-19 20:39:30 +02:00
Gergő Móricz
071b9a01c3 fix(scrapeURL/fire-engine): pass geolocation 2024-12-19 18:23:21 +01:00
Thomas Kosmas
6002bf3228 feat: dynamically import WebSocket module with error handling 2024-12-19 14:52:43 +02:00
rafaelmmiller
cf2ec77131 fixed title extra info 2024-12-19 08:32:10 -03:00
Nicolas
066071cd54 Update llmExtract.ts 2024-12-18 23:45:43 -03:00
Nicolas
05605112bb Update extract.ts 2024-12-18 23:34:07 -03:00
Nicolas
2d37dca9dc Nick: introduced system prompt to /extract 2024-12-18 22:10:41 -03:00
Nicolas
75984b4582 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-12-18 21:45:12 -03:00
Nicolas
a759a7ab7a Nick: small improvements 2024-12-18 21:45:06 -03:00
Eric Ciarla
94267ff406 Create o1_web_extractor.py 2024-12-18 17:31:45 -05:00
rafaelmmiller
19246f6289 feat-SDK/added crawl id to ws 2024-12-18 18:36:04 -03:00
Nicolas
e899ecbe44 Update llmExtract.ts 2024-12-18 16:52:05 -03:00
RutamBhagat
e776847c71 feat(js-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-17 13:40:14 -08:00
RutamBhagat
29cea4c51d feat(python-sdk): improve API key handling for cloud vs self-hosted services in FirecrawlApp 2024-12-17 13:31:35 -08:00
Móricz Gergő
bd36c441d3 feat(queue-worker): improve team-based logging 2024-12-17 22:06:36 +01:00
Móricz Gergő
780442d73b feat: improve billing logging 2024-12-17 22:02:31 +01:00
Nicolas
ac187452c3 Nick: better filtering for urls that should be scraped 2024-12-17 17:34:55 -03:00
Nicolas
3b6edef9fa chore: formatting 2024-12-17 16:58:57 -03:00
Nicolas
b9f621bed5 Nick: extract fixes 2024-12-17 16:58:35 -03:00
Nicolas
79e335636a Nick: fixed extract issues 2024-12-17 16:40:45 -03:00
Nicolas
6d77879d68 Update extract.ts 2024-12-17 15:22:25 -03:00
Nicolas
e26a0a65a7 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-12-17 15:19:53 -03:00
Nicolas
0f8b8a717d Update map.ts 2024-12-17 15:19:52 -03:00
Eric Ciarla
4ad6665d39
Merge pull request #987 from mendableai/default-to-pdf-parse 
Revert to pdf parse
 2024-12-17 12:13:46 -05:00
Eric Ciarla
a20a003c74 revert to pdf parse 2024-12-17 12:12:22 -05:00
Eric Ciarla
db8e9c36d6
Merge pull request #984 from mendableai/default-to-pdf-parse 
Default to pdf2md, if under 500 chars (indicating failure) use LlamaParse
 2024-12-17 10:06:40 -05:00
Eric Ciarla
194353af0d Remove pdf parse 2024-12-17 10:04:20 -05:00
Eric Ciarla
1402831a0a Replace pdf parse with pdf to md 2024-12-17 09:59:52 -05:00
Eric Ciarla
ed7d15d2af Update index.ts 2024-12-17 09:50:29 -05:00
Gergő Móricz
654d6c6e0b fix(scrapeURL): increase timeToRun 2024-12-17 13:21:24 +01:00
Gergő Móricz
47b968fede fix(scrapeURL/fire-engine): timeout calculation issues 2024-12-17 13:17:55 +01:00
Gergő Móricz
7f57c868be Revert "fix(scrapeURL): better timeToRun distribution" 
This reverts commit 284a6ccedd1baede825571ee933eb7e4f773e2de.
 2024-12-16 23:08:20 +01:00
Gergő Móricz
284a6ccedd fix(scrapeURL): better timeToRun distribution 2024-12-16 23:01:34 +01:00
Gergő Móricz
0013bdfcb4 feat(v1/scrape): add more context to timeout logs 2024-12-16 22:42:51 +01:00