UncleCode ac5f461d40 feat(crawler): add memory-adaptive dispatcher with rate limiting ...

Implements a new MemoryAdaptiveDispatcher class to manage concurrent crawling operations with memory monitoring and rate limiting capabilities. Changes include:

- Added RateLimitConfig dataclass for configuring rate limiting behavior
- Extended CrawlerRunConfig with dispatcher-related settings
- Refactored arun_many to use the new dispatcher system
- Added memory threshold and session permit controls
- Integrated optional progress monitoring display

BREAKING CHANGE: The arun_many method now uses MemoryAdaptiveDispatcher by default, which may affect concurrent crawling behavior

2025-01-10 16:01:18 +08:00

..

assets

- User agent

2024-06-08 17:59:42 +08:00

amazon_product_extraction_direct_url.py

Commit Message:

2024-12-29 20:05:18 +08:00

amazon_product_extraction_using_hooks.py

Commit Message:

2024-12-29 20:05:18 +08:00

amazon_product_extraction_using_use_javascript.py

Commit Message:

2024-12-29 20:05:18 +08:00

async_webcrawler_multiple_urls_example.py

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

2024-10-02 17:34:56 +08:00

browser_optimization_example.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

chainlit.md

Add research assistant example using Chainlit

2024-06-04 22:43:09 +08:00

crawlai_vs_firecrawl.py

chore: Update crawl4ai package with AsyncWebCrawler and JsonCssExtractionStrategy

2024-09-03 23:32:27 +08:00

dispatcher_example.py

feat(crawler): add memory-adaptive dispatcher with rate limiting

2025-01-10 16:01:18 +08:00

docker_example.py

Enhance Docker support and improve installation process

2024-11-29 20:52:51 +08:00

extraction_strategies_example.py

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

full_page_screenshot_and_pdf_export.md

Enhance crawler capabilities and documentation

2024-12-25 21:34:31 +08:00

hello_world.py

refactor():

2025-01-02 17:53:30 +08:00

hooks_example.py

Renames browser_config param to config in AsyncWebCrawler

2024-12-26 16:34:36 +08:00

language_support_example.py

Enhance AsyncWebCrawler with smart waiting and screenshot capabilities

2024-10-02 17:34:56 +08:00

llm_extraction_openai_pricing.py

Fix #340 example llm_extraction (#358)

2024-12-24 19:56:07 +08:00

quickstart_async.config.py

Commit Message:

2024-12-29 20:05:18 +08:00

quickstart_async.py

Commit Message:

2024-12-29 20:05:18 +08:00

quickstart_sync.py

Create series of quickstart files.

2024-09-04 15:33:24 +08:00

quickstart_v0.ipynb

Update documents, upload new version of quickstart.

2024-10-30 20:39:35 +08:00

quickstart.ipynb

Update documents, upload new version of quickstart.

2024-10-30 20:39:35 +08:00

research_assistant.py

Push async version last changes for merge to main branch

2024-09-24 20:52:08 +08:00

rest_call.py

- User agent

2024-06-08 17:59:42 +08:00

sample_ecommerce.html

Push async version last changes for merge to main branch

2024-09-24 20:52:08 +08:00

ssl_example.py

Commit Message:

2024-12-26 15:17:07 +08:00

storage_state_tutorial.md

Implement new async crawler features and stability updates

2024-12-10 17:55:29 +08:00

summarize_page.py

## [v0.2.74] - 2024-07-08

2024-07-08 16:33:25 +08:00

tutorial_dynamic_clicks.md

Add PDF & screenshot functionality, new tutorial

2024-12-10 20:10:39 +08:00

v0_4_24_walkthrough.py

refactor(build): simplify setup.py configuration

2025-01-01 15:52:01 +08:00

v0.3.74.overview.py

feat: add enhanced markdown generation example with citations and file output

2024-11-22 20:14:58 +08:00