4 Commits

Author SHA1 Message Date
UncleCode
9c58e4ce2e fix(docs): correct section numbering in deepcrawl_example.py tutorial 2025-03-04 20:57:33 +08:00
UncleCode
df6a6d5f4f refactor(docs): reorganize tutorial sections and update wrap-up example 2025-03-04 20:55:09 +08:00
UncleCode
56bc3c6e45 refactor(cli): improve CLI default command handling
Make 'crawl' the default command when no command is specified.
This improves user experience by allowing direct URL input without
explicitly specifying the 'crawl' command.

Also removes unnecessary blank lines in example code for better readability.
2025-03-04 20:28:16 +08:00
UncleCode
d024749633 refactor(deep-crawl): add max_pages limit and improve crawl control
Add max_pages parameter to all deep crawling strategies to limit total pages crawled.
Add score_threshold parameter to BFS/DFS strategies for quality control.
Remove legacy parameter handling in AsyncWebCrawler.
Improve error handling and logging in crawl strategies.

BREAKING CHANGE: Removed support for legacy parameters in AsyncWebCrawler.run_many()
2025-03-03 21:51:11 +08:00