crawl4ai/JOURNAL.md

# Development Journal

This journal tracks significant feature additions, bug fixes, and architectural decisions in the crawl4ai project. It serves as both documentation and a historical record of the project's evolution.

## [2025-04-17] Added Content Source Selection for Markdown Generation

**Feature:** Configurable content source for markdown generation

**Changes Made:**
1. Added `content_source: str = "cleaned_html"` parameter to `MarkdownGenerationStrategy` class
2. Updated `DefaultMarkdownGenerator` to accept and pass the content source parameter
3. Renamed the `cleaned_html` parameter to `input_html` in the `generate_markdown` method
4. Modified `AsyncWebCrawler.aprocess_html` to select the appropriate HTML source based on the generator's config
5. Added `preprocess_html_for_schema` import in `async_webcrawler.py`

**Implementation Details:**
- Added a new `content_source` parameter to specify which HTML input to use for markdown generation
- Options include: "cleaned_html" (default), "raw_html", and "fit_html"
- Used a dictionary dispatch pattern in `aprocess_html` to select the appropriate HTML source
- Added proper error handling with fallback to cleaned_html if content source selection fails
- Ensured backward compatibility by defaulting to "cleaned_html" option

**Files Modified:**
- `crawl4ai/markdown_generation_strategy.py`: Added content_source parameter and updated the method signature
- `crawl4ai/async_webcrawler.py`: Added HTML source selection logic and updated imports

**Examples:**
- Created `docs/examples/content_source_example.py` demonstrating how to use the new parameter

**Challenges:**
- Maintaining backward compatibility while reorganizing the parameter flow
- Ensuring proper error handling for all content source options
- Making the change with minimal code modifications

**Why This Feature:**
The content source selection feature allows users to choose which HTML content to use as input for markdown generation:
1. "cleaned_html" - Uses the post-processed HTML after scraping strategy (original behavior)
2. "raw_html" - Uses the original raw HTML directly from the web page
3. "fit_html" - Uses the preprocessed HTML optimized for schema extraction

This feature provides greater flexibility in how users generate markdown, enabling them to:
- Capture more detailed content from the original HTML when needed
- Use schema-optimized HTML when working with structured data
- Choose the approach that best suits their specific use case
## [2025-04-17] Implemented High Volume Stress Testing Solution for SDK

**Feature:** Comprehensive stress testing framework using `arun_many` and the dispatcher system to evaluate performance, concurrency handling, and identify potential issues under high-volume crawling scenarios.

**Changes Made:**
1.  Created a dedicated stress testing framework in the `benchmarking/` (or similar) directory.
2.  Implemented local test site generation (`SiteGenerator`) with configurable heavy HTML pages.
3.  Added basic memory usage tracking (`SimpleMemoryTracker`) using platform-specific commands (avoiding `psutil` dependency for this specific test).
4.  Utilized `CrawlerMonitor` from `crawl4ai` for rich terminal UI and real-time monitoring of test progress and dispatcher activity.
5.  Implemented detailed result summary saving (JSON) and memory sample logging (CSV).
6.  Developed `run_benchmark.py` to orchestrate tests with predefined configurations.
7.  Created `run_all.sh` as a simple wrapper for `run_benchmark.py`.

**Implementation Details:**
-   Generates a local test site with configurable pages containing heavy text and image content.
-   Uses Python's built-in `http.server` for local serving, minimizing network variance.
-   Leverages `crawl4ai`'s `arun_many` method for processing URLs.
-   Utilizes `MemoryAdaptiveDispatcher` to manage concurrency via the `max_sessions` parameter (note: memory adaptation features require `psutil`, not used by `SimpleMemoryTracker`).
-   Tracks memory usage via `SimpleMemoryTracker`, recording samples throughout test execution to a CSV file.
-   Uses `CrawlerMonitor` (which uses the `rich` library) for clear terminal visualization and progress reporting directly from the dispatcher.
-   Stores detailed final metrics in a JSON summary file.

**Files Created/Updated:**
-   `stress_test_sdk.py`: Main stress testing implementation using `arun_many`.
-   `benchmark_report.py`: (Assumed) Report generator for comparing test results.
-   `run_benchmark.py`: Test runner script with predefined configurations.
-   `run_all.sh`: Simple bash script wrapper for `run_benchmark.py`.
-   `USAGE.md`: Comprehensive documentation on usage and interpretation (updated).

**Testing Approach:**
-   Creates a controlled, reproducible test environment with a local HTTP server.
-   Processes URLs using `arun_many`, allowing the dispatcher to manage concurrency up to `max_sessions`.
-   Optionally logs per-batch summaries (when not in streaming mode) after processing chunks.
-   Supports different test sizes via `run_benchmark.py` configurations.
-   Records memory samples via platform commands for basic trend analysis.
-   Includes cleanup functionality for the test environment.

**Challenges:**
-   Ensuring proper cleanup of HTTP server processes.
-   Getting reliable memory tracking across platforms without adding heavy dependencies (`psutil`) to this specific test script.
-   Designing `run_benchmark.py` to correctly pass arguments to `stress_test_sdk.py`.

**Why This Feature:**
The high volume stress testing solution addresses critical needs for ensuring Crawl4AI's `arun_many` reliability:
1.  Provides a reproducible way to evaluate performance under concurrent load.
2.  Allows testing the dispatcher's concurrency control (`max_session_permit`) and queue management.
3.  Enables performance tuning by observing throughput (`URLs/sec`) under different `max_sessions` settings.
4.  Creates a controlled environment for testing `arun_many` behavior.
5.  Supports continuous integration by providing deterministic test conditions for `arun_many`.

**Design Decisions:**
-   Chose local site generation for reproducibility and isolation from network issues.
-   Utilized the built-in `CrawlerMonitor` for real-time feedback, leveraging its `rich` integration.
-   Implemented optional per-batch logging in `stress_test_sdk.py` (when not streaming) to provide chunk-level summaries alongside the continuous monitor.
-   Adopted `arun_many` with a `MemoryAdaptiveDispatcher` as the core mechanism for parallel execution, reflecting the intended SDK usage.
-   Created `run_benchmark.py` to simplify running standard test configurations.
-   Used `SimpleMemoryTracker` to provide basic memory insights without requiring `psutil` for this particular test runner.

**Future Enhancements to Consider:**
-   Create a separate test variant that *does* use `psutil` to specifically stress the memory-adaptive features of the dispatcher.
-   Add support for generated JavaScript content.
-   Add support for Docker-based testing with explicit memory limits.
-   Enhance `benchmark_report.py` to provide more sophisticated analysis of performance and memory trends from the generated JSON/CSV files.

---

## [2025-04-17] Refined Stress Testing System Parameters and Execution

**Changes Made:**
1.  Corrected `run_benchmark.py` and `stress_test_sdk.py` to use `--max-sessions` instead of the incorrect `--workers` parameter, accurately reflecting dispatcher configuration.
2.  Updated `run_benchmark.py` argument handling to correctly pass all relevant custom parameters (including `--stream`, `--monitor-mode`, etc.) to `stress_test_sdk.py`.
3.  (Assuming changes in `benchmark_report.py`) Applied dark theme to benchmark reports for better readability.
4.  (Assuming changes in `benchmark_report.py`) Improved visualization code to eliminate matplotlib warnings.
5.  Updated `run_benchmark.py` to provide clickable `file://` links to generated reports in the terminal output.
6.  Updated `USAGE.md` with comprehensive parameter descriptions reflecting the final script arguments.
7.  Updated `run_all.sh` wrapper to correctly invoke `run_benchmark.py` with flexible arguments.

**Details of Changes:**

1.  **Parameter Correction (`--max-sessions`)**:
    *   Identified the fundamental misunderstanding where `--workers` was used incorrectly.
    *   Refactored `stress_test_sdk.py` to accept `--max-sessions` and configure the `MemoryAdaptiveDispatcher`'s `max_session_permit` accordingly.
    *   Updated `run_benchmark.py` argument parsing and command construction to use `--max-sessions`.
    *   Updated `TEST_CONFIGS` in `run_benchmark.py` to use `max_sessions`.

2.  **Argument Handling (`run_benchmark.py`)**:
    *   Improved logic to collect all command-line arguments provided to `run_benchmark.py`.
    *   Ensured all relevant arguments (like `--stream`, `--monitor-mode`, `--port`, `--use-rate-limiter`, etc.) are correctly forwarded when calling `stress_test_sdk.py` as a subprocess.

3.  **Dark Theme & Visualization Fixes (Assumed in `benchmark_report.py`)**:
    *   (Describes changes assumed to be made in the separate reporting script).

4.  **Clickable Links (`run_benchmark.py`)**:
    *   Added logic to find the latest HTML report and PNG chart in the `benchmark_reports` directory after `benchmark_report.py` runs.
    *   Used `pathlib` to generate correct `file://` URLs for terminal output.

5.  **Documentation Improvements (`USAGE.md`)**:
    *   Rewrote sections to explain `arun_many`, dispatchers, and `--max-sessions`.
    *   Updated parameter tables for all scripts (`stress_test_sdk.py`, `run_benchmark.py`).
    *   Clarified the difference between batch and streaming modes and their effect on logging.
    *   Updated examples to use correct arguments.

**Files Modified:**
-   `stress_test_sdk.py`: Changed `--workers` to `--max-sessions`, added new arguments, used `arun_many`.
-   `run_benchmark.py`: Changed argument handling, updated configs, calls `stress_test_sdk.py`.
-   `run_all.sh`: Updated to call `run_benchmark.py` correctly.
-   `USAGE.md`: Updated documentation extensively.
-   `benchmark_report.py`: (Assumed modifications for dark theme and viz fixes).

**Testing:**
-   Verified that `--max-sessions` correctly limits concurrency via the `CrawlerMonitor` output.
-   Confirmed that custom arguments passed to `run_benchmark.py` are forwarded to `stress_test_sdk.py`.
-   Validated clickable links work in supporting terminals.
-   Ensured documentation matches the final script parameters and behavior.

**Why These Changes:**
These refinements correct the fundamental approach of the stress test to align with `crawl4ai`'s actual architecture and intended usage:
1.  Ensures the test evaluates the correct components (`arun_many`, `MemoryAdaptiveDispatcher`).
2.  Makes test configurations more accurate and flexible.
3.  Improves the usability of the testing framework through better argument handling and documentation.


**Future Enhancements to Consider:**
- Add support for generated JavaScript content to test JS rendering performance
- Implement more sophisticated memory analysis like generational garbage collection tracking
- Add support for Docker-based testing with memory limits to force OOM conditions
- Create visualization tools for analyzing memory usage patterns across test runs
- Add benchmark comparisons between different crawler versions or configurations

## [2025-04-17] Fixed Issues in Stress Testing System

**Changes Made:**
1. Fixed custom parameter handling in run_benchmark.py
2. Applied dark theme to benchmark reports for better readability
3. Improved visualization code to eliminate matplotlib warnings
4. Added clickable links to generated reports in terminal output
5. Enhanced documentation with comprehensive parameter descriptions

**Details of Changes:**

1. **Custom Parameter Handling Fix**
   - Identified bug where custom URL count was being ignored in run_benchmark.py
   - Rewrote argument handling to use a custom args dictionary
   - Properly passed parameters to the test_simple_stress.py command
   - Added better UI indication of custom parameters in use

2. **Dark Theme Implementation**
   - Added complete dark theme to HTML benchmark reports
   - Applied dark styling to all visualization components
   - Used Nord-inspired color palette for charts and graphs
   - Improved contrast and readability for data visualization
   - Updated text colors and backgrounds for better eye comfort

3. **Matplotlib Warning Fixes**
   - Resolved warnings related to improper use of set_xticklabels()
   - Implemented correct x-axis positioning for bar charts
   - Ensured proper alignment of bar labels and data points
   - Updated plotting code to use modern matplotlib practices

4. **Documentation Improvements**
   - Created comprehensive USAGE.md with detailed instructions
   - Added parameter documentation for all scripts
   - Included examples for all common use cases
   - Provided detailed explanations for interpreting results
   - Added troubleshooting guide for common issues

**Files Modified:**
- `tests/memory/run_benchmark.py`: Fixed custom parameter handling
- `tests/memory/benchmark_report.py`: Added dark theme and fixed visualization warnings
- `tests/memory/run_all.sh`: Added clickable links to reports
- `tests/memory/USAGE.md`: Created comprehensive documentation

**Testing:**
- Verified that custom URL counts are now correctly used
- Confirmed dark theme is properly applied to all report elements
- Checked that matplotlib warnings are no longer appearing
- Validated clickable links to reports work in terminals that support them

**Why These Changes:**
These improvements address several usability issues with the stress testing system:
1. Better parameter handling ensures test configurations work as expected
2. Dark theme reduces eye strain during extended test review sessions
3. Fixing visualization warnings improves code quality and output clarity
4. Enhanced documentation makes the system more accessible for future use

**Future Enhancements:**
- Add additional visualization options for different types of analysis
- Implement theme toggle to support both light and dark preferences
- Add export options for embedding reports in other documentation
- Create dedicated CI/CD integration templates for automated testing

## [2025-04-09] Added MHTML Capture Feature

**Feature:** MHTML snapshot capture of crawled pages

**Changes Made:**
1. Added `capture_mhtml: bool = False` parameter to `CrawlerRunConfig` class
2. Added `mhtml: Optional[str] = None` field to `CrawlResult` model
3. Added `mhtml_data: Optional[str] = None` field to `AsyncCrawlResponse` class
4. Implemented `capture_mhtml()` method in `AsyncPlaywrightCrawlerStrategy` class to capture MHTML via CDP
5. Modified the crawler to capture MHTML when enabled and pass it to the result

**Implementation Details:**
- MHTML capture uses Chrome DevTools Protocol (CDP) via Playwright's CDP session API
- The implementation waits for page to fully load before capturing MHTML content
- Enhanced waiting for JavaScript content with requestAnimationFrame for better JS content capture
- We ensure all browser resources are properly cleaned up after capture

**Files Modified:**
- `crawl4ai/models.py`: Added the mhtml field to CrawlResult
- `crawl4ai/async_configs.py`: Added capture_mhtml parameter to CrawlerRunConfig
- `crawl4ai/async_crawler_strategy.py`: Implemented MHTML capture logic
- `crawl4ai/async_webcrawler.py`: Added mapping from AsyncCrawlResponse.mhtml_data to CrawlResult.mhtml

**Testing:**
- Created comprehensive tests in `tests/20241401/test_mhtml.py` covering:
  - Capturing MHTML when enabled
  - Ensuring mhtml is None when disabled explicitly
  - Ensuring mhtml is None by default
  - Capturing MHTML on JavaScript-enabled pages

**Challenges:**
- Had to improve page loading detection to ensure JavaScript content was fully rendered
- Tests needed to be run independently due to Playwright browser instance management
- Modified test expected content to match actual MHTML output

**Why This Feature:**
The MHTML capture feature allows users to capture complete web pages including all resources (CSS, images, etc.) in a single file. This is valuable for:
1. Offline viewing of captured pages
2. Creating permanent snapshots of web content for archival
3. Ensuring consistent content for later analysis, even if the original site changes

**Future Enhancements to Consider:**
- Add option to save MHTML to file
- Support for filtering what resources get included in MHTML
- Add support for specifying MHTML capture options

## [2025-04-10] Added Network Request and Console Message Capturing

**Feature:** Comprehensive capturing of network requests/responses and browser console messages during crawling

**Changes Made:**
1. Added `capture_network_requests: bool = False` and `capture_console_messages: bool = False` parameters to `CrawlerRunConfig` class
2. Added `network_requests: Optional[List[Dict[str, Any]]] = None` and `console_messages: Optional[List[Dict[str, Any]]] = None` fields to both `AsyncCrawlResponse` and `CrawlResult` models
3. Implemented event listeners in `AsyncPlaywrightCrawlerStrategy._crawl_web()` to capture browser network events and console messages
4. Added proper event listener cleanup in the finally block to prevent resource leaks
5. Modified the crawler flow to pass captured data from AsyncCrawlResponse to CrawlResult

**Implementation Details:**
- Network capture uses Playwright event listeners (`request`, `response`, and `requestfailed`) to record all network activity
- Console capture uses Playwright event listeners (`console` and `pageerror`) to record console messages and errors
- Each network event includes metadata like URL, headers, status, and timing information
- Each console message includes type, text content, and source location when available
- All captured events include timestamps for chronological analysis
- Error handling ensures even failed capture attempts won't crash the main crawling process

**Files Modified:**
- `crawl4ai/models.py`: Added new fields to AsyncCrawlResponse and CrawlResult
- `crawl4ai/async_configs.py`: Added new configuration parameters to CrawlerRunConfig
- `crawl4ai/async_crawler_strategy.py`: Implemented capture logic using event listeners
- `crawl4ai/async_webcrawler.py`: Added data transfer from AsyncCrawlResponse to CrawlResult

**Documentation:**
- Created detailed documentation in `docs/md_v2/advanced/network-console-capture.md`
- Added feature to site navigation in `mkdocs.yml`
- Updated CrawlResult documentation in `docs/md_v2/api/crawl-result.md`
- Created comprehensive example in `docs/examples/network_console_capture_example.py`

**Testing:**
- Created `tests/general/test_network_console_capture.py` with tests for:
  - Verifying capture is disabled by default
  - Testing network request capturing
  - Testing console message capturing
  - Ensuring both capture types can be enabled simultaneously
  - Checking correct content is captured in expected formats

**Challenges:**
- Initial implementation had synchronous/asynchronous mismatches in event handlers
- Needed to fix type of property access vs. method calls in handlers
- Required careful cleanup of event listeners to prevent memory leaks

**Why This Feature:**
The network and console capture feature provides deep visibility into web page activity, enabling:
1. Debugging complex web applications by seeing all network requests and errors
2. Security analysis to detect unexpected third-party requests and data flows
3. Performance profiling to identify slow-loading resources
4. API discovery in single-page applications
5. Comprehensive analysis of web application behavior

**Future Enhancements to Consider:**
- Option to filter captured events by type, domain, or content
- Support for capturing response bodies (with size limits)
- Aggregate statistics calculation for performance metrics
- Integration with visualization tools for network waterfall analysis
- Exporting captures in HAR format for use with external tools