crawl4ai/docs/md_v2/advanced/managed_browser.md

Content Filtering in Crawl4AI

This guide explains how to use content filtering strategies in Crawl4AI to extract the most relevant information from crawled web pages. You'll learn how to use the built-in BM25ContentFilter and how to create your own custom content filtering strategies.

Relevance Content Filter

The RelevanceContentFilter is an abstract class that provides a common interface for content filtering strategies. Specific filtering algorithms, like PruningContentFilter or BM25ContentFilter, inherit from this class and implement the filter_content method. This method takes the HTML content as input and returns a list of filtered text blocks.

Pruning Content Filter

The PruningContentFilter is a tree-shaking algorithm that analyzes the HTML DOM structure and removes less relevant nodes based on various metrics like text density, link density, and tag importance. It evaluates each node using a composite scoring system and "prunes" nodes that fall below a certain threshold.

Usage

from crawl4ai import AsyncWebCrawler
from crawl4ai.content_filter_strategy import PruningContentFilter

async def filter_content(url):
    async with AsyncWebCrawler() as crawler:
        content_filter = PruningContentFilter(
            min_word_threshold=5,
            threshold_type='dynamic',
            threshold=0.45
        )
        result = await crawler.arun(url=url, extraction_strategy=content_filter, fit_markdown=True)
        if result.success:
            print(f"Cleaned Markdown:\n{result.fit_markdown}")

Parameters

• min_word_threshold: (Optional) Minimum number of words a node must contain to be considered relevant. Nodes with fewer words are automatically pruned.

• threshold_type: (Optional, default 'fixed') Controls how pruning thresholds are calculated:

  • 'fixed': Uses a constant threshold value for all nodes
  • 'dynamic': Adjusts threshold based on node characteristics like tag importance and text/link ratios

• threshold: (Optional, default 0.48) Base threshold value for node pruning:

  • For fixed threshold: Nodes scoring below this value are removed
  • For dynamic threshold: This value is adjusted based on node properties

How It Works

The pruning algorithm evaluates each node using multiple metrics:

• Text density: Ratio of actual text to overall node content
• Link density: Proportion of text within links
• Tag importance: Weight based on HTML tag type (e.g., article, p, div)
• Content quality: Metrics like text length and structural importance

Nodes scoring below the threshold are removed, effectively "shaking" less relevant content from the DOM tree. This results in a cleaner document containing only the most relevant content blocks.

The algorithm is particularly effective for:

• Removing boilerplate content
• Eliminating navigation menus and sidebars
• Preserving main article content
• Maintaining document structure while removing noise

BM25 Algorithm

The BM25ContentFilter uses the BM25 algorithm, a ranking function used in information retrieval to estimate the relevance of documents to a given search query. In Crawl4AI, this algorithm helps to identify and extract text chunks that are most relevant to the page's metadata or a user-specified query.

Usage

To use the BM25ContentFilter, initialize it and then pass it as the extraction_strategy parameter to the arun method of the crawler.

from crawl4ai import AsyncWebCrawler
from crawl4ai.content_filter_strategy import BM25ContentFilter

async def filter_content(url, query=None):
    async with AsyncWebCrawler() as crawler:
        content_filter = BM25ContentFilter(user_query=query)
        result = await crawler.arun(url=url, extraction_strategy=content_filter, fit_markdown=True) # Set fit_markdown flag to True to trigger BM25 filtering
        if result.success:
            print(f"Filtered Content (JSON):\n{result.extracted_content}")
            print(f"\nFiltered Markdown:\n{result.fit_markdown}") # New field in CrawlResult object
            print(f"\nFiltered HTML:\n{result.fit_html}") # New field in CrawlResult object. Note that raw HTML may have tags re-organized due to internal parsing.
        else:
            print("Error:", result.error_message)

# Example usage:
asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple", "fruit nutrition health")) # with query
asyncio.run(filter_content("https://en.wikipedia.org/wiki/Apple")) # without query, metadata will be used as the query.

Parameters

• user_query: (Optional) A string representing the search query. If not provided, the filter extracts relevant metadata (title, description, keywords) from the page and uses that as the query.
• bm25_threshold: (Optional, default 1.0) A float value that controls the threshold for relevance. Higher values result in stricter filtering, returning only the most relevant text chunks. Lower values result in more lenient filtering.

Fit Markdown Flag

Setting the fit_markdown flag to True in the arun method activates the BM25 content filtering during the crawl. The fit_markdown parameter instructs the scraper to extract and clean the HTML, primarily to prepare for a Large Language Model that cannot process large amounts of data. Setting this flag not only improves the quality of the extracted content but also adds the filtered content to two new attributes in the returned CrawlResult object: fit_markdown and fit_html.

Custom Content Filtering Strategies

You can create your own custom filtering strategies by inheriting from the RelevantContentFilter class and implementing the filter_content method. This allows you to tailor the filtering logic to your specific needs.

from crawl4ai.content_filter_strategy import RelevantContentFilter
from bs4 import BeautifulSoup, Tag
from typing import List

class MyCustomFilter(RelevantContentFilter):
    def filter_content(self, html: str) -> List[str]:
        soup = BeautifulSoup(html, 'lxml')
        # Implement custom filtering logic here
        # Example: extract all paragraphs within divs with class "article-body"
        filtered_paragraphs = []
        for tag in soup.select("div.article-body p"):
            if isinstance(tag, Tag):
                filtered_paragraphs.append(str(tag)) # Add the cleaned HTML element.  
        return filtered_paragraphs



async def custom_filter_demo(url: str):
    async with AsyncWebCrawler() as crawler:
        custom_filter = MyCustomFilter()
        result = await crawler.arun(url, extraction_strategy=custom_filter)
        if result.success:
            print(result.extracted_content)

This example demonstrates extracting paragraphs from a specific div class. You can customize this logic to implement different filtering strategies, use regular expressions, analyze text density, or apply other relevant techniques.

Conclusion

Content filtering strategies provide a powerful way to refine the output of your crawls. By using BM25ContentFilter or creating custom strategies, you can focus on the most pertinent information and improve the efficiency of your data processing pipeline.