yujunjun/crawl4ai
1
0
Fork 0
mirror of https://github.com/unclecode/crawl4ai.git synced 2025-12-29 11:27:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove excluded tags from website content

Browse Source
This commit is contained in:
unclecode 2024-09-12 16:11:20 +08:00
parent 396f430022
commit 30807f5535
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
crawl4ai/utils.py
View File

@ -441,6 +441,10 @@ def get_content_of_website_optimized(url: str, html: str, word_count_threshold:
image_description_min_word_threshold = kwargs.get('image_description_min_word_threshold', IMAGE_DESCRIPTION_MIN_WORD_THRESHOLD)
for tag in kwargs.get('excluded_tags', []) or []:
for el in body.select(tag):
el.decompose()
if css_selector:
selected_elements = body.select(css_selector)
if not selected_elements: