yujunjun/olmocr
1
0
Fork 0
mirror of https://github.com/allenai/olmocr.git synced 2025-11-03 11:35:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

filtering out stupid ads

Browse Source
This commit is contained in:
Jake Poznanski 2024-10-02 15:36:41 +00:00
parent 6ef8226347
commit 549e07bed0
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
pdelfin/filter/filter.py
View File

@ -85,6 +85,9 @@ class PdfFilter:
"file", "file",
"save", "save",
"casino", "casino",
"viagra",
"cialis",
"ciprofloxacin",
} }
seo_word_probs = {word: self.ngram_log_probs[word] for word in seo_words} seo_word_probs = {word: self.ngram_log_probs[word] for word in seo_words}