7 Commits

Author	SHA1	Message	Date
Daniel Augustus Bichuetti Silva	77a513fe49	Fix crawler long file names (#2723) ... * Changing the name that crawled page is saved to avoid long file names error on some file systems * Custom naming function for saving crawled files * Update Documentation & Code Style * Remove bad characters on file name and preffix * Add test for naming function * Update Documentation & Code Style * Fix expensive regex recalculation and linter warns * Check for exceptions on file dump * Remove param_naming variable * Fix file paths on Windows, Linux and Mac * Update Documentation & Code Style * Test using one of the docstrings examples * Change default naming function Update docstrings * Applying formatting rules * Update Documentation & Code Style * Fix mypy incompatible assignment error * Remove unused type declaration * Fix typo * Update tests for naming function * Update Documentation & Code Style Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2022-07-11 12:16:32 +02:00
Daniel Augustus Bichuetti Silva	e3b2ee956a	Improved crawler support for dynamically loaded pages (#2710) ... * Improved crawler support for dynamically loaded pages * Reduced scope of StaleElementReferenceException and removed deprecated code from WebDriver initialization * Improvements on crawler testing code * Code format and style applied on f028331948c170448613e86dfdfa222f7c2043fd * Update Documentation & Code Style * Remove unused imports/parameters Co-authored-by: Sara Zan <sara.zanzottera@deepset.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2022-07-01 10:47:33 +02:00
Sara Zan	e8546e2124	Replace deprecated Selenium methods (#2724) ... * Fix crawler.py * Fix test_connector.py * unused import Co-authored-by: danielbichuetti <daniel.bichuetti@gmail.com>	2022-06-24 12:05:32 +02:00
Stefano Fiorucci	c178f60e3a	Make crawler extract also hidden text (#2642) ... * make crawler extract also hidden text * Update Documentation & Code Style * try to adapt test for extract_hidden_text * Update Documentation & Code Style * fix test bug * fix bug in test * added test for hidden text" * Update Documentation & Code Style * fix bug in test * Update Documentation & Code Style * fix test * Update Documentation & Code Style * fix other test bug Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2022-06-10 09:51:41 +02:00
Sara Zan	c17969e001	Fix failing Crawler test (#2640) ... * Make tests insensntive to ordering of crawled pages * fix docstring	2022-06-07 18:14:43 +02:00
Sara Zan	83648b9bc0	[CI refactoring] Rewrite Crawler tests (#2557) ... * Rewrite crawler tests (very slow) and fix small crawler bug * Update Documentation & Code Style * compile the regex only once * Factor out the html files & add content check to most tests * Clarify that even starting URLs can be excluded * Update Documentation & Code Style * Change signature * Fix failing test * Update Documentation & Code Style Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2022-06-06 17:52:37 +02:00
Sara Zan	ff4303c51b	[CI refactoring] Categorize tests into folders (#2554) ... * Categorize tests into folders * Fix linux_ci.yml and an import * Wrong path	2022-05-17 09:55:53 +01:00