* recency reranker code
* removed
* readd
* edited code
* edit
* mypy test fix
* adding warnings for score method
* fix
* fix
* adding paper link
* comments implementation
* change to predict and predict_batch
* change to predict and predict_batch 2
* adding unit test
* fixes
* small fixes
* fix for unit test
* table driven test
* small fixes
* small fixes2
* adding predict_batch tests
* add recentness_ranker to api reference docs
* implementing feedback
* implementing feedback2
* implementing feedback3
* implementing feedback4
* implementing feedback5
* remove document_map, remove final check if score is not None
* add final check if doc score is not None for mypy
---------
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
* Adding embed_meta_fields to ranker nodes
* Fix tests by adding case where embed_meta_fields=None
* Adding unit test for _add_meta_fields_to_docs
* Fix pylint
* Add unit test
* Added another unit test. Caught a bug.
* Adding more unit tests
* Add unit test
* Updating some older tests into unit tests using mocking
* Convert another test to unit test
* Test run method
* One last unit test
* adding unit tests for sentence transformers ranker
* Adding more unit tests
* Remove empty line
* Undo static method
* Revert change
* Updated indentation and added match message
* Remove unneeded paranthesis
* bump ES version in CI
disable ssl
wait for service to start
set env vars
do not use choco to install ES
re-enable jobs deps
skip test on windows CI because of OOM
allocate more memory for ES
uniform ES installation and use default heap size
skip tests causing OOM
increase job timeout
restore memory limit for ES8
* Use latest elasticsearch version
* Add max_tokens to BaseGenerator params
* Make mypy happy
* Rebase and resolve conflicts
* Fix signature issues
* Update lg
* Add a mocked unit test method
* end-of-file-fixer corrected file
* Convert to unit test
* Mark test as integration
* make the test unit
---------
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* preserve root_node and add tests
* Added if statement to fix failing tests
---------
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Sebastian Husch Lee <sjrl423@gmail.com>
* Deprecate name parameter
* Adapt existing tests and uses of PromptTemplate
* Move parameter `name` to end
* Adapt existing tests
* lg update
---------
Co-authored-by: Darja Fokina <daria.f93@gmail.com>
* fixed test base for hub 0.13.3
* check if test succeed from branch
* 2nd check if test succeed from branch
* removed dependency changes
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
* Upgrade to transformers 4.28.1
* Commenting out failing piece of test
* trailing-whitespace
* Adjust regex for error match - it changed between releases
* Remove RAG tests failing with transformers update
* extract elasticsearch
* update pyproject.toml
* make more import optional
* move MockBaseRetriever in conftest
* install es in the es integration tests
* Starting adding support for TableCell
* Update tests to use row and col
* Added schema test to check to_dict and from_dict works for Table documents. Also updated Doc.__eq__ to work for tables.
* Update eval test to use TableCell
* Added more schema tests for table docs, labels and answers.
* Add boolean to toggle between Span and TableCell
* Add deprecation message
* Test that table answers work as responses in the rest API
---------
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
* Added warning messages for documents that are skipped by RouteDocuments. Begun adding support for new option return_remaining and List of List support for metadata value splitting.
* Simplify _split_by_content_type
* Added new unit test and updated _calculate_outgoing_edges
* Added some TODOs and turned assert into raising an error.
* Update logging messages and make new fixture in tests
* Update _split_by_metadata_values to work with return_remaining
* Remove unneeded code
* Documentation
* Add proper support for list of lists
* Fix mypy errors
* Added assert to make mypy happy
* Update haystack/nodes/other/route_documents.py
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* PR comments
* Remove check for logging level
* make mypy happy
* Update docstring of metadata_values
* Removed duplicate check. Make explicit check for metadata_values
---------
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* fix: list element and mapping logic around it added to ParsrConverter convert step + unit test covering the specific mapping of list content from Parsr's to Haystack's
* Code review changes
* changed the samples path after conftest changes
* added samples_path to function arg
---------
Co-authored-by: Namoush <fmpereira22@gmail.com>
Co-authored-by: Fernando Pereira <fernando.pereira@criticalsoftware.com>
Co-authored-by: Mayank Jobanputra <mayankjobanputra@gmail.com>
Co-authored-by: bogdankostic <bogdankostic@web.de>
* Add pytest fixture to block requests in unit tests
* Mark test correctly as integration
* Fix crawler unit test failing cause it tries to install chromedriver
* Initial commit, add search_engine
* Add TopPSampler
* Add more TopPSampler unit tests
* Remove SearchEngineSampler (converted to TopPSampler)
* Add some basic WebSearch unit tests
* Rename unit tests
* Add WebRetriever into agent_tools
* Adjust to WebRetriever
* Add WebRetriever mode [snippet|document]
* Minor changes
* SerperDev: add peopleAlsoAsk search results
* First agent for hotpotqa
* Making WebRetriever work on hotpotqa
* refactor: minor WebRetriever improvements (#4377)
* refactor: remove doc ids rebuild + antecipate cache
* refactor: improve caching, fix Document ids
* Minor WebRetriever improvements
* Overlooked minor fixes
* feat: add Bing API as search engine
* refactor: let kwargs pass-through
* feat: increase search context
* check sampler result, improve batch typing
* refactor: increase mypy compliance
* Initial commit, add search_engine
* Add TopPSampler
* Add more TopPSampler unit tests
* Remove SearchEngineSampler (converted to TopPSampler)
* Add some basic WebSearch unit tests
* Rename unit tests
* Add WebRetriever into agent_tools
* Adjust to WebRetriever
* Add WebRetriever mode [snippet|document]
* Minor changes
* SerperDev: add peopleAlsoAsk search results
* First agent for hotpotqa
* Making WebRetriever work on hotpotqa
* refactor: minor WebRetriever improvements (#4377)
* refactor: remove doc ids rebuild + antecipate cache
* refactor: improve caching, fix Document ids
* Minor WebRetriever improvements
* Overlooked minor fixes
* feat: add Bing API as search engine
* refactor: let kwargs pass-through
* feat: increase search context
* check sampler result, improve batch typing
* refactor: increase mypy compliance
* Fix mypy
* Minor example fixes
* Fix the descriptions
* PR feedback updates
* More fixes
* TopPSampler: handle top p None value, add unit test
* Add top_k to WebSearch
* Use boilerpy3 instead trafilatura
* Remove date finding
* Add more WebRetriever docs
* Refactor long methods
* making the preprocessor optional
* hide WebSearch and make NeuralWebSearch a pipeline
* remove unused imports
* add WebQAPipeline and split example into two
* change example search engine to SerperDev
* Turn off progress bars in WebRetriever's PreProcesssor
* Agent tool examples - final updates
* Add webqa test, search results ranking scores
* Better answer box handling for SerperDev and SerpAPI
* Minor fixes
* pylint
* pylint fixes
* extract TopPSampler from WebRetriever
* use sampler only for WebRetriever modes other than snippet
* add web retriever tests
* add web retriever tests
* exclude rdflib@6.3.2 due to license issues
* add test for preprocessed docs and kwargs examples in docstrings
* Move test_webqa_pipeline to test/pipelines
* change docstring for join_documents_and_scores
* Use WebQAPipeline in examples/web_lfqa.py
* Use WebQAPipeline in examples/web_lfqa.py
* Move test_webqa_pipeline to e2e
* Updated lg
* Sampler added automatically in WebQAPipeline, no need to add it
* Updated lg
* Updated lg
* :ignore Update agent tools examples to new templates (#4503)
* Update examples to new templates
* Add print back
* fix linting and black format issues
---------
Co-authored-by: Daniel Bichuetti <daniel.bichuetti@gmail.com>
Co-authored-by: agnieszka-m <amarzec13@gmail.com>
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
