* Fix duplicate question in Reader.eval()
* Add duplicate question support in document store
* Support duplicate questions in retriever eval
* Update tutorial
* Rename key_tuple
* Change error message
* Add warning when more than 6 labels
* Allow for label grouping options
* Add support for aggregating by label meta
* Satisfy mypy
* Fix duplicate question in Reader.eval()
* Add duplicate question support in document store
* Support duplicate questions in retriever eval
* Update tutorial
* Rename key_tuple
* Change error message
* Add warning when more than 6 labels
* Allow for label grouping options
* Add support for aggregating by label meta
* Satisfy mypy
* Make label field flexible, add docstrings
* Satisfy mypy
* Fix failing tests
* Adjust docstring
* Fix tutorial
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
* Adding ranker similar to retriever and reader
* Sort documents according to query-document similarity scores
* Reranking and model training runs for small example
* Added EvalRanker node
* Calculate recall@k in EvalRetriever and EvalRanker nodes
* Renaming EvalRetriever to EvalDocuments and EvalReader to EvalAnswers
* Added mean reciprocal rank as metric for EvalDocuments
* Fix bug that appeared when ranking documents with same score
* Remove commented code for unimplmented eval() of Ranker node
* Add documentation of k parameter in EvalDocuments
* Add Ranker docu and renaming top_k param
* Add knowledge graph module
* Fix type hint
* Add graph retriver module
* Change type annotations, change return format
* Add graph retriever that executes questions as sparql queries
* Linking only those entities that are in the knowledge graph
* Added logging and using relations extracted from Knowledge graph for linking
* Preventing entity linking from linking the same token to multiple entities
* Pruning triples that have no variables for select and count queries
* Support knowledge graphs with Pipelines
* Add text2sparql
* Entity linking and relation linking consider more special cases now based on evaluation on labelled data
* Separating example code from KGQA implementation
* Add eval on combined extarctive and kg questions
* Remove references to hp-test
* Add fields sparql_query and long_answer_list to metadata
* Removing modular Question2SPARQL approach
* Removing additional classes used for modular kgqa approach
* preparing lcquad data
* change graph db
* Translating namespaces in knowledge graph queries
* Creating graphdb index and loading triples from .ttl file
* Fetching graph config files, triples and model from S3
* Fix incompatibility issues with BaseGraphRetriever and BaseComponent
* Removing unused utility functions
* Adding doc strings and tutorial header
* Adding sparqlwrapper dependency
* Moving tutorial header
* Sorting tutorials by number within name of notebook
* Add latest docstring and tutorial changes
* Creating test cases for knowledge graph
* Changing knowledge graph example to harry potter
* Add latest docstring and tutorial changes
* Adapting the tutorial notebook to harry potter example
* Add GraphDB fixture for tests
* Add latest docstring and tutorial changes
* Added GraphDB docker launch to CI
* Use correct GraphDB fixture
* Check if GraphDB instance is already running
* Renaming question/query and incorporating other feedback from Timo and Tanay
* Removed type annotation
* Add latest docstring and tutorial changes
Co-authored-by: oryx1729 <oryx1729@protonmail.com>
Co-authored-by: Timo Moeller <timo.moeller@deepset.ai>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* WIP: First version of preprocessing tutorial
* stride renamed overlap, ipynb and py files created
* rename split_stride in test
* Update preprocessor api documentation
* define order for markdown files
* define order of modules in api docs
* Add colab links
* Incorporate review feedback
Co-authored-by: PiffPaffM <markuspaff.mp@gmail.com>
* Adding dummy generator implementation
* Adding tutorial to try the model
* Committing current non working code
* Committing current update where we need to call generate function directly and need to convert embedding to tensor way
* Addressing review comments.
* Refactoring finder, and implementing rag_generator class.
* Refined the implementation of RAGGenerator and now it is in clean shape
* Renaming RAGGenerator to RAGenerator
* Reverting change from finder.py and addressing review comments
* Remove support for RagSequenceForGeneration
* Utilizing embed_passage function from DensePassageRetriever
* Adding sample test data to verify generator output
* Updating testing script
* Updating testing script
* Fixing bug related to top_k
* Updating latest farm dependency
* Comment out farm dependency
* Reverting changes from TransformersReader
* Adding transformers dataset to compare transformers and haystack generator implementation
* Using generator_encoder instead of question_encoder to generate context_input_ids
* Adding workaround to install FARM dependency from master branch
* Removing unnecessary changes
* Fixing generator test
* Removing transformers datasets
* Fixing generator test
* Some cleanup and updating TODO comments
* Adding tutorial notebook
* Updating tutorials with comments
* Explicitly passing token model in RAG test
* Addressing review comments
* Fixing notebook
* Refactoring tests to reduce memory footprint
* Split generator tests in separate ci step and before running it reclaim memory by terminating containers
* Moving tika dependent test to separate dir
* Remove unwanted code
* Brining reader under session scope
* Farm is now session object hence restoring changes from default value
* Updating assert for pdf converter
* Dummy commit to trigger CI flow
* REducing memory footprint required for generator tests
* Fixing mypy issues
* Marking test with tika and elasticsearch markers. Reverting changes in CI and pytest splits
* reducing changes
* Fixing CI
* changing elastic search ci
* Fixing test error
* Disabling return of embedding
* Marking generator test as well
* Refactoring tutorials
* Increasing ES memory to 750M
* Trying another fix for ES CI
* Reverting CI changes
* Splitting tests in CI
* Generator and non-generator markers split
* Adding pytest.ini to add markers and enable strict-markers option
* Reducing elastic search container memory
* Simplifying generator test by using documents with embedding directly
* Bump up farm to 0.5.0
* Skeleton of doc website
* Flesh out documentation pages
* Split concepts into their own rst files
* add tutorial rsts
* Consistent level 1 markdown headers in tutorials
* Change theme to readthedocs
* Turn bullet points into prose
* Populate sections
* Add more text
* Add more sphinx files
* Add more retriever documentation
* combined all documenations in one structure
* rename of src to _src as it was ignored by git
* Incorporate MP2's changes
* add benchmark bar charts
* Adapt docstrings in Readers
* Improvements to intro, creation of glossary
* Adapt docstrings in Retrievers
* Adapt docstrings in Finder
* Adapt Docstrings of Finder
* Updates to text
* Edit text
* update doc strings
* proof read tutorials
* Edit text
* Edit text
* Add stacked chart
* populate graph with data
* Switch Documentation to markdown (#386)
* add way to generate markdown files to sphinx
* changed from rst to markdown and extended sphinx for it
* fix spelling
* Clean titles
* delete file
* change spelling
* add sections to document store usage
* add basic rest api docs
* fix readme in setup.py
* Update Tutorials
* Change section names
* add windows note to pip install
* update intro
* new renderer for markdown files
* Fix typos
* delete dpr_utils.py
* fix windows note in get started
* Fix docstrings
* deleted rest api docs in api
* fixed typo
* Fix docstring
* revert readme to rst
* Fix readme
* Update setup.py
Co-authored-by: deepset <deepset@Crenolape.localdomain>
Co-authored-by: PiffPaffM <markuspaff.mp@gmail.com>
Co-authored-by: Bogdan Kostić <bogdankostic@web.de>
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
* bump farm version to 0.4.8
* move back to original transformers pipeline
* remove dpr_utils and use transformers implementation
* update tutorial notebooks
* Add aggregate labels obj, use in retriever eval function
* Change launch ES param
* Move aggregation from ES document store to base class
* Fix type annotations
