* initial test cml
* Update cml.yaml
* WIP test workflow
* switch to general ubuntu ami
* switch to general ubuntu ami
* disable gpu for tests
* rm gpu infos
* rm gpu infos
* update token env
* switch github token
* add postgres
* test db connection
* fix typo
* remove tty
* add sleep for db
* debug runner
* debug removal postgres
* debug: reset to working commit
* debug: change github token
* switch to new bot token
* debug token
* add back postgres
* adjust network runner docker
* add elastic
* fix typo
* adjust working dir
* fix benchmark execution
* enable s3 downloads
* add query benchmark. fix path
* add saving of markdown files
* cat md files. add faiss+dpr. increase n_queries
* switch to GPU instance
* switch availability zone
* switch to public aws DL ami
* increase volume size
* rm faiss. fix error logging
* save markdown files
* add reader benchmarks
* add download of squad data
* correct reader metric normalization
* fix newlines between reports
* fix max_docs for reader eval data. remove max_docs from ci run config
* fix mypy. switch workflow trigger
* try trigger for label
* try trigger for label
* change trigger syntax
* debug machine shutdown with test workflow
* add es and postgres to test workflow
* Revert "add es and postgres to test workflow"
This reverts commit 6f038d3d7f12eea924b54529e61b192858eaa9d5.
* Revert "debug machine shutdown with test workflow"
This reverts commit db70eabae8850b88e1d61fd79b04d4f49d54990a.
* fix typo in action. set benchmark config back to original
* Add languages and preprocessing pages
* add content
* address review comments
* make link relative
* update api ref with latest docstrings
* move doc readme and update
* add generator API docs
* fix example code
* design and link fix
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
Co-authored-by: PiffPaffM <markuspaff.mp@gmail.com>
* Add save and load method for DPR
* lower memory footprint for test. change names to load() and save()
* add test cases
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
* Adding dummy generator implementation
* Adding tutorial to try the model
* Committing current non working code
* Committing current update where we need to call generate function directly and need to convert embedding to tensor way
* Addressing review comments.
* Refactoring finder, and implementing rag_generator class.
* Refined the implementation of RAGGenerator and now it is in clean shape
* Renaming RAGGenerator to RAGenerator
* Reverting change from finder.py and addressing review comments
* Remove support for RagSequenceForGeneration
* Utilizing embed_passage function from DensePassageRetriever
* Adding sample test data to verify generator output
* Updating testing script
* Updating testing script
* Fixing bug related to top_k
* Updating latest farm dependency
* Comment out farm dependency
* Reverting changes from TransformersReader
* Adding transformers dataset to compare transformers and haystack generator implementation
* Using generator_encoder instead of question_encoder to generate context_input_ids
* Adding workaround to install FARM dependency from master branch
* Removing unnecessary changes
* Fixing generator test
* Removing transformers datasets
* Fixing generator test
* Some cleanup and updating TODO comments
* Adding tutorial notebook
* Updating tutorials with comments
* Explicitly passing token model in RAG test
* Addressing review comments
* Fixing notebook
* Refactoring tests to reduce memory footprint
* Split generator tests in separate ci step and before running it reclaim memory by terminating containers
* Moving tika dependent test to separate dir
* Remove unwanted code
* Brining reader under session scope
* Farm is now session object hence restoring changes from default value
* Updating assert for pdf converter
* Dummy commit to trigger CI flow
* REducing memory footprint required for generator tests
* Fixing mypy issues
* Marking test with tika and elasticsearch markers. Reverting changes in CI and pytest splits
* reducing changes
* Fixing CI
* changing elastic search ci
* Fixing test error
* Disabling return of embedding
* Marking generator test as well
* Refactoring tutorials
* Increasing ES memory to 750M
* Trying another fix for ES CI
* Reverting CI changes
* Splitting tests in CI
* Generator and non-generator markers split
* Adding pytest.ini to add markers and enable strict-markers option
* Reducing elastic search container memory
* Simplifying generator test by using documents with embedding directly
* Bump up farm to 0.5.0
* Adding support to return embedding along with other result via query_by_embedding function
* Adding test case to check return embedding
* By default for all tests but DPR tests: disable return_embedding flag
* Reducing None test case and fixing query_by_embedding of ElasticsearchDocumentStore when it updating self.excluded_meta_data directly
* Fixing mypy reported issue
* add time and perf benchmark for es
* Add retriever benchmarking
* Add Reader benchmarking
* add nq to squad conversion
* add conversion stats
* clean benchmarks
* Add link to dataset
* Update imports
* add first support for neg psgs
* Refactor test
* set max_seq_len
* cleanup benchmark
* begin retriever speed benchmarking
* Add support for retriever query index benchmarking
* improve reader eval, retriever speed benchmarking
* improve retriever speed benchmarking
* Add retriever accuracy benchmark
* Add neg doc shuffling
* Add top_n
* 3x speedup of SQL. add postgres docker run. make shuffle neg a param. add more logging
* Add models to sweep
* add option for faiss index type
* remove unneeded line
* change faiss to faiss_flat
* begin automatic benchmark script
* remove existing postgres docker for benchmarking
* Add data processing scripts
* Remove shuffle in script bc data already shuffled
* switch hnsw setup from 256 to 128
* change es similarity to dot product by default
* Error includes stack trace
* Change ES default timeout
* remove delete_docs() from timing for indexing
* Add support for website export
* update website on push to benchmarks
* add complete benchmarks results
* new json format
* removed NaN as is not a valid json token
* versioning for docs
* unsaved changes
* cleaning
* cleaning
* Edit format of benchmarks data
* update also jsons in v0.4.0
Co-authored-by: brandenchan <brandenchan@icloud.com>
Co-authored-by: deepset <deepset@Crenolape.localdomain>
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
* 1. Prevent update_embeddings function in FAISSDocumentStore to set faiss_index as None when document store does not have any docs.
2. cleaning up tests by adding fixture for retriever.
* TfidfRetriever need document store with documents during initialization as it call fit() function in constructor so fixing it by checking self.paragraphs of None
* Fix naming of retriever's fixture (embedded to embedding and tfid to tfidf)
* add time and perf benchmark for es
* Add retriever benchmarking
* Add Reader benchmarking
* add nq to squad conversion
* add conversion stats
* clean benchmarks
* Add link to dataset
* Update imports
* add first support for neg psgs
* Refactor test
* set max_seq_len
* cleanup benchmark
* begin retriever speed benchmarking
* Add support for retriever query index benchmarking
* improve reader eval, retriever speed benchmarking
* improve retriever speed benchmarking
* Add retriever accuracy benchmark
* Add neg doc shuffling
* Add top_n
* 3x speedup of SQL. add postgres docker run. make shuffle neg a param. add more logging
* Add models to sweep
* add option for faiss index type
* remove unneeded line
* change faiss to faiss_flat
* begin automatic benchmark script
* remove existing postgres docker for benchmarking
* Add data processing scripts
* Remove shuffle in script bc data already shuffled
* switch hnsw setup from 256 to 128
* change es similarity to dot product by default
* Error includes stack trace
* Change ES default timeout
* remove delete_docs() from timing for indexing
* Add support for website export
* update website on push to benchmarks
* add complete benchmarks results
* new json format
* removed NaN as is not a valid json token
* fix benchmarking for faiss hnsw queries. do sql calls in update_embeddings() as batches
* update benchmarks for hnsw 128,20,80
* don't delete full index in delete_all_documents()
* update texts for charts
* update recall column for retriever
* change scale and add units to desc
* add units to legend
* add axis titles. update desc
* add html tags
Co-authored-by: deepset <deepset@Crenolape.localdomain>
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
Co-authored-by: PiffPaffM <markuspaff.mp@gmail.com>
* remove phi normalization
* add special case for hnsw
* rename vector_size to vector_dim
* fix loading. fix extra dim in tests
* switch to new ES syntax for vector similarity
* 3x sql speed up. cascade deletes. add train_index()
* add docstrings. remove vector_dim from load()
* delete docs from faiss and sql
* fix delete of docs in test
* relax type hint for faiss index
* rename metric to metric_type
Co-authored-by: lalitpagaria <19303690+lalitpagaria@users.noreply.github.com>
- Fixing issue when update_embeddings always create new FAISS index instead of clearing existing one. New index creation may not free existing used memory and cause memory leak.
Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
* Add tests for MultiLabel
* Add test for no_answer and is_correct_answer=False + fix bug in MultiLabel aggregation
* Fix bug in MultiLabel aggregation
