mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-07-26 02:10:41 +00:00
f2b6cc761b
* change_HFBertEncoder to transformers DPREncoder * Removed BertTensorizer * model download relative path * Refactor model load * Tutorial5 DPR updated * fix print_eval_results typo * copy transformers DPR modules in dpr_utils and test * transformer v3.0.2 import errors fixed * remove dependency of DPRConfig on attribute use_return_tuple * Adjust transformers 302 locally to work with dpr * projection layer removed from DPR encoders * fixed mypy errors * transformers DPR compatible code added * transformers DPR compatibility added * bug fix in tutorial 6 notebook * Docstring update and variable naming issues fix * tutorial modified to reflect DPR variable naming change * title addition to passage use-cases handled * modified handling untitled batch * resolved mypy errors * typos in docstrings and comments fixed * cleaned DPR code and added new test cases * warnings added for non-bert model [SEP] token removal * changed warning to logger warning * title mask creation refactored * bug fix on cuda issues * tutorial 6 instantiates modified DPR * tutorial 5 modified * tutorial 5 ipython notebook modified: DPR instantiation * batch_size added to DPR instantiation * tutorial 5 jupyter notebook typos fixed * improved docstrings, fixed typos * Update docstring Co-authored-by: Timo Moeller <timo.moeller@deepset.ai> Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai>
62 lines
3.0 KiB
Python
Executable File
62 lines
3.0 KiB
Python
Executable File
from haystack import Finder
|
|
from haystack.database.faiss import FAISSDocumentStore
|
|
from haystack.indexing.cleaning import clean_wiki_text
|
|
from haystack.indexing.utils import convert_files_to_dicts, fetch_archive_from_http
|
|
from haystack.reader.farm import FARMReader
|
|
from haystack.utils import print_answers
|
|
from haystack.retriever.dense import DensePassageRetriever
|
|
|
|
|
|
# FAISS is a library for efficient similarity search on a cluster of dense vectors.
|
|
# The FAISSDocumentStore uses a SQL(SQLite in-memory be default) database under-the-hood
|
|
# to store the document text and other meta data. The vector embeddings of the text are
|
|
# indexed on a FAISS Index that later is queried for searching answers.
|
|
document_store = FAISSDocumentStore()
|
|
|
|
# ## Cleaning & indexing documents
|
|
# Let's first get some documents that we want to query
|
|
doc_dir = "data/article_txt_got"
|
|
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"
|
|
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)
|
|
|
|
# convert files to dicts containing documents that can be indexed to our datastore
|
|
dicts = convert_files_to_dicts(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)
|
|
|
|
# Now, let's write the docs to our DB.
|
|
document_store.write_documents(dicts)
|
|
|
|
### Retriever
|
|
retriever = DensePassageRetriever(document_store=document_store,
|
|
query_embedding_model="facebook/dpr-question_encoder-single-nq-base",
|
|
passage_embedding_model="facebook/dpr-ctx_encoder-single-nq-base",
|
|
use_gpu=True,
|
|
embed_title=True,
|
|
remove_sep_tok_from_untitled_passages=True)
|
|
|
|
# Important:
|
|
# Now that after we have the DPR initialized, we need to call update_embeddings() to iterate over all
|
|
# previously indexed documents and update their embedding representation.
|
|
# While this can be a time consuming operation (depending on corpus size), it only needs to be done once.
|
|
# At query time, we only need to embed the query and compare it the existing doc embeddings which is very fast.
|
|
document_store.update_embeddings(retriever)
|
|
|
|
### Reader
|
|
# Load a local model or any of the QA models on
|
|
# Hugging Face's model hub (https://huggingface.co/models)
|
|
reader = FARMReader(model_name_or_path="deepset/roberta-base-squad2", use_gpu=True)
|
|
|
|
### Finder
|
|
# The Finder sticks together reader and retriever in a pipeline to answer our actual questions.
|
|
finder = Finder(reader, retriever)
|
|
|
|
### Voilà! Ask a question!
|
|
# You can configure how many candidates the reader and retriever shall return
|
|
# The higher top_k_retriever, the better (but also the slower) your answers.
|
|
prediction = finder.get_answers(question="Who is the father of Arya Stark?", top_k_retriever=10, top_k_reader=5)
|
|
|
|
|
|
# prediction = finder.get_answers(question="Who created the Dothraki vocabulary?", top_k_reader=5)
|
|
# prediction = finder.get_answers(question="Who is the sister of Sansa?", top_k_reader=5)
|
|
|
|
print_answers(prediction, details="minimal")
|