mirror of
https://github.com/deepset-ai/haystack.git
synced 2025-07-23 17:00:41 +00:00
13510aa753
* Files moved, imports all broken * Fix most imports and docstrings into * Fix the paths to the modules in the API docs * Add latest docstring and tutorial changes * Add a few pipelines that were lost in the inports * Fix a bunch of mypy warnings * Add latest docstring and tutorial changes * Create a file_classifier module * Add docs for file_classifier * Fixed most circular imports, now the REST API can start * Add latest docstring and tutorial changes * Tackling more mypy issues * Reintroduce from FARM and fix last mypy issues hopefully * Re-enable old-style imports * Fix some more import from the top-level package in an attempt to sort out circular imports * Fix some imports in tests to new-style to prevent failed class equalities from breaking tests * Change document_store into document_stores * Update imports in tutorials * Add latest docstring and tutorial changes * Probably fixes summarizer tests * Improve the old-style import allowing module imports (should work) * Try to fix the docs * Remove dedicated KnowledgeGraph page from autodocs * Remove dedicated GraphRetriever page from autodocs * Fix generate_docstrings.sh with an updated list of yaml files to look for * Fix some more modules in the docs * Fix the document stores docs too * Fix a small issue on Tutorial14 * Add latest docstring and tutorial changes * Add deprecation warning to old-style imports * Remove stray folder and import Dict into dense.py * Change import path for MLFlowLogger * Add old loggers path to the import path aliases * Fix debug output of convert_ipynb.py * Fix circular import on BaseRetriever * Missed one merge block * re-run tutorial 5 * Fix imports in tutorial 5 * Re-enable squad_to_dpr CLI from the root package and move get_batches_from_generator into document_stores.base * Add latest docstring and tutorial changes * Fix typo in utils __init__ * Fix a few more imports * Fix benchmarks too * New-style imports in test_knowledge_graph * Rollback setup.py * Rollback squad_to_dpr too Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
66 lines
4.1 KiB
Python
66 lines
4.1 KiB
Python
from tqdm import tqdm
|
|
from pprint import pprint
|
|
from haystack.nodes import QuestionGenerator, ElasticsearchRetriever, FARMReader
|
|
from haystack.document_stores import ElasticsearchDocumentStore
|
|
from haystack.pipelines import QuestionGenerationPipeline, RetrieverQuestionGenerationPipeline, QuestionAnswerGenerationPipeline
|
|
from haystack.utils import launch_es
|
|
|
|
"""
|
|
This is a bare bones tutorial showing what is possible with the QuestionGenerator Node which automatically generates
|
|
questions which the model thinks can be answered by a given document.
|
|
"""
|
|
|
|
# Start Elasticsearch service via Docker
|
|
launch_es()
|
|
|
|
text1 = "Python is an interpreted, high-level, general-purpose programming language. Created by Guido van Rossum and first released in 1991, Python's design philosophy emphasizes code readability with its notable use of significant whitespace."
|
|
text2 = "Princess Arya Stark is the third child and second daughter of Lord Eddard Stark and his wife, Lady Catelyn Stark. She is the sister of the incumbent Westerosi monarchs, Sansa, Queen in the North, and Brandon, King of the Andals and the First Men. After narrowly escaping the persecution of House Stark by House Lannister, Arya is trained as a Faceless Man at the House of Black and White in Braavos, using her abilities to avenge her family. Upon her return to Westeros, she exacts retribution for the Red Wedding by exterminating the Frey male line."
|
|
text3 = "Dry Cleaning are an English post-punk band who formed in South London in 2018.[3] The band is composed of vocalist Florence Shaw, guitarist Tom Dowse, bassist Lewis Maynard and drummer Nick Buxton. They are noted for their use of spoken word primarily in lieu of sung vocals, as well as their unconventional lyrics. Their musical stylings have been compared to Wire, Magazine and Joy Division.[4] The band released their debut single, 'Magic of Meghan' in 2019. Shaw wrote the song after going through a break-up and moving out of her former partner's apartment the same day that Meghan Markle and Prince Harry announced they were engaged.[5] This was followed by the release of two EPs that year: Sweet Princess in August and Boundary Road Snacks and Drinks in October. The band were included as part of the NME 100 of 2020,[6] as well as DIY magazine's Class of 2020.[7] The band signed to 4AD in late 2020 and shared a new single, 'Scratchcard Lanyard'.[8] In February 2021, the band shared details of their debut studio album, New Long Leg. They also shared the single 'Strong Feelings'.[9] The album, which was produced by John Parish, was released on 2 April 2021.[10]"
|
|
|
|
docs = [{"content": text1},
|
|
{"content": text2},
|
|
{"content": text3}]
|
|
|
|
# Initialize document store and write in the documents
|
|
document_store = ElasticsearchDocumentStore()
|
|
document_store.write_documents(docs)
|
|
|
|
# Initialize Question Generator
|
|
question_generator = QuestionGenerator()
|
|
|
|
"""
|
|
The most basic version of a question generator pipeline takes a document as input and outputs generated questions
|
|
which the the document can answer.
|
|
"""
|
|
|
|
# QuestionGenerationPipeline
|
|
question_generation_pipeline = QuestionGenerationPipeline(question_generator)
|
|
for document in document_store:
|
|
result = question_generation_pipeline.run(documents=[document])
|
|
pprint(result)
|
|
|
|
"""
|
|
This pipeline takes a query as input. It retrievers relevant documents and then generates questions based on these.
|
|
"""
|
|
|
|
# RetrieverQuestionGenerationPipeline
|
|
retriever = ElasticsearchRetriever(document_store=document_store)
|
|
rqg_pipeline = RetrieverQuestionGenerationPipeline(retriever, question_generator)
|
|
result = rqg_pipeline.run(query="Arya Stark")
|
|
pprint(result)
|
|
|
|
"""
|
|
This pipeline takes a document as input, generates questions on it, and attempts to answer these questions using
|
|
a Reader model
|
|
"""
|
|
|
|
# QuestionAnswerGenerationPipeline
|
|
reader = FARMReader("deepset/roberta-base-squad2")
|
|
qag_pipeline = QuestionAnswerGenerationPipeline(question_generator, reader)
|
|
for document in tqdm(document_store):
|
|
result = qag_pipeline.run(documents=[document])
|
|
pprint(result)
|
|
|
|
# This Haystack script was made with love by deepset in Berlin, Germany
|
|
# Haystack: https://github.com/deepset-ai/haystack
|
|
# deepset: https://deepset.ai/ |