mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-01-07 20:46:31 +00:00
51acf779f2
* first draft / notes on new primitives * wip label / feedback refactor * rename doc.text -> doc.content. add doc.content_type * add datatype for content * remove faq_question_field from ES and weaviate. rename text_field -> content_field in docstores. update tutorials for content field * update converters for . Add warning for empty * renam label.question -> label.query. Allow sorting of Answers. * WIP primitives * update ui/reader for new Answer format * Improve Label. First refactoring of MultiLabel. Adjust eval code * fixed workflow conflict with introducing new one (#1472) * Add latest docstring and tutorial changes * make add_eval_data() work again * fix reader formats. WIP fix _extract_docs_and_labels_from_dict * fix test reader * Add latest docstring and tutorial changes * fix another test case for reader * fix mypy in farm reader.eval() * fix mypy in farm reader.eval() * WIP ORM refactor * Add latest docstring and tutorial changes * fix mypy weaviate * make label and multilabel dataclasses * bump mypy env in CI to python 3.8 * WIP refactor Label ORM * WIP refactor Label ORM * simplify tests for individual doc stores * WIP refactoring markers of tests * test alternative approach for tests with existing parametrization * WIP refactor ORMs * fix skip logic of already parametrized tests * fix weaviate behaviour in tests - not parametrizing it in our general test cases. * Add latest docstring and tutorial changes * fix some tests * remove sql from document_store_types * fix markers for generator and pipeline test * remove inmemory marker * remove unneeded elasticsearch markers * add dataclasses-json dependency. adjust ORM to just store JSON repr * ignore type as dataclasses_json seems to miss functionality here * update readme and contributing.md * update contributing * adjust example * fix duplicate doc handling for custom index * Add latest docstring and tutorial changes * fix some ORM issues. fix get_all_labels_aggregated. * update drop flags where get_all_labels_aggregated() was used before * Add latest docstring and tutorial changes * add to_json(). add + fix tests * fix no_answer handling in label / multilabel * fix duplicate docs in memory doc store. change primary key for sql doc table * fix mypy issues * fix mypy issues * haystack/retriever/base.py * fix test_write_document_meta[elastic] * fix test_elasticsearch_custom_fields * fix test_labels[elastic] * fix crawler * fix converter * fix docx converter * fix preprocessor * fix test_utils * fix tfidf retriever. fix selection of docstore in tests with multiple fixtures / parameterizations * Add latest docstring and tutorial changes * fix crawler test. fix ocrconverter attribute * fix test_elasticsearch_custom_query * fix generator pipeline * fix ocr converter * fix ragenerator * Add latest docstring and tutorial changes * fix test_load_and_save_yaml for elasticsearch * fixes for pipeline tests * fix faq pipeline * fix pipeline tests * Add latest docstring and tutorial changes * Add MultimodalRetriever * Add latest docstring and tutorial changes * fix weaviate * Add latest docstring and tutorial changes * trigger CI * satisfy mypy * Add latest docstring and tutorial changes * satisfy mypy * Add latest docstring and tutorial changes * trigger CI * fix question generation test * fix ray. fix Q-generation * fix translator test * satisfy mypy * wip refactor feedback rest api * fix rest api feedback endpoint * fix doc classifier * remove relation of Labels -> Docs in SQL ORM * fix faiss/milvus tests * fix doc classifier test * fix eval test * fixing eval issues * Add latest docstring and tutorial changes * fix mypy * WIP replace dataclasses-json with manual serialization * Add methods to MultimodalRetriever * Add latest docstring and tutorial changes * revert to dataclass-json serialization for now. remove debug prints. * update docstrings * fix extractor. fix Answer Span init * fix api test * keep meta data of answers in reader.run() * fix meta handling * adress review feedback * Add latest docstring and tutorial changes * make document=None for open domain labels * add import * fix print utils * fix rest api * Add methods and tests * Add latest docstring and tutorial changes * Fix mypy * Add latest docstring and tutorial changes * Add type hints and doc strings * Make use of initialize_device_settings * Move serialization of pd.DataFrame to schema.py * Fix mypy * Adapt Document's from_dict method * Update docstrings * Add latest docstring and tutorial changes * Fix mypy * Fix mypy * Fix Document's from_dict method * Fix Document's to_dict method * Change handling of table metadata * Add latest docstring and tutorial changes * Change naming from Multimodal to TableText * Turn off tokenizers_parallelism in retriever tests * Add latest docstring and tutorial changes * Remove turning off tokenizers_parallelism in retriever tests * Adapt convert_es_hit_to_document * Change embed_surrounding_context to embed_meta_fields * Add latest docstring and tutorial changes * Add check if torch.distributed is available * Set n_gpu to 0 in training test * Set HIP_LAUNCH_BLOCKING to 1 * Set HIP_LAUNCH_BLOCKING to "1" * Set use_gpu to False * Use DataParallel only if more than one device * Remove --find-links=https://download.pytorch.org/whl/torch_stable.html Co-authored-by: Malte Pietsch <malte.pietsch@deepset.ai> Co-authored-by: Markus Paff <markuspaff.mp@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
65 lines
1.5 KiB
Plaintext
65 lines
1.5 KiB
Plaintext
# basics
|
|
setuptools
|
|
wheel
|
|
# PyTorch
|
|
# Temp. disabled the next line as it gets currently resolved to https://download.pytorch.org/whl/rocm3.8/torch-1.7.1%2Brocm3.8-cp38-cp38-linux_x86_64.whl
|
|
# --find-links=https://download.pytorch.org/whl/torch_stable.html
|
|
torch>1.5,<1.10
|
|
# progress bars in model download and training scripts
|
|
tqdm
|
|
# Used for downloading models over HTTP
|
|
requests
|
|
# Scipy & sklearn for stats in run_classifier
|
|
scipy>=1.3.2
|
|
sklearn
|
|
# Metrics or logging related
|
|
seqeval
|
|
mlflow<=1.13.1
|
|
# huggingface repository
|
|
transformers==4.7.0
|
|
# pickle extension for (de-)serialization
|
|
dill
|
|
# Inference with ONNX models. Install onnxruntime-gpu for Inference on GPUs
|
|
# onnxruntime
|
|
# onnxruntime_tools
|
|
psutil
|
|
# haystack
|
|
fastapi
|
|
uvicorn
|
|
gunicorn
|
|
pandas
|
|
psycopg2-binary; sys_platform != 'win32' and sys_platform != 'cygwin'
|
|
elasticsearch>=7.7,<=7.10
|
|
elastic-apm
|
|
tox
|
|
coverage
|
|
langdetect # for PDF conversions
|
|
# for PDF conversions using OCR
|
|
pytesseract==0.3.7
|
|
pillow==8.3.2
|
|
pdf2image==1.14.0
|
|
sentence-transformers>=0.4.0
|
|
python-multipart
|
|
python-docx
|
|
sqlalchemy>=1.4.2
|
|
sqlalchemy_utils
|
|
# for using FAISS with GPUs, install faiss-gpu
|
|
faiss-cpu>=1.6.3
|
|
tika
|
|
uvloop==0.14; sys_platform != 'win32' and sys_platform != 'cygwin'
|
|
httptools
|
|
nltk
|
|
more_itertools
|
|
networkx
|
|
# Refer milvus version support matrix at https://github.com/milvus-io/pymilvus#install-pymilvus
|
|
# For milvus 2.x version use this library `pymilvus===2.0.0rc6`
|
|
pymilvus
|
|
# Optional: For crawling
|
|
#selenium
|
|
#webdriver-manager
|
|
SPARQLWrapper
|
|
mmh3
|
|
weaviate-client==2.5.0
|
|
ray==1.5.0
|
|
dataclasses-json
|
|
quantulum3 |