yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-12-31 09:10:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,866 Commits 101 Branches 267 Tags
Commit Graph

425 Commits

Author SHA1 Message Date
Silvano Cerza
db759b0717
Add black step when testing examples (#6425) 2023-11-27 15:01:33 +01:00
Malte Pietsch
09b4f53ce5
docs: Add example for loop in pipeline to autocorrect JSON (#6418) 
* add example for pipeline loop

* add pydantic to CI

* Fix comment

---------

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
 2023-11-27 13:29:16 +01:00
Silvano Cerza
892625a6c7
ci: Add back workflows that runs in place of linting.yml and tests.yml (#6421) 
* Add back workflow that runs in place of linting.yml

* Add back workflow that runs in place of tests.yml
 2023-11-27 13:18:47 +01:00
Silvano Cerza
8bfaf0a56a
ci: Add catch-all job in tests.yml (#6419) 
* Add catch-all job in tests.yml

* Trigger for testing

* Remove trigger for testing
 2023-11-27 12:57:33 +01:00
Massimiliano Pippi
5a256df7cb
use different token 2023-11-24 17:41:32 +01:00
Silvano Cerza
9338de1790 Add missing tests workflow dependency 2023-11-24 16:00:59 +01:00
Massimiliano Pippi
763d2d8e4c
remove rest_api 2023-11-24 15:49:54 +01:00
Massimiliano Pippi
d3ab8afede
clean up labeller 2023-11-24 15:30:06 +01:00
Massimiliano Pippi
4a1fe163b6
fix names in workflows 2023-11-24 14:59:31 +01:00
Massimiliano Pippi
09e7831f60
clean up 1.x code 
---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
 2023-11-24 11:47:47 +01:00
ZanSara
4ec6a60a76
feat: CohereGenerator (#6395) 
* added CohereGenerator with unit tests

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* 1. added releasenote
2. removed commented files in test-cohere_generators
3. removed unused imports

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* 1. move client creation to __init__
2. remove dict casting of metadata in run

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* few fixes

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* add cohere to git workflows

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* 1. CohereGenerator as top level import in generators
2. small change in doc string

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* 1. corrected git workflow files for cohere import
2. changed api key env var from CO_API_KEY to COHERE_API_KEY

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* added cohere in missed out workflow installs

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* 1. Removed default_streaming_callback from cohere.py and added in test.
2. Added kwargs doc strings for CohereGenerator
3. removed type hints for metadata and replies
4. use COHERE_API_URL instead of hard coded URL.

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>

* Update haystack/preview/components/generators/cohere/cohere.py

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* Update haystack/preview/components/generators/cohere/cohere.py

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* Update haystack/preview/components/generators/cohere/cohere.py

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* Update haystack/preview/components/generators/cohere/cohere.py

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* Update haystack/preview/components/generators/cohere/cohere.py

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* move out of folder

* black

* fix tests

* feedback

* black

* remove api key from tests

* read api key from env var if missing

* typo

* black

* missing import

---------

Signed-off-by: sunilkumardash9 <sunilkumardash9@gmail.com>
Co-authored-by: sunilkumardash9 <sunilkumardash9@gmail.com>
Co-authored-by: Daria Fokina <daria.f93@gmail.com>
 2023-11-23 17:21:07 +01:00
Silvano Cerza
3e79de7043
ci: Add workflow to test code snippets (#6364) 
* initial

* Add workflow to test code snippets

---------

Co-authored-by: Timo Möller <timo.moeller@deepset.ai>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
 2023-11-23 11:26:53 +01:00
Silvano Cerza
c26a932423
Change preview tests to run all tests except integration ones (#6325) 2023-11-17 15:33:43 +01:00
Stefano Fiorucci
dd6e35d675
build: upgrade to transformers==4.35.2 (#6322) 
* upgrade transformers to 4.35.2

* reno
 2023-11-17 10:12:34 +01:00
Julian Risch
1c85e44156
test: Add langdetect installation to e2e tests (#6327) 
* Add langdetect installation to e2e tests

* compare doc content and id only
 2023-11-17 10:12:05 +01:00
Julian Risch
34ecff1d19
build: Upgrade openai-whisper and re-introduce audio extra (#6319) 
* upgrade openai-whisper and re-introduce audio extra

* add audio extra to
 2023-11-16 15:04:50 +01:00
Vivek Silimkhan
f998bf4a4f
feat: add Amazon Bedrock support (#6226) 
* Add Bedrock

* Update supported models for Bedrock

* Fix supports and add extract response in Bedrock

* fix errors imports

* improve and refactor supports

* fix install

* fix mypy

* fix pylint

* fix existing tests

* Added Anthropic Bedrock

* fix tests

* fix sagemaker tests

* add default prompt handler, constructor and supports tests

* more tests

* invoke refactoring

* refactor model_kwargs

* fix mypy

* lstrip responses

* Add streaming support

* bump boto3 version

* add class docstrings, better exception names

* fix layer name

* add tests for anthropic and cohere model adapters

* update cohere params

* update ai21 args and add tests

* support cohere command light model

* add tital tests

* better class names

* support meta llama 2 model

* fix streaming support

* more future-proof model adapter selection

* fix import

* fix mypy

* fix pylint for preview

* add tests for streaming

* add release notes

* Apply suggestions from code review

Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>

* fix format

* fix tests after msg changes

* fix streaming for cohere

---------

Co-authored-by: tstadel <60758086+tstadel@users.noreply.github.com>
Co-authored-by: tstadel <thomas.stadelmann@deepset.ai>
Co-authored-by: Agnieszka Marzec <97166305+agnieszka-m@users.noreply.github.com>
 2023-11-15 13:26:29 +01:00
Massimiliano Pippi
f019896335
ci: Generate release notes in a Github workflow (#6211) 
* first try

* Update config.yaml

* Update github_release.yml

* set the rc0 tag more explicitly
 2023-11-08 12:29:37 +01:00
Massimiliano Pippi
58e357148e
ci: tag when branching off for a release (#6206) 
* tag when branching off

* change minor bump workflow

* Update .github/workflows/minor_version_release.yml

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update minor_version_release.yml

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
 2023-11-08 11:06:45 +01:00
Ashwin Mathur
6bf0b9dc7c
feat: Add MarkdownToTextDocument (v2) (#6159) 
* Add MarkdownToTextDocument

* Add release notes

* Update GitHub workflows

* Update GitHub workflows

* Refactor code with minimal dependencies

* Update docstrings

* Apply suggestions from code review

Co-authored-by: Daria Fokina <daria.f93@gmail.com>

* Update document with content and meta for backward compatibility

* Refactor Document Class for Backward Compatibility

Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>

* Update tests

* Improve test assertions

---------

Co-authored-by: Daria Fokina <daria.f93@gmail.com>
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
 2023-10-31 18:28:13 +01:00
Massimiliano Pippi
209e349be3
do not run preview tests twice (#6204) 2023-10-31 13:13:32 +01:00
Nripesh Niketan
708d33a657
feat: add apple silicon GPU acceleration (#6151) 
* feat: add apple silicon GPU acceleration

* add release notes

* small fix

* Update utils.py

* Update utils.py

* ci fix mps

* Revert "ci fix mps"

This reverts commit 783ae503940d9ff8270a970a321549fb9e69dce7.

* mps fix

* Update experiment_tracking.py

* try removing upper watermark limit

* disable mps CI

* Use xl runner

* initialise env

* small fix

* black linting

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
 2023-10-30 11:26:46 +01:00
dependabot[bot]
8a2a3c9a3f
build(deps): bump tj-actions/changed-files from 39 to 40 (#6175) 
Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 39 to 40.
- [Release notes](https://github.com/tj-actions/changed-files/releases)
- [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md)
- [Commits](https://github.com/tj-actions/changed-files/compare/v39...v40)

---
updated-dependencies:
- dependency-name: tj-actions/changed-files
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-27 16:58:38 +02:00
Julian Risch
db36d6277a
docs: Add readme sync for API docs 2.0 (#6173) 
* add sync docs for preview

* add example config for audio docs

* hardcode version in renderer

* use custom renderer for preview docs

* update comment and excerpt
 2023-10-27 14:53:03 +02:00
Massimiliano Pippi
7c07fb3290
Update labeler.yml (#6169) 2023-10-26 09:50:43 +02:00
Grant Williams
1cf70d3dce
build: Upgrade transformers to the latest version 4.34.1 (#5994) 
* Upgrade transformers to the latest version 4.34.0 so that Haystack can support the new Mistral, Nougat, and other models.

* update release notes

* updated missing lazy import

* Update .github workflows imports

* bump more versions in .github workflows

* rever import sorting

* Update  to catch runtime errors to match haystack_hub changes

* add language parameter value to whisper test

* bump transformers version in linting preview workflow

* bump transformers version in linting preview workflow

* bump version to v4.34.1

* resolve mypy issue with reused variables

* install openai-whisper without dependencies

* remove audio extra, update whisper install instructions

* remove audio extra, update whisper install instructions

* keep audio extra but add version

* keep audio extra with no constraints

* remove audio extra

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
 2023-10-24 19:13:12 +02:00
Stefano Fiorucci
c4187eeebe
CI: make only test_preview run when preview e2e tests are changed (#6078) 
* make only test_preview workflow run when e2e tests are modified

* revert wrong changes to test_preview

* revert wrong order
 2023-10-17 10:06:39 +02:00
Stefano Fiorucci
167700de4d
CI: make only linting_preview run on preview e2e tests (#6077) 
* apply only linting_preview to preview e2e tests

* add paths to linting_skipper
 2023-10-16 18:18:17 +02:00
Silvano Cerza
c78e1a7eb3
Add a workflow to verify haystack.preview doesn't import non preview modules (#6053) 2023-10-16 09:36:45 +02:00
ZanSara
adf7e49af3
chore: review all extra (#6029) 2023-10-12 21:50:53 +02:00
Nicola Procopio
c102b152dc
fix: Run update_embeddings in examples (#6008) 
* added hybrid search example

Added an example about hybrid search for faq pipeline on covid dataset

* formatted with back formatter

* renamed document

* fixed

* fixed typos

* added test

added test for hybrid search

* fixed withespaces

* removed test for hybrid search

* fixed pylint

* commented logging

* updated hybrid search example

* release notes

* Update hybrid_search_faq_pipeline.py-815df846dca7e872.yaml

* Update hybrid_search_faq_pipeline.py

* mention hybrid search example in release notes

* reduce installed dependencies in examples test workflow

* do not install cuda dependencies

* skip models if API key not set; delete document indices

* skip models if API key not set; delete document indices

* skip models if API key not set; delete document indices

* keep roberta-base model and inference extra

* pylint

* disable pylint no-logging-basicconfig rule

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
 2023-10-10 16:38:52 +02:00
Vladimir Blagojevic
c05f564359
feat: Split linting preview into a separate file (#6017) 
* Split linting preview into seperate file

* Add not trigger paths in old workflow
 2023-10-10 14:54:27 +02:00
dependabot[bot]
a4beec3013
build(deps): bump aws-actions/configure-aws-credentials (#5968) 
Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 4.0.0 to 4.0.1.
- [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases)
- [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md)
- [Commits](8c3f20df09...010d0da01d)

---
updated-dependencies:
- dependency-name: aws-actions/configure-aws-credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-04 17:20:17 +02:00
dependabot[bot]
58192d35f1
build(deps): bump iterative/setup-cml from 1 to 2 (#5911) 
Bumps [iterative/setup-cml](https://github.com/iterative/setup-cml) from 1 to 2.
- [Release notes](https://github.com/iterative/setup-cml/releases)
- [Commits](https://github.com/iterative/setup-cml/compare/v1...v2)

---
updated-dependencies:
- dependency-name: iterative/setup-cml
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-03 17:39:22 +02:00
ZanSara
b844ab8e22
chore: remove matrix from Linux CI (#5955) 
* remove matrix

* workflow names
 2023-10-03 17:39:04 +02:00
Silvano Cerza
a933a42749 Fix release_notes.yml syntax 2023-10-02 13:24:08 -07:00
Zubeen
b8c3b68141
Update release_notes.yml (#5949) 
Ignoring release notes check for PRs of type doc/ci/test
 2023-10-02 22:17:55 +02:00
dependabot[bot]
69232612d0
build(deps): bump actions/checkout from 3 to 4 (#5928) 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-01 12:38:57 +02:00
ZanSara
81b2e83d04
feat: separate out preview tests (#5639) 
* add preview workflows

* feedback

* feedback

* use preview extra

* remove coverage and add separate e2e

* rename workflow file for consistency

* trigger ci

* undo trigger

* torch import in testing

* add deps to unit tests

* feedback

* run container instead of service

* comment

* add if statement

* fix tika version

* separate out win integration tests

* separate out all CIs

* try installing docker on macos

* exclude tika

* remove tika docker
 2023-09-29 13:16:08 +02:00
ZanSara
578f2b4bbf
feat: update canals to 0.8.1 (#5900) 
* Update canals to 0.8.1

* scale up runner
 2023-09-28 17:50:46 +02:00
Massimiliano Pippi
dfa48eece9
clean up the Slack integrations (#5908) 2023-09-28 15:49:19 +02:00
ZanSara
6665e8ec7f
Add preview extra to e2e tests (#5898) 2023-09-27 10:36:00 +02:00
bogdankostic
80192589b1
feat: Add AzureOCRDocumentConverter (2.0) (#5855) 
* Add AzureOCRDocumentConverter

* Add tests

* Add release note

* Formatting

* update docstrings

* Apply suggestions from code review

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

* PR feedback

* PR feedback

* PR feedback

* Add secrets as environment variables

* Adapt test

* Add azure dependency to CI

* Add azure dependency to CI

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
 2023-09-26 15:57:55 +02:00
ZanSara
6cb7d16e22
feat: preview extra (#5869) 
* copy the deps list over from haystack-ai

* fix lazyimport usage

* keep jinja and openai

* fix ci

* reno

* separate out preview unit tests

* fix import error message for tika

* tika

* add preview to all

* wrap torch

* remove comment

* unwrap openai and jinja
 2023-09-26 12:48:15 +02:00
Stefano Fiorucci
e9d34fc0e3
test: e2e tests for RAG Pipelines (#5876) 
* relax extractive reader integration tests

* force reader to CPU

* ensure integration tests reproducibility

* e2e rag tests

* move set_all_seeds to testing package

* refine rag tests

* Update e2e/preview/pipelines/test_rag_pipelines.py

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
 2023-09-26 11:49:50 +02:00
bogdankostic
9a4373bf8e
feat: Add TikaDocumentConverter (2.0) (#5847) 
* Add TikaFileToDocument component

* Add tests

* Add tika service to CI

* Add release note

* Change name

* PR feedback

* Fix naming in tests

* Fix tika version in CI

* Update tests

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
 2023-09-25 11:47:21 +02:00
ZanSara
23fdef929e
chore: move GPT35Generator tests in the main test suite (#5844) 
* move tests

* fix no-test-found error from pytest

* missing self

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
 2023-09-21 11:42:32 +02:00
ZanSara
28f5c4c780
fix: Whisper integration tests (#5851) 
* fix tests

* add ffmpeg

* apt update for ffmpeg

* not run on windows
 2023-09-21 00:14:07 +02:00
ZanSara
ea2a5595ca
add missing dependency (#5849) 2023-09-20 12:57:53 +02:00
bogdankostic
57d33ee6da
ci: Run preview integration tests in CI (#5843) 
* Run preview integration tests in CI

* Only install inference extra
 2023-09-20 11:54:41 +02:00