dependabot[bot]
58192d35f1
build(deps): bump iterative/setup-cml from 1 to 2 ( #5911 )
...
Bumps [iterative/setup-cml](https://github.com/iterative/setup-cml ) from 1 to 2.
- [Release notes](https://github.com/iterative/setup-cml/releases )
- [Commits](https://github.com/iterative/setup-cml/compare/v1...v2 )
---
updated-dependencies:
- dependency-name: iterative/setup-cml
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-03 17:39:22 +02:00
ZanSara
b844ab8e22
chore: remove matrix from Linux CI ( #5955 )
...
* remove matrix
* workflow names
2023-10-03 17:39:04 +02:00
Silvano Cerza
a933a42749
Fix release_notes.yml syntax
2023-10-02 13:24:08 -07:00
Zubeen
b8c3b68141
Update release_notes.yml ( #5949 )
...
Ignoring release notes check for PRs of type doc/ci/test
2023-10-02 22:17:55 +02:00
dependabot[bot]
69232612d0
build(deps): bump actions/checkout from 3 to 4 ( #5928 )
...
Bumps [actions/checkout](https://github.com/actions/checkout ) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases )
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md )
- [Commits](https://github.com/actions/checkout/compare/v3...v4 )
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-01 12:38:57 +02:00
ZanSara
81b2e83d04
feat: separate out preview tests ( #5639 )
...
* add preview workflows
* feedback
* feedback
* use preview extra
* remove coverage and add separate e2e
* rename workflow file for consistency
* trigger ci
* undo trigger
* torch import in testing
* add deps to unit tests
* feedback
* run container instead of service
* comment
* add if statement
* fix tika version
* separate out win integration tests
* separate out all CIs
* try installing docker on macos
* exclude tika
* remove tika docker
2023-09-29 13:16:08 +02:00
ZanSara
578f2b4bbf
feat: update canals to 0.8.1 ( #5900 )
...
* Update canals to 0.8.1
* scale up runner
2023-09-28 17:50:46 +02:00
Massimiliano Pippi
dfa48eece9
clean up the Slack integrations ( #5908 )
2023-09-28 15:49:19 +02:00
ZanSara
6665e8ec7f
Add preview extra to e2e tests ( #5898 )
2023-09-27 10:36:00 +02:00
bogdankostic
80192589b1
feat: Add AzureOCRDocumentConverter (2.0) ( #5855 )
...
* Add AzureOCRDocumentConverter
* Add tests
* Add release note
* Formatting
* update docstrings
* Apply suggestions from code review
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* PR feedback
* PR feedback
* PR feedback
* Add secrets as environment variables
* Adapt test
* Add azure dependency to CI
* Add azure dependency to CI
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-09-26 15:57:55 +02:00
ZanSara
6cb7d16e22
feat: preview extra ( #5869 )
...
* copy the deps list over from haystack-ai
* fix lazyimport usage
* keep jinja and openai
* fix ci
* reno
* separate out preview unit tests
* fix import error message for tika
* tika
* add preview to all
* wrap torch
* remove comment
* unwrap openai and jinja
2023-09-26 12:48:15 +02:00
Stefano Fiorucci
e9d34fc0e3
test: e2e tests for RAG Pipelines ( #5876 )
...
* relax extractive reader integration tests
* force reader to CPU
* ensure integration tests reproducibility
* e2e rag tests
* move set_all_seeds to testing package
* refine rag tests
* Update e2e/preview/pipelines/test_rag_pipelines.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-26 11:49:50 +02:00
bogdankostic
9a4373bf8e
feat: Add TikaDocumentConverter (2.0) ( #5847 )
...
* Add TikaFileToDocument component
* Add tests
* Add tika service to CI
* Add release note
* Change name
* PR feedback
* Fix naming in tests
* Fix tika version in CI
* Update tests
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-25 11:47:21 +02:00
ZanSara
23fdef929e
chore: move GPT35Generator tests in the main test suite ( #5844 )
...
* move tests
* fix no-test-found error from pytest
* missing self
---------
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-09-21 11:42:32 +02:00
ZanSara
28f5c4c780
fix: Whisper integration tests ( #5851 )
...
* fix tests
* add ffmpeg
* apt update for ffmpeg
* not run on windows
2023-09-21 00:14:07 +02:00
ZanSara
ea2a5595ca
add missing dependency ( #5849 )
2023-09-20 12:57:53 +02:00
bogdankostic
57d33ee6da
ci: Run preview integration tests in CI ( #5843 )
...
* Run preview integration tests in CI
* Only install inference extra
2023-09-20 11:54:41 +02:00
Christian Clauss
41126397d6
Revert "ci: Speed up pylint GitHub Action ( #5828 )" ( #5832 )
...
This reverts commit d49c86c845ef9ba5bfc17909cd6cf456910516e1.
2023-09-18 10:05:17 +02:00
Christian Clauss
d49c86c845
ci: Speed up pylint GitHub Action ( #5828 )
2023-09-16 16:30:13 +02:00
Massimiliano Pippi
c7971a809d
ci: skip mandatory release notes check when not needed ( #5817 )
2023-09-14 17:00:41 +02:00
Massimiliano Pippi
6fc12a2bd1
ci: run apt-get update ( #5816 )
...
* run apt-get update
* run when changing the workflow file
2023-09-14 16:37:42 +02:00
Silvano Cerza
c23cac3215
Try to send event to Datadog only if possible ( #5795 )
2023-09-13 14:10:30 +02:00
Stefano Fiorucci
784034ffc3
Revert "build(deps): bump readmeio/rdme from 8.3.1 to 8.6.6 ( #5789 )" ( #5792 )
...
This reverts commit 55a2e7ab7fc16e4e311ea994b95553031711a506.
2023-09-13 11:56:42 +02:00
dependabot[bot]
55a2e7ab7f
build(deps): bump readmeio/rdme from 8.3.1 to 8.6.6 ( #5789 )
...
Bumps [readmeio/rdme](https://github.com/readmeio/rdme ) from 8.3.1 to 8.6.6.
- [Release notes](https://github.com/readmeio/rdme/releases )
- [Changelog](https://github.com/readmeio/rdme/blob/next/CHANGELOG.md )
- [Commits](https://github.com/readmeio/rdme/compare/8.3.1...8.6.6 )
---
updated-dependencies:
- dependency-name: readmeio/rdme
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-13 11:03:12 +02:00
Silvano Cerza
7e544d4f60
Fix license compliance workflow ( #5791 )
...
* Formatting
* Try to send event to Datadog only if possible
2023-09-13 10:43:06 +02:00
dependabot[bot]
e688d3dddb
build(deps): bump aws-actions/configure-aws-credentials ( #5790 )
...
Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials ) from 2.2.0 to 4.0.0.
- [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases )
- [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md )
- [Commits](5fd3084fc3...8c3f20df09
2023-09-13 10:25:54 +02:00
Massimiliano Pippi
de6c57e20b
let dependabot update github actions ( #5788 )
2023-09-13 10:23:30 +02:00
Christian Clauss
75dc60b0bb
ci: Upgrade GitHub Actions ( #5787 )
2023-09-13 09:58:47 +02:00
Christian Clauss
23f7308bec
ci: pre-commit autoupdate ( #5777 )
2023-09-12 14:34:41 +02:00
Christian Clauss
45cc40bf51
linting.yml: Upgrade GitHub Actions ( #5752 )
2023-09-11 20:49:20 +02:00
Vladimir Blagojevic
c5edb45c10
feat: Add SerperDevWebSearch Haystack 2.0 component ( #5712 )
...
* Add SerperDev
* Add release note
* PR Feedback
* Simplify, remove one-liner
* Update haystack/preview/components/websearch/serper_dev.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* Update haystack/preview/components/websearch/serper_dev.py
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
* Fix formatting
* PR feedback
* Fix tests
* Function rename
* Remove scoring, update tests
* PR feedback
* Fix return
* small adjustments
* fix tests
* add e2e test
* fix release notes
* fix tests
* fix e2e
---------
Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-06 17:31:42 +02:00
ZanSara
0bbc219a59
chore: enable e2e preview tests ( #5730 )
...
* enable e2e preview tests
* fix transcriber test
* quotes
* add missing dep
* missing comma
* ffmpeg
2023-09-06 16:48:45 +02:00
Timo Moeller
d540883469
Add api keys to CI workflows ( #5722 )
2023-09-05 16:21:17 +02:00
ZanSara
ce06268990
test: fix e2e test failures ( #5685 )
...
* fix test errors
* fix pipeline yaml
* disable cache
* fix errors
* remove stray fixture
2023-08-30 12:24:03 +02:00
ZanSara
1709be162c
auto trigger e2e workflow on PRs that affect it ( #5684 )
2023-08-30 10:25:47 +02:00
ZanSara
5985b6d358
chore: refactor pipeline tests for e2e testing ( #5576 )
...
* enable pipeline filder in e2e
* merge standard pipeline tests with stanrdard pipeline batch tests
* merge summarization tests into standard pipelines tests
* Update test_standard_pipelines.py
* black
2023-08-29 11:22:39 +02:00
Silvano Cerza
444edce126
Add workflow to trigger preview package release ( #5631 )
2023-08-25 17:10:28 +02:00
Silvano Cerza
cb894061f7
Add terminate-runner job in benchmarks.yml ( #5611 )
2023-08-25 10:14:39 +02:00
Silvano Cerza
b53fad4c4f
Add missing integration tests to catch-all required step in tests.yml ( #5598 )
2023-08-18 17:58:26 +02:00
bogdankostic
ee2745bad8
ci: Add Github workflow to automate benchmark runs ( #5399 )
...
* Add config files
* log benchmarks to stdout
* Add top-k and batch size to configs
* Add batch size to configs
* fix: don't download files if they already exist
* Add batch size to configs
* refine script
* Remove configs using 1m docs
* update run script
* update run script
* update run script
* datadog integration
* remove out folder
* gitignore benchmarks output
* test: send benchmarks to datadog
* remove uncommented lines in script
* feat: take branch/tag argument for benchmark setup script
* fix: run.sh should ignore errors
* Add GH workflow to run benchmarks periodically
* Remove unused script
* Adapt cml.yml
* Adapt cml.yml
* Rename cml.yml to benchmarks.yml
* Revert "Rename cml.yml to benchmarks.yml"
This reverts commit 897299433a71a55827124728adff5de918d46d21.
* remove benchmarks.yml
* Use same file extension for all config files
* Use checkout@v3
* Run benchmarks sequentially
* Add timeout-minutes parameter
* Remove changes unrelated to datadog
* Apply black
* use haystack-oss aws account
* Update test/benchmarks/utils.py
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* PR feedback
* fix aws credentials step
* Fix path
* check docker
* Allow spinning up containers from within container
* Allow spinning up containers from within container
* Separate launching doc stores from benchmarks
* Remove docker related commands
* run only retrievers
* change port
* Revert "change port"
This reverts commit 6e5bcebb1d16e03ba7672be7e8a089084c7fc3a7.
* Run opensearch benchmark only
* Run weaviate benchmark only
* Run bm25 benchmarks only
* Changes host of doc stores
* add step to get docker logs
* Revert "add step to get docker logs"
This reverts commit c10e6faa76bde5df406a027203bd775d18c93c90.
* Install docker
* Launch doc store containers from wtihin runner container
* Remove kill command
* Change host
* dump docker logs
* change port
* Add cloud startup script
* dump docker logs
* add network param
* add network to startup.sh
* check cluster health
* move steps
* change port
* try using services
* check cluster health
* use services
* run only weaviate
* change host
* Upload benchmark results as artifacts
* Update configs
* Delete index after benchmark run
* Use correct index name
* Run only failing config
* Use smaller batch size
* Increase memory for opensearch
* Reduce batch size further
* Provide more storage
* Reduce batch size
* dump docker logs
* add java opts
* Spin up only opensearch container
* Create separate job for each doc store
* Run benchmarks sequentially
* Set working directory
* Account for reader benchmarks not doing indexing
* Change key of reader metrics
* Apply PR feedback
* Remove whitespace
* Adapt workflow to changes in datadog scripts
* Adapt workflow to changes in datadog scripts
* Increase memory for opensearch
* Reduce batch size
* Add preprocessing_batch_size to Readers
* Remove unrelated change
* Move order
* Fix path
* Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
Manually terminate EC2 instance
* Manually terminate EC2 instance
* Manually terminate EC2 instance
* Always terminate runner
* Always terminate runner
* Remove unnecessary terminate-runner job
* Add cron schedule
* Disable telemetry
* Rename cml.yml to benchmarks.yml
---------
Co-authored-by: rjanjua <rohan.janjua@gmail.com>
Co-authored-by: Paul Steppacher <p.steppacher91@gmail.com>
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-08-17 12:56:45 +02:00
Silvano Cerza
bc152d953c
Skip running tests in CI when editing docs Python files ( #5482 )
2023-08-01 12:31:24 +02:00
Silvano Cerza
9a359101fd
chore: Rework docs generation ( #5481 )
...
* Change docs generation to use id for parent doc instead of slug
* Rename step
2023-08-01 12:18:33 +02:00
Massimiliano Pippi
5f01391827
add workflow to check presence of release notes ( #5449 )
2023-07-27 10:40:40 +02:00
Massimiliano Pippi
363f3edbf7
feat: add reno to manage release notes ( #5397 )
...
* first draft
* add release notes
* remove old settings
* add reno usage instructions
* page the docs team when release notes are added
* add reno to the dev dependencies
* Apply suggestions from code review
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
---------
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-07-24 17:02:46 +02:00
Julian Risch
eeb29b5686
test: Re-activate end-to-end tests workflow ( #5343 )
...
* Install haystack with required extras
* remove whitespaces
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
* Add sleep
* Add s for seconds
* Move container initialization in workflow
* Update e2e.yml
add nightly run
* use new folder for initial e2e test
* use file hash for caching and trigger on push to branch
* remove \n from model names read from file
* remove trigger on push to branch
---------
Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-07-20 11:48:51 +02:00
bogdankostic
41668f26d6
ci: Update labeler.yml to account for Elasticsearch changes ( #5318 )
2023-07-11 11:51:49 +02:00
bogdankostic
b7f683bfa4
ci: Add unit test for Elasticsearch8 ( #5300 )
...
* Add job for ES8 integration tests
* Add unit test for Elasticsearch 8
* Add tests.yml
* Adapt tests.yml
* Remove added white space
* Adapt tests.yml
* Adapt tests.yml
* Add dependencies to unit test name
* Adapt unit test matrix
* Adapt unit test matrix
* Adapt unit test matrix
* Adapt unit test matrix
* Update tests.yml
* Create separate tests where necessary
* Fix skip
* Adapt tests
2023-07-10 16:03:50 +02:00
bogdankostic
048fc7f640
ci: Add job for ES8 integration tests ( #5297 )
...
* Add job for ES8 integration tests
* Remove whitespace
* Fix filename
* Add tests.yml
* Revert "Add tests.yml"
This reverts commit ec12654d4e146b5ef6cba04ad82f5973935d8520.
2023-07-10 10:43:05 +02:00
Vladimir Blagojevic
395854d823
Add cpu-remote-inference Docker image ( #5225 )
...
* Add cpu-remote-inference Docker image
* Add web lfqa pipeline as an example for cpu-remote-inference Docker image
* WebRetriever must have document_store attribute
* Add cpu-remote-inference-latest
* Add image testing in CI
---------
Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-07-07 10:23:14 +02:00
ZanSara
4b380d8fb0
fix: install inference in REST API tests ( #5252 )
...
* install inference in restapi tests
* add workflow dispatch to test the REST API CI in PR
* trigger ci
* tablecell
* tablecell
* revert ci trigger
* mypy
2023-07-03 15:10:14 +02:00
