130 Commits

Author SHA1 Message Date
Silvano Cerza
db759b0717
Add black step when testing examples (#6425) 2023-11-27 15:01:33 +01:00
Silvano Cerza
892625a6c7
ci: Add back workflows that runs in place of linting.yml and tests.yml (#6421)
* Add back workflow that runs in place of linting.yml

* Add back workflow that runs in place of tests.yml
2023-11-27 13:18:47 +01:00
Silvano Cerza
8bfaf0a56a
ci: Add catch-all job in tests.yml (#6419)
* Add catch-all job in tests.yml

* Trigger for testing

* Remove trigger for testing
2023-11-27 12:57:33 +01:00
Silvano Cerza
9338de1790 Add missing tests workflow dependency 2023-11-24 16:00:59 +01:00
Massimiliano Pippi
4a1fe163b6
fix names in workflows 2023-11-24 14:59:31 +01:00
Massimiliano Pippi
09e7831f60
clean up 1.x code
---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-11-24 11:47:47 +01:00
Massimiliano Pippi
209e349be3
do not run preview tests twice (#6204) 2023-10-31 13:13:32 +01:00
Grant Williams
1cf70d3dce
build: Upgrade transformers to the latest version 4.34.1 (#5994)
* Upgrade transformers to the latest version 4.34.0 so that Haystack can support the new Mistral, Nougat, and other models.

* update release notes

* updated missing lazy import

* Update .github workflows imports

* bump more versions in .github workflows

* rever import sorting

* Update  to catch runtime errors to match haystack_hub changes

* add language parameter value to whisper test

* bump transformers version in linting preview workflow

* bump transformers version in linting preview workflow

* bump version to v4.34.1

* resolve mypy issue with reused variables

* install openai-whisper without dependencies

* remove audio extra, update whisper install instructions

* remove audio extra, update whisper install instructions

* keep audio extra but add version

* keep audio extra with no constraints

* remove audio extra

---------

Co-authored-by: Julian Risch <julian.risch@deepset.ai>
2023-10-24 19:13:12 +02:00
Stefano Fiorucci
c4187eeebe
CI: make only test_preview run when preview e2e tests are changed (#6078)
* make only test_preview workflow run when e2e tests are modified

* revert wrong changes to test_preview

* revert wrong order
2023-10-17 10:06:39 +02:00
ZanSara
adf7e49af3
chore: review all extra (#6029) 2023-10-12 21:50:53 +02:00
ZanSara
81b2e83d04
feat: separate out preview tests (#5639)
* add preview workflows

* feedback

* feedback

* use preview extra

* remove coverage and add separate e2e

* rename workflow file for consistency

* trigger ci

* undo trigger

* torch import in testing

* add deps to unit tests

* feedback

* run container instead of service

* comment

* add if statement

* fix tika version

* separate out win integration tests

* separate out all CIs

* try installing docker on macos

* exclude tika

* remove tika docker
2023-09-29 13:16:08 +02:00
Massimiliano Pippi
dfa48eece9
clean up the Slack integrations (#5908) 2023-09-28 15:49:19 +02:00
bogdankostic
80192589b1
feat: Add AzureOCRDocumentConverter (2.0) (#5855)
* Add AzureOCRDocumentConverter

* Add tests

* Add release note

* Formatting

* update docstrings

* Apply suggestions from code review

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

* PR feedback

* PR feedback

* PR feedback

* Add secrets as environment variables

* Adapt test

* Add azure dependency to CI

* Add azure dependency to CI

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
Co-authored-by: Daria Fokina <daria.fokina@deepset.ai>
2023-09-26 15:57:55 +02:00
ZanSara
6cb7d16e22
feat: preview extra (#5869)
* copy the deps list over from haystack-ai

* fix lazyimport usage

* keep jinja and openai

* fix ci

* reno

* separate out preview unit tests

* fix import error message for tika

* tika

* add preview to all

* wrap torch

* remove comment

* unwrap openai and jinja
2023-09-26 12:48:15 +02:00
bogdankostic
9a4373bf8e
feat: Add TikaDocumentConverter (2.0) (#5847)
* Add TikaFileToDocument component

* Add tests

* Add tika service to CI

* Add release note

* Change name

* PR feedback

* Fix naming in tests

* Fix tika version in CI

* Update tests

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-09-25 11:47:21 +02:00
ZanSara
28f5c4c780
fix: Whisper integration tests (#5851)
* fix tests

* add ffmpeg

* apt update for ffmpeg

* not run on windows
2023-09-21 00:14:07 +02:00
ZanSara
ea2a5595ca
add missing dependency (#5849) 2023-09-20 12:57:53 +02:00
bogdankostic
57d33ee6da
ci: Run preview integration tests in CI (#5843)
* Run preview integration tests in CI

* Only install inference extra
2023-09-20 11:54:41 +02:00
Christian Clauss
75dc60b0bb
ci: Upgrade GitHub Actions (#5787) 2023-09-13 09:58:47 +02:00
Silvano Cerza
b53fad4c4f
Add missing integration tests to catch-all required step in tests.yml (#5598) 2023-08-18 17:58:26 +02:00
Silvano Cerza
bc152d953c
Skip running tests in CI when editing docs Python files (#5482) 2023-08-01 12:31:24 +02:00
bogdankostic
b7f683bfa4
ci: Add unit test for Elasticsearch8 (#5300)
* Add job for ES8 integration tests

* Add unit test for Elasticsearch 8

* Add tests.yml

* Adapt tests.yml

* Remove added white space

* Adapt tests.yml

* Adapt tests.yml

* Add dependencies to unit test name

* Adapt unit test matrix

* Adapt unit test matrix

* Adapt unit test matrix

* Adapt unit test matrix

* Update tests.yml

* Create separate tests where necessary

* Fix skip

* Adapt tests
2023-07-10 16:03:50 +02:00
bogdankostic
048fc7f640
ci: Add job for ES8 integration tests (#5297)
* Add job for ES8 integration tests

* Remove whitespace

* Fix filename

* Add tests.yml

* Revert "Add tests.yml"

This reverts commit ec12654d4e146b5ef6cba04ad82f5973935d8520.
2023-07-10 10:43:05 +02:00
Julian Risch
30fdf2b5df
feat!: Add extra for inference dependencies such as torch (#5147)
* feat!: add extra for inference dependencies such as torch

* add inference extra to 'all' and 'all-gpu' extra

* install inference extra in selected integration tests

* import LazyImport

* review feedback

* add import error messages and update readme

* remove extra dot
2023-06-20 09:54:10 +02:00
ZanSara
8487cddc69
add cli to the jobs list (#5060) 2023-06-01 13:22:17 +02:00
Massimiliano Pippi
929b8d1fb0
ci: run Elasticsearch 8.6 in compatibility mode (#3853)
* bump ES version in CI

disable ssl

wait for service to start

set env vars

do not use choco to install ES

re-enable jobs deps

skip test on windows CI because of OOM

allocate more memory for ES

uniform ES installation and use default heap size

skip tests causing OOM

increase job timeout

restore memory limit for ES8

* Use latest elasticsearch version
2023-05-24 18:53:54 +02:00
Silvano Cerza
f235d30af8
Add workflow name to Datadog event (#4968) 2023-05-19 17:42:33 +02:00
Silvano Cerza
ce4cf3bc55
Add workflow id to Datadog event tags (#4965) 2023-05-19 16:52:39 +02:00
Silvano Cerza
d5cc6ff9a9
ci: Remove legacy tests (#4961)
* Remove legacy tests

* Remove unecessary env vars
2023-05-19 15:49:07 +02:00
Silvano Cerza
69bae2a3d6
Set calculator shell explicitly to handle Windows runs (#4960) 2023-05-19 15:15:18 +02:00
Silvano Cerza
2d76237508
Fix step failing to calculate Datadog event type (#4958) 2023-05-19 15:03:09 +02:00
Silvano Cerza
21ca24f70b
Send tests outcomes to Datadog instead of sending message to Slack (#4957) 2023-05-19 14:45:36 +02:00
Massimiliano Pippi
d322beed6c
build: do not install 'dev' extras with 'all' (#4888)
* do not install 'dev' with 'all'

* some fixes around
2023-05-11 19:24:47 +02:00
Silvano Cerza
6c84a05d98
Upload coverage only if all unit tests pass (#4874) 2023-05-11 14:29:44 +02:00
Silvano Cerza
06193e08b1
Add missing unit tests topics to coverage upload step (#4873) 2023-05-10 12:51:52 +02:00
Sebastian
707f1c3546
Add modeling to unit tests so it we can get coverage for that (#4809)
* Add modeling to unit tests so it we can get coverage for that

* fix unit tests

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-08 19:05:21 +02:00
Silvano Cerza
9b67611169
Add others folder to unit test job (#4800) 2023-05-03 10:47:21 +02:00
Silvano Cerza
645a5fe5ba
ci: Add coverage tracking with Coveralls (#4772)
* Format tests.yml properly

* Add pytest-cov dependency

* Add coverage in unit tests

* Ignore cov.info

* Change report format

* Unignore cov.info
2023-04-28 11:59:09 +02:00
ZanSara
1b57b96210
refactor!: extract elasticsearch (#4668)
* extract elasticsearch

* update pyproject.toml

* make more import optional

* move MockBaseRetriever in conftest

* install es in the es integration tests
2023-04-26 10:14:20 +02:00
bogdankostic
91b775bf43
Execute pipelines and utils unit tests in CI (#4749) 2023-04-26 10:00:52 +02:00
Massimiliano Pippi
0c081f19e2
fix: remove warnings from the more recent Elasticsearch client (#4602)
* clean up the ES instance in a more robust way

* do not sleep, refresh the index instead

* remove client warnings

* fix unit tests

* fix opensearch compatibility

* fix unit tests

* update ES version

* bump elasticsearch-py

* adjust docs

* use recreate_index param

* use same fixture strategy for Opensearch

* Update lg

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-04-18 15:40:17 +02:00
ZanSara
d8ac30fa47
refactor!: extract preprocessing and file conversion deps (#4605)
* isolate file-conversion deps

* pylint

* add to all extra

* chain was missing

* move langdetect into preprocessing and fix tika

* add file-conversion extra
2023-04-14 11:34:16 +02:00
ZanSara
174d80ab41
skip tests (#4654) 2023-04-13 17:56:51 +02:00
Silvano Cerza
3782ebc835
ci: Fix slack messages formatting (#4556)
* Fix slack messages formatting

* Remove unneeded file
2023-03-30 10:56:20 +02:00
Silvano Cerza
85ade5c878
Fix Slack messages formatting on job failure (#4520) 2023-03-29 09:24:41 +02:00
Silvano Cerza
098342da32
Use new Slack action to send failure messages (#4464) 2023-03-28 10:49:32 +02:00
ZanSara
9518bcb7a8
remove env var (#4497) 2023-03-27 10:33:58 +02:00
Silvano Cerza
0f605118d9
ci: remove python_cache internal action (#4429) 2023-03-17 13:55:07 +01:00
Massimiliano Pippi
8d4c56720c
do not run tests on osx (#4443) 2023-03-16 15:00:29 +01:00
Vladimir Blagojevic
2538b4cbc9
Make promptnode test unit (#4420) 2023-03-15 22:17:23 +01:00