542 Commits

Author SHA1 Message Date
Stefano Fiorucci
637433841e
chore: remove deprecated Seq2SeqGenerator and RAGenerator (#5180)
* first draft of removal

* more removals

* don't download unused models
2023-06-21 16:38:45 +02:00
Julian Risch
30fdf2b5df
feat!: Add extra for inference dependencies such as torch (#5147)
* feat!: add extra for inference dependencies such as torch

* add inference extra to 'all' and 'all-gpu' extra

* install inference extra in selected integration tests

* import LazyImport

* review feedback

* add import error messages and update readme

* remove extra dot
2023-06-20 09:54:10 +02:00
ZanSara
97d5db3b9c
revert fix: change the Docker workflow runner (#5078) 2023-06-05 19:11:38 +02:00
ZanSara
be3eb3cdb5
fix: change Docker workflow runner (#5077) 2023-06-05 15:59:58 +02:00
ZanSara
8487cddc69
add cli to the jobs list (#5060) 2023-06-01 13:22:17 +02:00
Silvano Cerza
6c9e062052
chore: Change checklist to simple list in PR template (#4872)
* Change checklist to simple list in PR template

* Update .github/pull_request_template.md

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>

---------

Co-authored-by: ZanSara <sara.zanzottera@deepset.ai>
2023-05-29 12:40:18 +02:00
Massimiliano Pippi
4aaf4fcc31
ci: fix Datadog event body (#5024)
* fix Datadog event body

* Update .github/workflows/license_compliance.yml

Co-authored-by: bogdankostic <bogdankostic@web.de>

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>
2023-05-27 18:12:53 +02:00
Massimiliano Pippi
8392e813a8
migrate to Datadog all the jobs (#5022) 2023-05-25 14:28:26 +02:00
Massimiliano Pippi
b69f0b3dd5
track failures on Datadog (#5020) 2023-05-25 11:26:09 +02:00
Massimiliano Pippi
929b8d1fb0
ci: run Elasticsearch 8.6 in compatibility mode (#3853)
* bump ES version in CI

disable ssl

wait for service to start

set env vars

do not use choco to install ES

re-enable jobs deps

skip test on windows CI because of OOM

allocate more memory for ES

uniform ES installation and use default heap size

skip tests causing OOM

increase job timeout

restore memory limit for ES8

* Use latest elasticsearch version
2023-05-24 18:53:54 +02:00
Massimiliano Pippi
8228081e7a
chore: leftovers from removing knowledge graph support (#4974)
* leftovers from removing knowledge graph support

* more leftovers
2023-05-22 10:03:51 +02:00
Silvano Cerza
f235d30af8
Add workflow name to Datadog event (#4968) 2023-05-19 17:42:33 +02:00
Silvano Cerza
ce4cf3bc55
Add workflow id to Datadog event tags (#4965) 2023-05-19 16:52:39 +02:00
Silvano Cerza
58bb5f09e4
Standardize workflows file names (#4964) 2023-05-19 16:41:56 +02:00
Massimiliano Pippi
4974bf7ab3
chore: remove deprecated MilvusDocumentStore (#4951)
* remove deprecated MilvusDocumentStore

* remove leftovers

* fix pylint
2023-05-19 16:37:38 +02:00
Silvano Cerza
d5cc6ff9a9
ci: Remove legacy tests (#4961)
* Remove legacy tests

* Remove unecessary env vars
2023-05-19 15:49:07 +02:00
Silvano Cerza
69bae2a3d6
Set calculator shell explicitly to handle Windows runs (#4960) 2023-05-19 15:15:18 +02:00
Silvano Cerza
dd9245531a
Add Datadog event send in examples tests workflow (#4959) 2023-05-19 15:15:10 +02:00
Silvano Cerza
2d76237508
Fix step failing to calculate Datadog event type (#4958) 2023-05-19 15:03:09 +02:00
Silvano Cerza
21ca24f70b
Send tests outcomes to Datadog instead of sending message to Slack (#4957) 2023-05-19 14:45:36 +02:00
Massimiliano Pippi
428096733d
ci: add a job to vet license of direct dependencies only (#4885)
* add conversion script

* run job in CI

* typo

* invoke python

* install toml

* fix pylint error

* more exclusions

* add toml to dev dependencies

* fix exclusions list

* fix mypy and remove test clause
2023-05-12 11:20:48 +02:00
Massimiliano Pippi
d322beed6c
build: do not install 'dev' extras with 'all' (#4888)
* do not install 'dev' with 'all'

* some fixes around
2023-05-11 19:24:47 +02:00
Silvano Cerza
6c84a05d98
Upload coverage only if all unit tests pass (#4874) 2023-05-11 14:29:44 +02:00
Massimiliano Pippi
c619aa29ec
ci: add new license checker (#4779)
* try

* add exclusions

* fix vanilla distribution

* use different requirements files

* fix comments and file name

* try with a recent version of pip

* use cpu version of torch

* try

* again

* exclude nvidia libraries

* revert old change

* send report to FOSSA

* add gpu section

* display job names

* remove FOSSA check

* send complete report to FOSSA

* removed FIXME
2023-05-10 16:33:08 +02:00
Silvano Cerza
06193e08b1
Add missing unit tests topics to coverage upload step (#4873) 2023-05-10 12:51:52 +02:00
ZanSara
28463e38e5
multi-os dep checker (#4845) 2023-05-09 11:46:53 +02:00
Sebastian
707f1c3546
Add modeling to unit tests so it we can get coverage for that (#4809)
* Add modeling to unit tests so it we can get coverage for that

* fix unit tests

---------

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2023-05-08 19:05:21 +02:00
ZanSara
28260c5c3f
feat: introduce generalimport (#4662)
* introduce generalimport

* pylint

* fix optional deps typing for schema

* leftover

* typo

* typing with faiss

* make Base generation optional too

* handle sqlalchemy

* (almost) all import are optional

* TO REMOVE hijacking CI for tests

* some deps are actually needed

* get feature branch in CI

* get feature branch in CI

* fix array_equal

* pylint

* pandas also required

* improve imports.yml

* fix SquadData

* fix SquadData again

* generalimport imports list

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* Update haystack/utils/openai_utils.py

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* review feedback

* remove todos

* reference main release

* pylint

* circular import

* review feedback

* move is_imported in init

* pylint

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-05-08 15:20:10 +02:00
Massimiliano Pippi
d8dc0d7403
chore: move custom linter to a separate package (#4790)
* move custom linter to its own package

* install the custom linter

* fix formatting

* drop python 3.7
2023-05-04 15:49:26 +02:00
Silvano Cerza
4cc69eeb29
Fix release_docs.py to create docs with correct version (#4803) 2023-05-03 15:04:50 +02:00
Silvano Cerza
9b67611169
Add others folder to unit test job (#4800) 2023-05-03 10:47:21 +02:00
Silvano Cerza
645a5fe5ba
ci: Add coverage tracking with Coveralls (#4772)
* Format tests.yml properly

* Add pytest-cov dependency

* Add coverage in unit tests

* Ignore cov.info

* Change report format

* Unignore cov.info
2023-04-28 11:59:09 +02:00
ZanSara
1b57b96210
refactor!: extract elasticsearch (#4668)
* extract elasticsearch

* update pyproject.toml

* make more import optional

* move MockBaseRetriever in conftest

* install es in the es integration tests
2023-04-26 10:14:20 +02:00
bogdankostic
91b775bf43
Execute pipelines and utils unit tests in CI (#4749) 2023-04-26 10:00:52 +02:00
Massimiliano Pippi
0c081f19e2
fix: remove warnings from the more recent Elasticsearch client (#4602)
* clean up the ES instance in a more robust way

* do not sleep, refresh the index instead

* remove client warnings

* fix unit tests

* fix opensearch compatibility

* fix unit tests

* update ES version

* bump elasticsearch-py

* adjust docs

* use recreate_index param

* use same fixture strategy for Opensearch

* Update lg

---------

Co-authored-by: agnieszka-m <amarzec13@gmail.com>
2023-04-18 15:40:17 +02:00
ZanSara
809ca73649
fix: make langdetect truly optional (#4686)
* make al langdetect imports optional

* add workflow

* fix workflow triggers

* change extra name
2023-04-17 11:35:53 +02:00
ZanSara
d8ac30fa47
refactor!: extract preprocessing and file conversion deps (#4605)
* isolate file-conversion deps

* pylint

* add to all extra

* chain was missing

* move langdetect into preprocessing and fix tika

* add file-conversion extra
2023-04-14 11:34:16 +02:00
ZanSara
174d80ab41
skip tests (#4654) 2023-04-13 17:56:51 +02:00
Silvano Cerza
db69141642
Fix docstring-labeler.yml not working in PR from forks (#4648) 2023-04-12 21:16:06 +02:00
Massimiliano Pippi
322652c306
fix: provide a fallback for PyMuPDF (#4564)
* add a fallback xpdf alternative to PyMuPDF

* add xpdpf to the base images

* to be reverted

* silence mypy on conditional error

* do not install pdf extras in base images

* bring back the xpdf build strategy

* remove leftovers from old build

* fix indentation

* Apply suggestions from code review

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>

* revert test workflow

---------

Co-authored-by: Silvano Cerza <3314350+silvanocerza@users.noreply.github.com>
2023-03-31 14:37:05 +02:00
Silvano Cerza
458e9f1897
Checkout correct ref in docstring-labeler.yml (#4563) 2023-03-30 18:11:43 +02:00
Silvano Cerza
3782ebc835
ci: Fix slack messages formatting (#4556)
* Fix slack messages formatting

* Remove unneeded file
2023-03-30 10:56:20 +02:00
Silvano Cerza
e00f1461bc
Use bigger runner for Docker release (#4538) 2023-03-29 13:14:46 +02:00
Silvano Cerza
78216196d1
Fix docker images testing (#4536) 2023-03-29 12:20:06 +02:00
Silvano Cerza
85ade5c878
Fix Slack messages formatting on job failure (#4520) 2023-03-29 09:24:41 +02:00
Massimiliano Pippi
0dfa5d6ad7
fix: do not override bake's platform definitions (#4518)
* do not override bake's platform definitions

* test

* fix job name and remove override from minor version job

* test

* bump docker login action

* fix plurals

* Remove platform from matrix and test both platform in a single job

* Remove branch trigger used for testing

---------

Co-authored-by: Silvano Cerza <silvanocerza@gmail.com>
2023-03-28 17:57:29 +02:00
Silvano Cerza
f4fb8dd946
Revert "ci: Change docker_release.yml workflow to run after successful PyPi release (#4293)" (#4513)
This reverts commit 6e241262ada9e59359d653a779246d2ad03c1223.
2023-03-28 15:28:15 +02:00
Silvano Cerza
098342da32
Use new Slack action to send failure messages (#4464) 2023-03-28 10:49:32 +02:00
Silvano Cerza
dbdb682225
Enhance release_docs.py (#4459) 2023-03-28 09:56:42 +02:00
ZanSara
9518bcb7a8
remove env var (#4497) 2023-03-27 10:33:58 +02:00