60 Commits

Author SHA1 Message Date
Stefano Fiorucci
3b4f4ac7e9
ci: Release orchestration workflow (#10233)
* ci: release orchestration workflow

* offload notification preparation to external script

* swap if/needs

* slim where makes sense

* tmp test for ubuntu slim

* revet

* concurrency
2025-12-12 16:30:02 +01:00
Stefano Fiorucci
0205781af1
ci: add scripts for release automation (#10226)
* ci: add scripts for release automation

* improvements from review
2025-12-11 17:06:24 +01:00
Stefano Fiorucci
af75774c3f
chore: rename docs directory and other adjustments (#10157)
* chore: rename docs directory and other adjustments

* fixes
2025-11-28 08:41:35 +01:00
Stefano Fiorucci
8bdcd34610
chore: remove CI scripts and configs for Readme API (#10147) 2025-11-27 13:05:55 +01:00
Stefano Fiorucci
40daaf6cf5
ci: workflow to populate the index to search docs (#10132)
* try feed docs search

* fix

* checkout

* update

* start developing workflow

* fix dir

* tru

* fix lint

* progress

* fixes

* more fixes

* improve script

* refinements

* install requests

* small improvements

* use npm install

* further refinements

* run this workflow manually and scheduled

* fmt

* try monitor

* retry

* clean up
2025-11-26 09:28:33 +00:00
Stefano Fiorucci
244dd824cb
docs: CI scripts for docs versioning at Haystack release (#9980)
* draft

* progress

* progress

* todo comments

* refinements
2025-10-31 10:46:44 +01:00
Stefano Fiorucci
d77a853b49
chore: remove unused files (#9876) 2025-10-15 09:48:37 +02:00
Abdelrahman Kaseb
5f3c37d287
chore: adopt PEP 585 type hints (#9678)
* chore(lint): enforce and apply PEP 585 type hinting

* Run fmt fixes

* Fix all typing imports using some regex

* Fix all typing written in string in tests

* undo changes in the e2e tests

* make e2e test use list instead of List

* type fixes

* remove type:ignore

* pylint

* Remove typing from Usage example comments

* Remove typing from most of comments

* try to fix e2e tests on comm PRs

* fix

* Add tests typing.List in to adjust test compatiplity
- test/components/agents/test_state_class.py
- test/components/converters/test_output_adapter.py
- test/components/joiners/test_list_joiner.py

* simplify pyproject

* improve relnote

---------

Co-authored-by: anakin87 <stefanofiorucci@gmail.com>
2025-08-07 10:23:14 +02:00
Stefano Fiorucci
c18f81283c
chore: fix deepset_sync.py for pylint + general linting improvements (#9558)
* chore: fix deepset_sync.py for pylint

* check .github with ruff

* fix

* Update .github/utils/pyproject_to_requirements.py

Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>

---------

Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com>
2025-06-27 07:54:22 +00:00
mathislucka
101e9cdc34
docs: sync code to deepset workspace (#9555)
* docs: sync code to deepset workspace

* fix: naming

* fix: actionlint
2025-06-27 07:51:59 +02:00
Michele Pangrazzi
3207a76d50
chore: Update pydoc-markdown.sh (#9547)
* Make config path a $1 param ; Add usage in comment ; Add echo log

* Update sync command
2025-06-24 14:01:51 +02:00
Stefano Fiorucci
3339097e99
ci: refactor job to check imports (#8892)
* refactor

* Trigger CI

* run tests if this file changes

* show failure

* revert

* rm duplicate subdir and explain in comment
2025-02-21 11:37:41 +01:00
Stefano Fiorucci
09adf856dc
rm openapi spec util (#8613) 2024-12-09 10:59:21 +01:00
Stefano Fiorucci
de7099e560
ci: add job to check imports (#8594)
* try checking imports

* clarify error message

* better fmt

* do not show complete list of successfully imported packages

* refinements

* relnote

* add missing forward references

* better function name

* linting

* fix linting

* Update .github/utils/check_imports.py

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-11-29 14:00:59 +00:00
Silvano Cerza
9dc602ebf3
Remove type hint (#8367) 2024-09-16 11:50:56 +02:00
Silvano Cerza
145dfce3aa
ci: Slightly rework docs release process (#8363)
* Slightly rework docs release process

* Apply suggestions from code review

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>

---------

Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
2024-09-16 09:28:40 +00:00
Silvano Cerza
2fea67d4a9
Fix delete_outdated_docs.py to get the correct api key (#7242) 2024-02-28 12:52:13 +01:00
Silvano Cerza
85491433c6
ci: Add script to delete documentation that don't exist anymore (#7209)
* Add script to delete documentation that don't exist anymore

* Remove prints
2024-02-28 11:02:59 +01:00
Massimiliano Pippi
4efe40664c
use haystack-pydoc-tools package instead of local code (#6818) 2024-01-23 18:28:52 +01:00
Daria Fokina
3fedd9f67f
docs: add Data Classes API ref and cleanup (#6482)
* Data Classes and cleanup

* Update pydoc-markdown.sh

* Update converter.yml
2023-12-04 14:03:28 +01:00
Julian Risch
db36d6277a
docs: Add readme sync for API docs 2.0 (#6173)
* add sync docs for preview

* add example config for audio docs

* hardcode version in renderer

* use custom renderer for preview docs

* update comment and excerpt
2023-10-27 14:53:03 +02:00
Massimiliano Pippi
428096733d
ci: add a job to vet license of direct dependencies only (#4885)
* add conversion script

* run job in CI

* typo

* invoke python

* install toml

* fix pylint error

* more exclusions

* add toml to dev dependencies

* fix exclusions list

* fix mypy and remove test clause
2023-05-12 11:20:48 +02:00
Silvano Cerza
4cc69eeb29
Fix release_docs.py to create docs with correct version (#4803) 2023-05-03 15:04:50 +02:00
Silvano Cerza
dbdb682225
Enhance release_docs.py (#4459) 2023-03-28 09:56:42 +02:00
Massimiliano Pippi
1498aacc77
chore: make the docs generator runnable without an API key (#4405)
* spit a warning instead of exiting

* print which file is being converted (useful to debug CI)

* pin docspec for the time being
2023-03-14 16:15:19 +01:00
Silvano Cerza
ee74421212
ci: Refactor docs config and generation (#4280)
* Change docs yml category config

* Update docs renderers to fetch categories from Readme.io

* Update readme_sync.yml to handle new docs rendering

* Remove unecessary script and related workflow step

* Fix sys.exits
2023-03-01 09:51:02 +01:00
Silvano Cerza
181e5474e8
ci: Automate OpenAPI specs upload to Readme.io (#4228)
* Remove OpenAPI specs file

* OpenAPI specs are now automatically uploaded when necessary

* Rename openapi workflow
2023-02-22 18:01:18 +01:00
Silvano Cerza
689f2cd250
Update docstring-labeler.yml workflow to safely run in PRs from forks (#4146) 2023-02-16 16:02:41 +01:00
Silvano Cerza
2cc938ff90
ci: Add workflow to label PRs that edit docstrings (#4115)
* Add workflow to label PRs that edit docstrings

* Add python-version arg in setup-python steps

* Run workflow only in haystack and rest_api python files edit

* Fix labeling job

* Fix labeling conditional

* Fix files globbing in docstrings_checksum.py

* Fix typing

* Rework workflow to use a single job
2023-02-09 18:57:30 +01:00
Massimiliano Pippi
8824f3a10a
re-organize pydoc config files (#4042) 2023-02-03 12:51:10 +01:00
Bilge Yücel
ddba75021a
fix: add additional settings to OpenAPI schema (#3788)
* "proxy-enabled": disable CORS proxy
* "samples-languages": display two languages initially
2022-12-30 16:10:37 +03:00
bogdankostic
36cfd41713
Add newline when generating OpenAPI specs (#3782) 2022-12-29 17:55:43 +01:00
Massimiliano Pippi
6cd0e337d0
refactor: Generate JSON schema when missing (#3533)
* removed unused script

* print info logs when generating openapi schema

* create json schema only when needed

* fix tests

* Remove leftover

Co-authored-by: ZanSara <sarazanzo94@gmail.com>
2022-11-17 11:09:27 +01:00
Massimiliano Pippi
0c1de3745d
fix milvus imports (#3576) 2022-11-15 10:58:51 +01:00
Branden Chan
cf4642a5f8
[CI] Create Github Workflow that creates a new version branch in Haystack and Readme (#3335)
* Test readme_integration.yml

* Test readme_integration.yml

* Test variables

* Test variables

* Test variables

* Test variables

* Test commit

* Test commit

* Test commit

* Trigger action

* Add v

* Trigger action

* Trigger action

* Trigger action

* Trigger action

* Update API docs headers

* Revert "Update API docs headers"

This reverts commit 34e665063f4de29854befe575a795dbfef04415c.

* Trigger action

* Trigger action

* Trigger action

* Update release

* Update release

* Update release

* Delete File

* Split steps into own files

* Edit action names

* Start making changes

* Start implementing version bump

* Implement minor version release

* Fix github action

* Test action

* Test action

* Test action

* Test action

* Test action

* Change back to main

* Add comments

* Remove line

* Format docstring

* Incorporate reviewer feedback

* Fix variable name

* Print version.txt

* Incorporate Reviewer feedback

* Rename variables for clarity

* Add fetch

* Change branch

* Change branch

* Change branch

* Change branch

* Change branch

* Revert docstring changes

* Incorporate reviewer feedback

* Run black

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2022-10-18 17:09:43 +02:00
Vladimir Blagojevic
eba7cf51b1
chore: Remove Update API documentation hook (#3271)
* Remove Update API documentation hook

* Remove .github/utils/pydoc-markdown.py file
2022-09-23 08:54:08 -04:00
Branden Chan
492a8046d8
docs: sync Haystack API with Readme (#3223)
* First pass at syncing Haystack API with Readme

* Reapply changes

* Regularize slugs

* Regularize slugs

* Regularize slugs

* Set category id and regen

* Trigger workflow

* Delete old md files

* Test sync

* Undo test string

* Incorporate reviewer feedback

* Test on the fly API generation and sync

* Test on the fly API generation and sync

* Test on the fly API generation and sync

* Test on the fly API generation and sync

* Test on the fly API generation and sync

* Change name of pydoc-markdown scripts

* Test on the fly API generation and sync

* Remove version tag

* Test version tag

* Test version tag

* Test version tag

* Revert test docstring

* Revert md file changes

* Revert md file changes

* Revert script naming

* Test on the fly generation and sync

* Adjust for on the fly generation and sync

* Revert test string

* Remove old documentation workflow

* Set workflow to work on main

* Change readme version name
2022-09-21 17:18:34 +02:00
Massimiliano Pippi
15bb6c2ea2
remove tutorials from the repo (#3244) 2022-09-20 18:32:45 +02:00
Massimiliano Pippi
9399ddf949
fix pydoc-markdown hook (#3238) 2022-09-19 18:20:35 +02:00
Massimiliano Pippi
6790eaf7d8
refactor: update package strategy in rest_api (#3148)
* update packaging

* fix author metadata

* add newline

* add empty readme

* fix path to pipeline files

* fix pylint job

* fix metadata
2022-09-05 16:58:43 +02:00
Daniel Bichuetti
e1f399284f
refactor: update dependencies and remove pins (#3147)
* refactor: remove azure-core, pydoc and hf-hub pins

* fix: remove extra-comma

* fix: force minimum version of azure forms recognizer

* refactor: allow newer ocr libs

* refactor: update more dependencies and container versions

* refactor: remove extra comment

* docs: pre-commit manual run

* refactor: remove unnecessary dependency

* tests: update weaviate container image version
2022-09-05 14:30:35 +02:00
Massimiliano Pippi
0e8efdafa9
Add enhanced pydoc-markdown pre-hook (#2979)
* add pydoc-markdown pre-hook

* add more comments, remove debug prints
2022-08-08 12:41:21 +02:00
Sara Zan
2d65c380f1
pre-commit hooks (#2819)
* Add pre-commit config

* update contributing guidelines

* try failing the workflow

* add pre-commit to the deps

* updating uninstall instructions

* separate jobs in CI

* make tutorials check fail

* make black check fail

* make openapi check fail

* make yaml schema and api docs checks fail

* highlight the instructions

* Update .pre-commit-config.yaml

Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>

* Update CONTRIBUTING.md

Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>

* Update CONTRIBUTING.md

Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>

* Use black --check

* Add images of the CI

* title level

* feedback

Co-authored-by: Tobias Wochinger <mail@tobias-wochinger.de>
2022-07-26 15:02:15 +02:00
Sara Zan
5d8476eb58
Restart containers in tutorials.sh (#2858)
* restart tutorials in the loop

* remove container steps in tutorials.yml

* forgotten quotes

* unmatched bracket

* give names to containers

* try to limit the log size

* make the containers restart on the scripts as well

* feedback

* Raise integration tests timeout

* raising limit again
2022-07-25 17:35:36 +02:00
Massimiliano Pippi
8ee2b6b403
Add a custom pydoc renderer for Readme.io (#2825)
* add custom pydoc renderer

* create an example

* revert example code
2022-07-22 10:43:51 +02:00
Sara Zan
091711b8c4
Fix Tutorials and Tutorials (nightly) (#2737)
* Remove caching and install audio deps

* Fix `Tutorials` as well

* Run all tutorials even though some fail

* Forgot fi

* fix failure condition

* proper bash string equality

* Enable debug logs

* remove audio files

* Update Documentation & Code Style

* Use the setup action in the Tutorial CI as well

* Try with a file that exists

* Update Documentation & Code Style

* Fix the comments in the tutorials

* Update Documentation & Code Style

* Fix tutorials.sh

* Remove debug logging

* import pprint and try editable install

* Update Documentation & Code Style

* extract no run list

* Add tutorial18 to no run list nightly

* import pprint correctly

* Update Documentation & Code Style

* try making site-packages editable

* Make pythonpath editable every time Tut17 is run on CI

* typo

* fix imports in tut5

* add git clean

* Update Documentation & Code Style

* add comments and remove` -e`

* accidentally deleted a line

* Update .github/utils/tutorials.sh

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
2022-07-12 11:22:17 +02:00
Vladimir Blagojevic
b08c5f81d1
Add GPL adaptation tutorial (#2632)
* Add GPL adaptation tutorial

* Latest round of Aga's corrections

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-06-26 02:44:57 -04:00
Sara Zan
a26c042994
Fix typo in code_and_docs.sh (#2662)
* Fix typo in code_and_docs.sh & install ffmpeg in autoformat.yml

* apt update to get ffmpeg

* Update Documentation & Code Style

* Add header and better error message

* Update Documentation & Code Style

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-06-15 13:50:55 +02:00
Sara Zan
735ffa635b
[CI refactoring] Tutorials on CI (#2547)
* Experimental Ci workflow for running tutorials

* Run on every push for now

* Not starting?

* Disabling paths temporarily

* Sort tutorials in natural order

* Install ipython

* remove ipython install

* Try running ipython with sudo

* env.pythonLocation

* Skipping tutorial2 and 9 for speed

* typo

* Use one runner per tutorial, for now

* Typo in dependend job

* Missing quotes broke scripts matrix

* Simplify setup for the tutorials, try to prevent containers conflict

* Remove needless job dependencies

* Try prevent cache issues, fix small Tut10 bug

* Missing deps for running notebook tutorials

* Create three groups of tutorials excluding the longest among them

* remove deps

* use proper bash loop

* Try with a single string

* Fix typo in echo

* Forgot do

* Typo

* Try to make the GraphDB tutorial without launching its own container

* Run notebook and script together

* Whitespace

* separate scrpits and notebooks execution

* Run notebooks first

* Try caching the GoT data before running the scripts

* add note

* fix mkdir

* Fix path

* Update Documentation & Code Style

* missing -r

* Fix folder numbering

* Run notebooks as well

* Typo in notebook command

* complete path in notebook command

* Try with TIKA_LOG_PATH

* Fix folder naming

* Do not use cached data in Tut9

* extracting the number better

* Small tweaks

* Same fix on Tut10 on the notebook

* Exclude GoT cache for tut5 too

* Remove faiss files after tutorial run

* Layout

* fix remove command

* Fix path in tut10 notebook

* Fix typo in node name in tut14

* Third block was too long, rebancing

* Reduce GoT dataset even more, why wasting time after all...

* Fix paths in tut10 again

* do git clean to make sure to cleanup everything (breaks post Python)

* Remove ES file with bad permission at the end of the run

* Split first block, takes >30mins

* take out tut15 for a moment, has an actual bug

* typo

* Forgot rm option

* Simply remove all ES files

* Improve logs of GoT reduction

* Exclude also tut16 from cache to try fix bug

* Replace ll with ls

* Reintroduce 15_TableQA

* Small regrouping

* regrouping to make the min num of runners go for about 30mins

* Add cron schedule and PR paths conditions

* Add some timing information

* Separate tutorials by diff and tutorials by cron

* temp add pull_request to tutorials nightly

* Add badge in README to keep track of the nightly tutorials run

* Remove prefixes from data folder names

* Add fetch depth to get diff with master

* Fix paths again

* typo

* Exclude long-running ones

* Typo

* Fix tutorials.yml as well

* Use head_ref

* Using an action for now

* exclude other files

* Use only the correct command to run the tutorial

* Add long running tutorials in separate runners, just for experiment

* Factor out the complex bash script

* Pass the python path to the bash script

* Fix paths

* adding log statement

* Missing dollarsign

* Resetting variable in loop

* using mini GoT dataset and improving bash script

* change dataset name

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2022-06-15 09:53:36 +02:00
Sara Zan
89bb1ca139
[CI refactoring] Improve autoformat.yml (#2556)
* Restructure autoformat to run a single script

* Reduce diff for autoforma.yml

* Reduce diff on linux_ci.yml
2022-05-18 20:02:43 +02:00