haystack/load-tokenizer-if-not-load-by-transformers-5841cdc9ff69bcc2.yaml at 1c69070db6ed26d8b4910c8e679ae7a11ec77dba - haystack - Gitea: Git with a cup of tea

yujunjun/haystack

mirror of https://github.com/deepset-ai/haystack.git synced 2025-10-29 16:59:47 +00:00

Fanli Lin f7fd5eeb4f

feat: enable loading tokenizer for models that are not supported by the transformers library (#5314 )

* add tokenizer load

* change import order

* move imports

* refactor code

* import lib

* remove pretrainedmodel

* fix linting

* update patch

* fix order

* remove tokenizer class

* use tokenizer class

* no copy

* add case for model is an instance

* fix optional

* add ut

* set default to None

* change models

* Update haystack/nodes/prompt/invocation_layer/hugging_face.py

Co-authored-by: bogdankostic <bogdankostic@web.de>

* Update haystack/nodes/prompt/invocation_layer/hugging_face.py

Co-authored-by: bogdankostic <bogdankostic@web.de>

* add unit tests

* add unit tests

* remove lib

* formatting

* formatting

* formatting

* add release note

* Update releasenotes/notes/load-tokenizer-if-not-load-by-transformers-5841cdc9ff69bcc2.yaml

Co-authored-by: bogdankostic <bogdankostic@web.de>

---------

Co-authored-by: bogdankostic <bogdankostic@web.de>

2023-08-02 11:42:23 +02:00

5 lines

152 B

YAML

Raw Blame History

 ---
 enhancements:
   - |
     Allow loading Tokenizers for prompt models not natively supported by transformers by setting `trust_remote_code` to `True`.