haystack/releasenotes/notes/load-tokenizer-if-not-load-by-transformers-5841cdc9ff69bcc2.yaml

---
enhancements:
  - |
    Allow loading Tokenizers for prompt models not natively supported by transformers by setting `trust_remote_code` to `True`.
feat: enable loading tokenizer for models that are not supported by the transformers library (#5314) * add tokenizer load * change import order * move imports * refactor code * import lib * remove pretrainedmodel * fix linting * update patch * fix order * remove tokenizer class * use tokenizer class * no copy * add case for model is an instance * fix optional * add ut * set default to None * change models * Update haystack/nodes/prompt/invocation_layer/hugging_face.py Co-authored-by: bogdankostic <bogdankostic@web.de> * Update haystack/nodes/prompt/invocation_layer/hugging_face.py Co-authored-by: bogdankostic <bogdankostic@web.de> * add unit tests * add unit tests * remove lib * formatting * formatting * formatting * add release note * Update releasenotes/notes/load-tokenizer-if-not-load-by-transformers-5841cdc9ff69bcc2.yaml Co-authored-by: bogdankostic <bogdankostic@web.de> --------- Co-authored-by: bogdankostic <bogdankostic@web.de> 2023-08-02 17:42:23 +08:00			`---`
			`enhancements:`
			`- \|`
			Allow loading Tokenizers for prompt models not natively supported by transformers by setting `trust_remote_code` to `True`.