yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-09-25 16:15:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update hugging_face.py (#5488)

Browse Source
This commit is contained in:
Daria Fokina 2023-08-03 13:34:45 +02:00 committed by GitHub
parent 56cea8cbbd
commit 1f88cd165f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
haystack/nodes/prompt/invocation_layer/hugging_face.py
View File

@ -334,16 +334,16 @@ class HFLocalInvocationLayer(PromptModelInvocationLayer):
self, model: Union[str, "PreTrainedModel"], hub_kwargs: Dict, model_kwargs: Optional[Dict] = None
) -> Union["PreTrainedTokenizer", "PreTrainedTokenizerFast", None]:
"""
this method prepares the tokenizer before passing it to transformers' pipeline, so that the instantiated pipeline
This method prepares the tokenizer before passing it to transformers' pipeline, so that the instantiated pipeline
object has a working tokenizer.
It basically check whether the pipeline method in the transformers library will load the tokenizer.
- If yes, None will be returned, because in this case, the pipeline is intelligent enough to load the tokenizer by itself
- If not, we will load the tokenizer and an tokenizer instance is returned
It checks whether the pipeline method in the transformers library will load the tokenizer.
- If yes, None will be returned, because in this case, the pipeline is intelligent enough to load the tokenizer by itself.
- If not, we will load the tokenizer and an tokenizer instance is returned.
:param model: the name or path of the underlying model
:hub_kwargs: keyword argument related to hugging face hub, including revision, trust_remote_code and use_auth_token
:model_kwargs: keyword arguments passed to the underlying model
:param model: The name or path of the underlying model.
:hub_kwargs: Keyword argument related to hugging face hub, including revision, trust_remote_code and use_auth_token.
:model_kwargs: Keyword arguments passed to the underlying model.
"""
if isinstance(model, str):