yujunjun/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2025-11-12 08:03:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make ctx_segment_ids a list instead of np.zeros_like

Browse Source 
* fix #1687

* fix - UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow..

* fix RuntimeError: received 0 items of ancdata

* Remove set_sharing_strategy from this branch and replace numpy.zeros_like with python numpy
This commit is contained in:
Alon Eirew 2022-01-03 09:33:55 +02:00 committed by GitHub
parent 39573cf0a9
commit a1fb70bbbd
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
haystack/modeling/data_handler/processor.py
View File

@ -1102,8 +1102,7 @@ class TextSimilarityProcessor(Processor):
return_token_type_ids=True return_token_type_ids=True
) )
# TODO check if we need this and potentially remove ctx_segment_ids = [[0] * len(ctx_inputs["token_type_ids"][0])] * len(ctx_inputs["token_type_ids"])
ctx_segment_ids = np.zeros_like(ctx_inputs["token_type_ids"], dtype=np.int32)
# get tokens in string format # get tokens in string format
tokenized_passage = [self.passage_tokenizer.convert_ids_to_tokens(ctx) for ctx in ctx_inputs["input_ids"]] tokenized_passage = [self.passage_tokenizer.convert_ids_to_tokens(ctx) for ctx in ctx_inputs["input_ids"]]