yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-14 17:39:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix argument name in LlamaTokenizer constructor (#421)

Browse Source 
This PR addresses an oversight in the LlamaTokenizer class by changing the constructor argument from filepath to tokenizer_file.
This commit is contained in:
ROHAN WINSOR 2024-10-30 04:31:36 +05:30 committed by GitHub
parent e8c2f962e9
commit cd24a27161
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb
View File

@ -1216,7 +1216,7 @@
"\n", "\n",
"\n", "\n",
"class LlamaTokenizer:\n", "class LlamaTokenizer:\n",
" def __init__(self, filepath):\n", " def __init__(self, tokenizer_file):\n",
" sp = spm.SentencePieceProcessor()\n", " sp = spm.SentencePieceProcessor()\n",
" sp.load(tokenizer_file)\n", " sp.load(tokenizer_file)\n",
" self.tokenizer = sp\n", " self.tokenizer = sp\n",