yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-31 01:41:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

text -> dataset

Browse Source
This commit is contained in:
rasbt 2024-05-08 08:14:03 -05:00
parent 24e9110fa8
commit a09a70c345
No known key found for this signature in database
GPG Key ID: 3C6E5C7C075611DB
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ch06/01_main-chapter-code/ch06.ipynb
View File

@ -519,7 +519,7 @@
" - 1. truncate all messages to the length of the shortest message in the dataset or batch\n", " - 1. truncate all messages to the length of the shortest message in the dataset or batch\n",
" - 2. pad all messages to the length of the longest message in the dataset or batch\n", " - 2. pad all messages to the length of the longest message in the dataset or batch\n",
"\n", "\n",
"- We choose option 2 and pad all messages to the longest message in the text\n", "- We choose option 2 and pad all messages to the longest message in the dataset\n",
"- For that, we use `<|endoftext|>` as a padding token, as discussed in chapter 2" "- For that, we use `<|endoftext|>` as a padding token, as discussed in chapter 2"
] ]
}, },