yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-08-29 19:10:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

make code more general for larger models

Browse Source
This commit is contained in:
rasbt 2024-05-05 10:18:40 -05:00
parent 3328b29521
commit a63b0f626c
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
ch06/01_main-chapter-code/ch06.ipynb
View File

@ -1316,7 +1316,8 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n", "- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
"- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"ham\"), we can replace the output layer as shown below, which will be trainable by default" "- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"ham\"), we can replace the output layer as shown below, which will be trainable by default\n",
"- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
] ]
}, },
{ {
@ -1329,7 +1330,7 @@
"torch.manual_seed(123)\n", "torch.manual_seed(123)\n",
"\n", "\n",
"num_classes = 2\n", "num_classes = 2\n",
"model.out_head = torch.nn.Linear(in_features=768, out_features=num_classes)" "model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
] ]
}, },
{ {