yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-12-04 02:50:29 +00:00
Code Issues Packages Projects Releases Wiki Activity

update mha dim

Browse Source
This commit is contained in:
rasbt 2024-05-27 07:46:29 -05:00
parent 68123a5433
commit c0f564ee87
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb
View File

@ -239,7 +239,7 @@
" self.head_dim = d_out // num_heads\n", " self.head_dim = d_out // num_heads\n",
"\n", "\n",
" self.qkv = nn.Linear(d_in, 3 * d_out, bias=qkv_bias)\n", " self.qkv = nn.Linear(d_in, 3 * d_out, bias=qkv_bias)\n",
" self.proj = nn.Linear(d_in, d_out)\n", " self.proj = nn.Linear(d_out, d_out)\n",
" self.dropout = nn.Dropout(dropout)\n", " self.dropout = nn.Dropout(dropout)\n",
"\n", "\n",
" self.register_buffer(\n", " self.register_buffer(\n",