yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-24 06:20:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

add dropout scaling note

Browse Source
This commit is contained in:
rasbt 2024-11-06 05:52:47 -06:00
parent 2fd07e2cfd
commit 5348565e0f
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
ch03/01_main-chapter-code/ch03.ipynb
View File

@ -1485,7 +1485,8 @@
"id": "5a575458-a6da-4e54-8688-83e155f2de06", "id": "5a575458-a6da-4e54-8688-83e155f2de06",
"metadata": {}, "metadata": {},
"source": [ "source": [
"- If we apply a dropout rate of 0.5 (50%), the non-dropped values will be scaled accordingly by a factor of 1/0.5 = 2." "- If we apply a dropout rate of 0.5 (50%), the non-dropped values will be scaled accordingly by a factor of 1/0.5 = 2\n",
"- The scaling is calculated by the formula 1 / (1 - `dropout_rate`)"
] ]
}, },
{ {