yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-11-14 17:13:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

add clarifying note about GELU

Browse Source
This commit is contained in:
rasbt 2024-06-29 07:14:36 -05:00
parent 1e61943bf2
commit 1ffb7500e4
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ch04/01_main-chapter-code/ch04.ipynb
View File

@ -667,7 +667,7 @@
"metadata": {}, "metadata": {},
"source": [ "source": [
"- As we can see, ReLU is a piecewise linear function that outputs the input directly if it is positive; otherwise, it outputs zero\n", "- As we can see, ReLU is a piecewise linear function that outputs the input directly if it is positive; otherwise, it outputs zero\n",
"- GELU is a smooth, non-linear function that approximates ReLU but with a non-zero gradient for negative values\n", "- GELU is a smooth, non-linear function that approximates ReLU but with a non-zero gradient for negative values (except at approximately -0.75)\n",
"\n", "\n",
"- Next, let's implement the small neural network module, `FeedForward`, that we will be using in the LLM's transformer block later:" "- Next, let's implement the small neural network module, `FeedForward`, that we will be using in the LLM's transformer block later:"
] ]