mirror of
https://github.com/rasbt/LLMs-from-scratch.git
synced 2025-08-29 19:10:19 +00:00
27 lines
938 B
Markdown
27 lines
938 B
Markdown
# More Efficient Multi-Head Attention Implementations
|
|
|
|
- [mha-implementations.ipynb](mha-implementations.ipynb) contains and compares different implementations of multi-head attention
|
|
|
|
|
|
|
|
### Summary
|
|
|
|
The figures below summarize the performance benchmarks (lower is better).
|
|
|
|
|
|
|
|
#### Forward pass only
|
|
|
|
<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/1_forward-only.webp?1" width="500px"></a>
|
|
|
|
|
|
#### Forward and backward pass
|
|
|
|
<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/2_forward-and-backward.webp?1" width="500px"></a>
|
|
|
|
|
|
#### Forward and backward pass after compilation
|
|
|
|
<a href="mha-implementations.ipynb"><img src="https://sebastianraschka.com/images/LLMs-from-scratch-images/bonus/mha-benchmark/3_forward-and-backward-compiled.webp?1" width="500px"></a>
|
|
|