yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-09-25 08:05:45 +00:00
Code Issues Packages Projects Releases Wiki Activity
777 Commits 10 Branches 0 Tags
Commit Graph

777 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sebastian Raschka
7e2092dd01 More pythonic way to find the longest sequence (#512) 
* More pythonic way to find the longest sequence

* pep8 fix
 2025-02-01 10:22:47 -06:00
Sebastian Raschka
c1c10c89c8 Test PyTorch nightly releases (#509) 2025-01-30 12:45:48 -06:00
Sebastian Raschka
fd24a3679a Alternative weight loading via .safetensors (#507) 2025-01-29 08:15:29 -06:00
Sebastian Raschka
ebfc50f076 Fix default argument in ex 7.2 (#506) 2025-01-25 10:46:48 -06:00
Sebastian Raschka
d387696f93 A few cosmetic updates (#504) 2025-01-23 09:38:55 -06:00
Sebastian Raschka
bb13ff519e Test for PyTorch 2.6 release candidate (#500) 
* Test for PyTorch 2.6 release candidate

* update

* update

* remove extra added file
 2025-01-22 18:37:48 -06:00
Sebastian Raschka
dcaac28b92 Bonus material: extending tokenizers (#496) 
* Bonus material: extending tokenizers

* small wording update
 2025-01-22 09:26:54 -06:00
Daniel Kleine
9175590ea4 add GPT2TokenizerFast to BPE comparison (#498) 
* added HF BPE Fast

* update benchmarks

* add note about performance

* revert accidental changes

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-22 09:26:44 -06:00
Austin Welch
654734053a fix: preserve newline tokens in BPE encoder (#495) 
* fix: preserve newline tokens in BPE encoder

* further fixes

* more fixes

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-21 12:47:15 -06:00
Daniel Kleine
3f9facbc55 BPE: fixed typo (#492) 
* fixed typo

* use rel path if exists

* mod gitignore and use existing vocab files

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-20 20:49:53 -06:00
Sebastian Raschka
b17d097742 Implementingthe BPE Tokenizer from Scratch (#487) 2025-01-17 12:22:00 -06:00
rvaneijk
f936ad4b4e 04_optional-aws-sagemaker-notebook (#451) 
* 04_optional-aws-sagemaker-notebook

* Update setup/04_optional-aws-sagemaker-notebook/cloudformation-template.yml

* Update README.md

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2025-01-17 10:07:10 -06:00
Sebastian Raschka
37aed8fc2c Include mathematical breakdown for exercise solution 4.1 (#483) 2025-01-14 19:23:00 -06:00
Henry Shi
15af754304 Print out embeddings for more illustrative learning (#481) 
* print out embeddings for illustrative learning

* suggestion print embeddingcontents

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-13 14:44:06 -06:00
rasbt
bed5f89378 fix reward margins plot label in dpo nb 2025-01-12 14:04:05 -06:00
Sebastian Raschka
992f3068d1 Auto download DPO dataset if not already available in path (#479) 
* Auto download DPO dataset if not already available in path

* update tests to account for latest HF transformers release in unit tests

* pep 8
 2025-01-12 12:27:28 -06:00
Sebastian Raschka
05f2a398b8 adds no-grad context for reference model to DPO (#473) 2025-01-07 20:49:01 -06:00
Sebastian Raschka
b3dc7073ee fix ch07 unit test (#470) 2025-01-05 17:40:57 -06:00
Sebastian Raschka
7659af7cdd Add backup URL for gpt2 weights (#469) 
* Add backup URL for gpt2 weights

* newline
 2025-01-05 11:28:09 -06:00
QS
976c92010c typo fixed (#468) 
* typo fixed

* only update plot

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-05 09:17:13 -06:00
Tao Qian
65ee619d3b Minor readability improvement in dataloader.ipynb (#461) 
* Minor readability improvement in dataloader.ipynb

- The tokenizer and encoded_text variables at the root level are unused.
- The default params for create_dataloader_v1 are confusing, especially for the default batch_size 4, which happens to be the same as the max_length.

* readability improvements

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2025-01-04 11:26:10 -06:00
Sebastian Raschka
05a816e270 fix misplaced parenthesis and update license (#466) 2025-01-04 11:14:08 -06:00
casinca
57fdd94358 [minor] typo & comments (#441) 
* typo & comment

- safe -> save
- commenting code: batch_size, seq_len = in_idx.shape

* comment

- adding # NEW for assert num_heads % num_kv_groups == 0

* update memory wording

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-11-18 19:52:42 +09:00
Daniel Kleine
dcef9b7d6f Fixed command for row 16 additional experiment (#439) 
* fixed command for row 16 experiment

* Update README.md

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-11-17 06:50:00 +09:00
Sebastian Raschka
841b64f2b9 Add flexible padding bonus experiment (#438) 
* Add flexible padding bonus experiment

* fix links
 2024-11-15 08:51:01 +09:00
Sebastian Raschka
541b237eff Add utility to prevent double execution of certain cells (#437) 2024-11-14 19:56:49 +09:00
Sebastian Raschka
129d0d740f Add missing device transfer in gpt_generate.py (#436) 2024-11-14 19:12:53 +09:00
Sebastian Raschka
6d1bb2cfb4 Add chapter names 2024-11-08 08:39:34 -06:00
Sebastian Raschka
74e04a9169 Add "What's next" section (#432) 
* Add What's next section

* Delete appendix-D/01_main-chapter-code/appendix-D-Copy2.ipynb

* Delete ch03/01_main-chapter-code/ch03-Copy1.ipynb

* Delete appendix-D/01_main-chapter-code/appendix-D-Copy1.ipynb

* Update ch07.ipynb

* Update ch07.ipynb
 2024-11-07 20:12:59 -06:00
rasbt
5348565e0f add dropout scaling note 2024-11-06 05:52:47 -06:00
casinca
2fd07e2cfd potential little fixes appendix-D4 .ipynb (#427) 
* Update appendix-D.ipynb

- lr missing argument for passing peak_lr to the optimizer
- filling 1 step gap for gradient clipping

* adjustments

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-11-03 12:12:58 -06:00
Sebastian Raschka
95f8a4084f Update CITATION.cff 2024-11-01 21:32:17 -05:00
Sebastian Raschka
9d2fd4c22e Update CITATION.cff 2024-11-01 21:29:22 -05:00
rasbt
81e78d0ad8 Add citation file 2024-11-01 21:21:25 -05:00
rasbt
50500e94b5 Note about warm-up steps 2024-11-01 16:47:12 -05:00
Daniel Kleine
7e6f8ce020 updated RoPE statement (#423) 
* updated RoPE statement

* updated .gitignore

* Update ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-10-30 08:00:08 -05:00
Sebastian Raschka
62e44b2415 Update README.md 2024-10-29 20:20:48 -05:00
ROHAN WINSOR
e85d154522 Fix argument name in LlamaTokenizer constructor (#421) 
This PR addresses an oversight in the LlamaTokenizer class by changing the constructor argument from filepath to tokenizer_file.
 2024-10-29 18:01:36 -05:00
Daniel Kleine
2b24a7ef30 minor fixes: Llama 3.2 standalone (#420) 
* minor fixes

* reformat rope base as float

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-10-25 21:08:06 -05:00
Sebastian Raschka
75ede3e340 RoPE theta rescaling (#419) 
* rope fixes

* update

* update

* cleanup
 2024-10-25 15:27:23 -05:00
Daniel Kleine
0ed1e0d099 fixed typos (#414) 
* fixed typos

* fixed formatting

* Update ch03/02_bonus_efficient-multihead-attention/mha-implementations.ipynb

* del weights after load into model

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-10-24 18:23:53 -05:00
Daniel Kleine
8b60460319 Updated Llama 2 to 3 paths (#413) 
* llama 2 and 3 path fixes

* updated llama 3, 3.1 and 3.2 paths

* updated .gitignore

* Typo fix

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-10-24 07:40:08 -05:00
Sebastian Raschka
632d7772b2 Update test-requirements-extra.txt 2024-10-23 19:19:58 -05:00
Sebastian Raschka
f8bdfe12e1 RoPE updates (#412) 
* RoPE updates

* Apply suggestions from code review

* updates

* updates

* updates
 2024-10-23 18:07:49 -05:00
Sebastian Raschka
6dd3fbd79d Update tests.py 2024-10-23 07:48:33 -05:00
Daniel Kleine
cba4f89514 updates for PyTorch 2.5 (#408) 
* updated Dockerfile

* updated MHA implementations for PT 2.5

* fixed typo

* update installation instruction

* Update setup/03_optional-docker-environment/.devcontainer/Dockerfile

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-10-22 20:23:31 -05:00
Sebastian Raschka
9726ca6546 RoPE increase (#407) 2024-10-21 19:58:38 -05:00
Sebastian Raschka
208d8030c9 Set sampler in DDP example 2024-10-21 09:26:01 -05:00
Sebastian Raschka
3c3dae0967 Add mean pooling experiment to classifier bonus experiments (#406) 
* Add mean pooling experiment to classifier bonus  experiments

* formatting

* add average embeddings option

* pep8
 2024-10-20 11:04:18 -05:00
Sebastian Raschka
c4bac22bff Test PyTorch 2.5 (#405) 2024-10-20 10:23:31 -05:00