yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-10-29 17:01:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
658 Commits 11 Branches 0 Tags
Commit Graph

118 Commits

Author SHA1 Message Date
Sebastian Raschka
c443035d56
Note about MPS in ch06 and ch07 (#325) 2024-08-19 08:11:33 -05:00
rasbt
e7cb2ebd8d
update lora experiments 2024-08-16 08:57:46 -05:00
TITC
d2befdf00e
typo fix (#323) 2024-08-16 06:57:30 -05:00
TITC
f6526f7766
typo fix (#321) 
* typo fix

experiment 5 is the same size as experiment 1, both are 124 Million. and experiment 6 is 355M ~3x 124M.

* typo fix

all layer FT vs all layer FT, "slightly worse by 1.3% " indicates it's exp 5 vs exp 9

* exp 5 vs 9

* Update ch06/02_bonus_additional-experiments/README.md

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-08-15 08:45:34 -05:00
Sebastian Raschka
4ea526908e
Update README.md 2024-08-15 08:22:47 -05:00
TITC
38390b2a8d
track tokens seen in chapter5, track examples seen in chapter6 (#319) 2024-08-13 07:09:05 -05:00
Sebastian Raschka
c5eaae11b1
Revert accidental edit 2024-08-06 19:54:34 -05:00
rasbt
06151a809e
note about logistic sigmoid 2024-08-06 19:48:30 -05:00
Sebastian Raschka
70e5714556
improve gradient accumulation (#300) 2024-08-05 18:27:20 -05:00
Sebastian Raschka
08040f024c
Test code in pytorch 2.4 (#285) 
* test code in pytorch 2.4

* update
 2024-07-24 21:53:41 -05:00
Sebastian Raschka
8d02cb1cee
Add download help message (#274) 2024-07-19 08:29:29 -05:00
Jeroen Van Goey
48bd72c890
fix typos, add codespell pre-commit hook (#264) 
* fix typos, add codespell pre-commit hook

* Update .pre-commit-config.yaml

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-07-16 07:07:04 -05:00
Sebastian Raschka
4f0a107692 show how to use the finetuned model 2024-07-09 06:43:26 -07:00
Daniel Kleine
90b25ece3d
fixed spelling typos (#258) 2024-07-03 07:47:33 -05:00
Sebastian Raschka
f6bcdd37bd
Fix links in summary sections (#254) 2024-06-29 07:51:31 -05:00
rasbt
31806828d0
add links to summary sections 2024-06-29 07:33:26 -05:00
Daniel Kleine
1e69c8e0b5
fixed minor issues (#252) 
* fixed typo

* fixed var name in md text
 2024-06-29 06:38:25 -05:00
Daniel Kleine
06921f3333
minor markdown fixes (#236) 2024-06-21 13:55:34 -05:00
Sebastian Raschka
6c0dc2362b
Add standalone finetuning and evaluation scripts for chapter 7 (#234) 
* add finetuning and eval scripts

* update link

* update links

* fix link
 2024-06-21 05:23:24 -05:00
rasbt
283397aaf2
add main and optional sections 2024-06-19 17:48:25 -05:00
Daniel Kleine
bbb2a0c3d5
fixed num_workers (#229) 
* fixed num_workers

* ch06 & ch07: added num_workers to create_dataloader_v1
 2024-06-19 17:36:46 -05:00
Jinge Wang
10018e00ff
Fixed some typos in ch06.ipynb (#219) 2024-06-18 05:54:01 -05:00
rasbt
3e0b0c66a8
fix spelling 2024-06-18 05:50:40 -05:00
rasbt
19c5784f82
replace figure 2024-06-18 05:46:36 -05:00
Daniel Kleine
ef40f2f9ad
minor bug fixes (#207) 
* fixed path arg for create_dataset_csvs()

* updated assign_check() to remove user warning
 2024-06-12 06:27:56 -05:00
rasbt
e24fd98cdf
distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00
Daniel Kleine
dcbdc1d2e5
fixes for code (#206) 
* updated .gitignore

* removed unused GELU import

* fixed model_configs, fixed all tensors on same device

* removed unused tiktoken

* update

* update hparam search

* remove redundant tokenizer argument

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-06-11 20:59:48 -05:00
rasbt
1d278c65da
fix gradient comment 2024-06-09 20:23:18 -05:00
rasbt
1b1fd21d64
fix typo in comment 2024-06-09 06:14:02 -05:00
Sebastian Raschka
72a073bbbf
Remove leftover instances of self.tokenizer (#201) 
* Remove leftover instances of self.tokenizer

* add endoftext token
 2024-06-08 14:57:34 -05:00
Daniel Kleine
ba1c1e74aa
updated Dockerfile and Additional Classification Finetuning Experiments (#195) 
* accuracy to .2f

* added curl
 2024-06-05 20:17:49 -05:00
Sebastian Raschka
49b8e2e767
Update README.md 2024-05-25 10:45:29 -05:00
rasbt
98d453b666
update formatting 2024-05-24 07:20:37 -05:00
rasbt
b43a7a8820
trainable token -> trainable token position 2024-05-23 11:43:20 -05:00
rasbt
18e729643d
add assertion about data set length 2024-05-23 06:50:43 -05:00
rasbt
86f6c2df43
Fix device setting 2024-05-22 17:51:51 -05:00
Daniel Kleine
982cbe5e40 removed empty line 2024-05-22 16:15:13 +00:00
Daniel Kleine
195849cf8d fixed last_two_blocks 2024-05-22 02:02:43 +00:00
rasbt
caf3725001
fix table alignment 2024-05-21 19:51:22 -05:00
rasbt
725bed56f7
experiment with last two blocks 2024-05-21 19:49:34 -05:00
Daniel Kleine
f39087e573 improved readability of Additional Experiments table 2024-05-21 19:26:25 +00:00
rasbt
c2028871e4
update lora init 2024-05-19 20:11:56 -05:00
rasbt
a8a28017c0
remove duplicated text 2024-05-19 11:34:47 -05:00
rasbt
a5593f9860
change defaults to 0 temp 2024-05-19 09:04:49 -05:00
rasbt
1463b2ae47
use default value for temperature 2024-05-19 08:48:10 -05:00
rasbt
1b340c9eb6 add ignore index experiment 2024-05-19 07:24:49 -05:00
rasbt
02e6f06a11
add test mode for dataset download 2024-05-18 17:38:19 -05:00
rasbt
5ef4edf2b5
new experiment w/o causal mask 2024-05-18 17:03:36 -05:00
Sebastian Raschka
57634f2045
fix row number typo 2024-05-18 15:54:13 -05:00
rasbt
4851d5a0fa
add eos_id option for ch07 2024-05-18 12:35:40 -05:00