yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-08-30 03:20:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
661 Commits 8 Branches 0 Tags
Commit Graph

48 Commits

Author SHA1 Message Date
rasbt
1a962f3983 update lora experiments 2024-08-16 08:57:46 -05:00
TITC
70a7cddf9c typo fix (#323) 2024-08-16 06:57:30 -05:00
TITC
e1072bbcd6 typo fix (#321) 
* typo fix

experiment 5 is the same size as experiment 1, both are 124 Million. and experiment 6 is 355M ~3x 124M.

* typo fix

all layer FT vs all layer FT, "slightly worse by 1.3% " indicates it's exp 5 vs exp 9

* exp 5 vs 9

* Update ch06/02_bonus_additional-experiments/README.md

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-08-15 08:45:34 -05:00
Sebastian Raschka
e4cfcc5b66 Update README.md 2024-08-15 08:22:47 -05:00
Sebastian Raschka
895512ebee Revert accidental edit 2024-08-06 19:54:34 -05:00
rasbt
7a5771932b note about logistic sigmoid 2024-08-06 19:48:30 -05:00
Sebastian Raschka
192bdc3501 improve gradient accumulation (#300) 2024-08-05 18:27:20 -05:00
Sebastian Raschka
d0f3b034d8 Add download help message (#274) 2024-07-19 08:29:29 -05:00
Jeroen Van Goey
70cfced899 fix typos, add codespell pre-commit hook (#264) 
* fix typos, add codespell pre-commit hook

* Update .pre-commit-config.yaml

---------

Co-authored-by: Sebastian Raschka <mail@sebastianraschka.com>
 2024-07-16 07:07:04 -05:00
Daniel Kleine
87f47a281a fixed spelling typos (#258) 2024-07-03 07:47:33 -05:00
Daniel Kleine
fb4e37ae15 fixed minor issues (#252) 
* fixed typo

* fixed var name in md text
 2024-06-29 06:38:25 -05:00
Daniel Kleine
73be1c592f fixed num_workers (#229) 
* fixed num_workers

* ch06 & ch07: added num_workers to create_dataloader_v1
 2024-06-19 17:36:46 -05:00
Daniel Kleine
e5c3c5ce99 minor bug fixes (#207) 
* fixed path arg for create_dataset_csvs()

* updated assign_check() to remove user warning
 2024-06-12 06:27:56 -05:00
Daniel Kleine
79210eb393 fixes for code (#206) 
* updated .gitignore

* removed unused GELU import

* fixed model_configs, fixed all tensors on same device

* removed unused tiktoken

* update

* update hparam search

* remove redundant tokenizer argument

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>
 2024-06-11 20:59:48 -05:00
rasbt
b9ed5811c3 fix gradient comment 2024-06-09 20:23:18 -05:00
Sebastian Raschka
40ba3a4068 Remove leftover instances of self.tokenizer (#201) 
* Remove leftover instances of self.tokenizer

* add endoftext token
 2024-06-08 14:57:34 -05:00
Daniel Kleine
e637393056 updated Dockerfile and Additional Classification Finetuning Experiments (#195) 
* accuracy to .2f

* added curl
 2024-06-05 20:17:49 -05:00
Sebastian Raschka
605b4b03d7 Update README.md 2024-05-25 10:45:29 -05:00
rasbt
30ba6a3f4b trainable token -> trainable token position 2024-05-23 11:43:20 -05:00
rasbt
c35cf65dbf add assertion about data set length 2024-05-23 06:50:43 -05:00
Daniel Kleine
a81ba9bd8b fixed last_two_blocks 2024-05-22 02:02:43 +00:00
rasbt
80d857c605 fix table alignment 2024-05-21 19:51:22 -05:00
rasbt
8a27baf7c2 experiment with last two blocks 2024-05-21 19:49:34 -05:00
Daniel Kleine
130a69ce27 improved readability of Additional Experiments table 2024-05-21 19:26:25 +00:00
rasbt
7b9b53c9f2 update lora init 2024-05-19 20:11:56 -05:00
rasbt
bc5cbbf1bd change defaults to 0 temp 2024-05-19 09:04:49 -05:00
rasbt
59f5ed8d68 use default value for temperature 2024-05-19 08:48:10 -05:00
rasbt
faffebae4b add ignore index experiment 2024-05-19 07:24:49 -05:00
rasbt
bdea15f6c6 new experiment w/o causal mask 2024-05-18 17:03:36 -05:00
Sebastian Raschka
00a466f0b9 fix row number typo 2024-05-18 15:54:13 -05:00
rasbt
9d84935b69 add eos_id option for ch07 2024-05-18 12:35:40 -05:00
rasbt
10ebc47720 Add experiment with gradient accumulation 2024-05-17 21:31:22 -05:00
rasbt
623bc19665 fix no padding option 2024-05-17 21:06:51 -05:00
rasbt
f1db50fe9a fix indent 2024-05-17 07:58:01 -05:00
rasbt
2653c36957 Add new experiment without padding 2024-05-17 07:55:51 -05:00
Sebastian Raschka
7370ad1615 val before test acc 2024-05-13 07:36:18 -05:00
Sebastian Raschka
03d3b6ca72 Update README.md 2024-05-11 06:42:05 -05:00
Sebastian Raschka
cf299777b6 Move GPU column so that test accuracy is always visible 2024-05-11 06:41:25 -05:00
Sebastian Raschka
cfa1d0f997 Add LoRA experiments (#151) 
* Add LoRA experiments

* Update ch06/02_bonus_additional-experiments/additional-experiments.py
 2024-05-10 07:26:41 -05:00
rasbt
75545e4c1b experiments with largest model 2024-05-09 07:40:09 -05:00
rasbt
9457676640 ouput -> output 2024-05-05 12:21:10 -05:00
rasbt
354bb35726 use training set len 2024-04-29 21:50:07 -05:00
Sebastian Raschka
a5b353667d Rename drop_resid to drop_shortcut (#136) 2024-04-28 14:31:27 -05:00
Sebastian Raschka
1887b89af6 Update README.md 2024-04-27 07:59:42 -05:00
rasbt
51f4980a42 style checks 2024-04-24 07:48:51 -05:00
rasbt
d311bae25a add usage 2024-04-24 07:27:04 -05:00
rasbt
fb54b064c9 add more experiments 2024-04-24 07:23:11 -05:00
rasbt
881075aeb0 rename folder 2024-04-23 21:02:57 -05:00