mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-12-12 23:42:17 +00:00

History

* updated .gitignore

* removed unused GELU import

* fixed model_configs, fixed all tensors on same device

* removed unused tiktoken

* update

* update hparam search

* remove redundant tokenizer argument

---------

Co-authored-by: rasbt <mail@sebastianraschka.com>

2024-06-11 20:59:48 -05:00

01_main-chapter-code

fix typo in comment

2024-06-09 06:14:02 -05:00

02_alternative_weight_loading

fixes for code (#206 )

2024-06-11 20:59:48 -05:00

03_bonus_pretraining_on_gutenberg

Rename drop_resid to drop_shortcut (#136 )

2024-04-28 14:31:27 -05:00

04_learning_rate_schedulers

Add and link bonus material (#84 )

2024-03-23 07:27:43 -05:00

05_bonus_hparam_tuning

fixes for code (#206 )

2024-06-11 20:59:48 -05:00

README.md

add readme

2024-05-13 08:50:55 -05:00

README.md

Chapter 5: Pretraining on Unlabeled Data

01_main-chapter-code contains the main chapter code
02_alternative_weight_loading contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI
03_bonus_pretraining_on_gutenberg contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg
04_learning_rate_schedulers contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping
05_bonus_hparam_tuning contains an optional hyperparameter tuning script