LLMs-from-scratch/ch05/README.md

# Chapter 5: Pretraining on Unlabeled Data

## Main Chapter Code

- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code

## Bonus Materials

- [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI
- [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg
- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping
- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script
hparam tuning script 2024-02-27 08:51:03 -06:00			`# Chapter 5: Pretraining on Unlabeled Data`

distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00			`## Main Chapter Code`

Add alternative weight loading strategy as backup (#82) 2024-03-20 08:43:18 -05:00			`- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code`
distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00
			`## Bonus Materials`

Add alternative weight loading strategy as backup (#82) 2024-03-20 08:43:18 -05:00			`- [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI`
			`- [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg`
add readme 2024-05-13 08:50:55 -05:00			`- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping`
			`- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script`