LLMs-from-scratch/ch05/README.md

# Chapter 5: Pretraining on Unlabeled Data

&nbsp;
## Main Chapter Code

- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code

&nbsp;
## Bonus Materials

- [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI
- [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg
- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping
- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script
- [06_user_interface](06_user_interface) implements an interactive user interface to interact with the pretrained LLM
- [07_gpt_to_llama](07_gpt_to_llama) contains a step-by-step guide for converting a GPT architecture implementation to Llama 3.2 and loads pretrained weights from Meta AI
- [08_memory_efficient_weight_loading](08_memory_efficient_weight_loading) contains a bonus notebook showing how to load model weights via PyTorch's `load_state_dict` method more efficiently
hparam tuning script 2024-02-27 08:51:03 -06:00			`# Chapter 5: Pretraining on Unlabeled Data`

Update bonus section formatting (#400) 2024-10-12 10:26:08 -05:00			` `
distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00			`## Main Chapter Code`

Add alternative weight loading strategy as backup (#82) 2024-03-20 08:43:18 -05:00			`- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code`
distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00
Update bonus section formatting (#400) 2024-10-12 10:26:08 -05:00			` `
distinguish better between main chapter code and bonus materials 2024-06-11 21:07:42 -05:00			`## Bonus Materials`

Add alternative weight loading strategy as backup (#82) 2024-03-20 08:43:18 -05:00			`- [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI`
			`- [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg`
add readme 2024-05-13 08:50:55 -05:00			`- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping`
			`- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script`
Add chatpgpt-like user interface (#360) * Add chatpgpt-like user interface * fixes 2024-09-17 08:26:44 -05:00			`- [06_user_interface](06_user_interface) implements an interactive user interface to interact with the pretrained LLM`
Llama 3 (#384) * Implement Llama 3.2 * Add Llama 3.2 files * exclude IMDB link because stanford website seems down 2024-10-05 07:52:15 -05:00			`- [07_gpt_to_llama](07_gpt_to_llama) contains a step-by-step guide for converting a GPT architecture implementation to Llama 3.2 and loads pretrained weights from Meta AI`
Memory efficient weight loading (#401) * memory efficient weight loading * remove unused code 2024-10-14 10:30:25 -05:00			- [08_memory_efficient_weight_loading](08_memory_efficient_weight_loading) contains a bonus notebook showing how to load model weights via PyTorch's `load_state_dict` method more efficiently