From cd7ea15e8de0ae74b59bf2500a87985adef0a33d Mon Sep 17 00:00:00 2001 From: rasbt Date: Mon, 13 May 2024 08:50:55 -0500 Subject: [PATCH] add readme --- ch05/01_main-chapter-code/README.md | 6 ++++-- ch05/README.md | 5 +++-- ch06/01_main-chapter-code/README.md | 9 ++++++++- ch06/README.md | 5 +++++ 4 files changed, 20 insertions(+), 5 deletions(-) create mode 100644 ch06/README.md diff --git a/ch05/01_main-chapter-code/README.md b/ch05/01_main-chapter-code/README.md index 34f6690..74597b5 100644 --- a/ch05/01_main-chapter-code/README.md +++ b/ch05/01_main-chapter-code/README.md @@ -1,7 +1,9 @@ # Chapter 5: Pretraining on Unlabeled Data - [ch05.ipynb](ch05.ipynb) contains all the code as it appears in the chapter -- [previous_chapters.py](previous_chapters.py) is a Python module that contains the `MultiHeadAttention` module from the previous chapter, which we import in [ch05.ipynb](ch05.ipynb) to pretrain the GPT model -- [gpt_train.py](gpt_train.py) is a standalone Python script file with the code that we implemented in [ch05.ipynb](ch05.ipynb) to train the GPT model +- [previous_chapters.py](previous_chapters.py) is a Python module that contains the `MultiHeadAttention` module and `GPTModel` class from the previous chapters, which we import in [ch05.ipynb](ch05.ipynb) to pretrain the GPT model +- [gpt_train.py](gpt_train.py) is a standalone Python script file with the code that we implemented in [ch05.ipynb](ch05.ipynb) to train the GPT model (you can think of it as a code file summarizing this chapter) - [gpt_generate.py](gpt_generate.py) is a standalone Python script file with the code that we implemented in [ch05.ipynb](ch05.ipynb) to load and use the pretrained model weights from OpenAI +- [gpt_download.py](gpt_download.py) contains the utility functions for downloading the pretrained GPT model weights +- [exercise-solutions.ipynb](exercise-solutions.ipynb) contains the exercise solutions for this chapter diff --git a/ch05/README.md b/ch05/README.md index 805b8f4..1a5a8a2 100644 --- a/ch05/README.md +++ b/ch05/README.md @@ -3,5 +3,6 @@ - [01_main-chapter-code](01_main-chapter-code) contains the main chapter code - [02_alternative_weight_loading](02_alternative_weight_loading) contains code to load the GPT model weights from alternative places in case the model weights become unavailable from OpenAI - [03_bonus_pretraining_on_gutenberg](03_bonus_pretraining_on_gutenberg) contains code to pretrain the LLM longer on the whole corpus of books from Project Gutenberg -- [04_learning_rate_schedulers] contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping -- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script \ No newline at end of file +- [04_learning_rate_schedulers](04_learning_rate_schedulers) contains code implementing a more sophisticated training function including learning rate schedulers and gradient clipping +- [05_bonus_hparam_tuning](05_bonus_hparam_tuning) contains an optional hyperparameter tuning script + diff --git a/ch06/01_main-chapter-code/README.md b/ch06/01_main-chapter-code/README.md index 9ccd531..27dd034 100644 --- a/ch06/01_main-chapter-code/README.md +++ b/ch06/01_main-chapter-code/README.md @@ -1 +1,8 @@ -In progress. \ No newline at end of file +# Chapter 6: Finetuning for Classification + +- [ch06.ipynb](ch06.ipynb) contains all the code as it appears in the chapter +- [previous_chapters.py](previous_chapters.py) is a Python module that contains the GPT model we coded and trained in previous chapters, alongside many utility functions, which we reuse in this chapter +- [gpt-class-finetune.py](gpt-class-finetune.py) is a standalone Python script file with the code that we implemented in [ch06.ipynb](ch06.ipynb) to finetune the GPT model (you can think of it as a chapter summary) +- [gpt_download.py](gpt_download.py) contains the utility functions for downloading the pretrained GPT model weights +- [exercise-solutions.ipynb](exercise-solutions.ipynb) contains the exercise solutions for this chapter + diff --git a/ch06/README.md b/ch06/README.md new file mode 100644 index 0000000..6c852c2 --- /dev/null +++ b/ch06/README.md @@ -0,0 +1,5 @@ +# Chapter 6: Finetuning for Classification + +- [01_main-chapter-code](01_main-chapter-code) contains the main chapter code +- [02_bonus_additional-experiments](02_bonus_additional-experiments) includes additional experiments (e.g., training the last vs first token, extending the input length, etc.) +- [03_bonus_imdb-classification](03_bonus_imdb-classification) compares the LLM from chapter 6 with other models on a 50k IMDB movie review sentiment classification dataset \ No newline at end of file