yujunjun/LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2025-11-02 10:50:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

total training iters may equal to warmup_iters (#301)

Browse Source 
total_training_iters=20, warmup_iters=20= len(train_loader) 4 multiply n_epochs 5, then ZeroDivisionError occurred.
```shell
Traceback (most recent call last):                                                                                                                                                                                                                                                                                              
  File "LLMs-from-scratch/ch05/05_bonus_hparam_tuning/hparam_search.py", line 191, in <module>                                             
    train_loss, val_loss = train_model(                                                                                                                                                                                                                                                                                         
                           ^^^^^^^^^^^^                                                                                                                         
  File "/mnt/raid1/docker/ai/LLMs-from-scratch/ch05/05_bonus_hparam_tuning/hparam_search.py", line 90, in train_model                                                                                                                                                                                                           
    progress = (global_step - warmup_iters) / (total_training_iters - warmup_iters)                                                                             
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                                                                                                                                                             
ZeroDivisionError: division by zero 
```
This commit is contained in:
TITC 2024-08-06 20:10:05 +08:00 committed by GitHub
parent 192bdc3501
commit 7374d617b4
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ch05/05_bonus_hparam_tuning/hparam_search.py
View File

@ -82,7 +82,7 @@ def train_model(model, train_loader, val_loader, optimizer, device,
global_step += 1
# Warmup: adjust learning rate linearly
if global_step < warmup_iters:
if global_step <= warmup_iters:
lr = initial_lr + global_step * lr_increment
# Cosine annealing phase
else: