transformers:2.10.0
这里就不使用pytorch中的dataset和dataloader了,简单的模拟下:
from transformers import AdanW, get_linear_schedule_with_warmup
optimizer = AdamW(参数, lr=lr, eps=adam_epsilon)
len_dataset = 3821 # 可以根据pytorch中的len(Dataset)计算
epoch = 30
batch_size = 32
total_steps = (len_dataset // batch_size) * epoch if len_dataset % batch_size = 0 else (len_dataset // batch_size + 1) * epoch # 每一个epoch中有多少个step可以根据len(DataLoader)计算:total_steps = len(DataLoader) * epoch
warm_up_ratio = 0.1 # 定义要预热的step
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps = warm_up_ratio * total_steps, num_training_steps = total_steps)
具体可以去看下知乎:https://www.zhihu.com/question/338066667
关于warm up(transformers.get_linear_schedule_with_warmup)
原文:https://www.cnblogs.com/xiximayou/p/14836577.html