學習率迭代策略

base_lr =

0.01
lr_policy =
"fixed"

lr×g

amma

lr=lr×gamma

lr=lr×

gamm

a

base_lr =

0.01
lr_policy =
"step"
gamma =
0.1stepsize=
10000

t)=b

ase_

lr×(

tt)p

ower

lr(t)=base\_lr\times(\frac)^

lr(t)=

base

_lr×

(tt​

)pow

er

base_lr =

0.01
lr_policy =
"step"
gamma =
0.1stepsize=
10000

t)=b

ase_

lr×(

1+ga

mma×

iter

)pow

er

lr(t)=base\_lr\times(1+gamma\times iter)^

lr(t)=

base

_lr×

(1+g

amma

×ite

r)po

wer

nit_lr =5e-

4max_iter =
15000
optimizer = sgd(params=net.parameters(
), lr=init_lr, momentum=
0.9, weight_decay=
0.0005
)lambda_func =
lambda step:(1
-step/max_iter)
**0.9
scheduler = torch.optim.lr_scheduler.lambdalr(optimizer, lr_lambda=lambda_func)
for epoch in
range(50
):for i in
range
(1000):
...        optimizer.step(
)        scheduler.step(
)

需要繼承torch.optim.lr_scheduler._lrscheduler類:

from torch.optim.lr_scheduler import _lrscheduler

class
polylrdecay
(_lrscheduler)
:def
__init__
(self, optimizer, max_decay_steps, end_learning_rate=
0.0001
, power=
1)

def

get_lr
(self)
:def
step
(self, step=
none
):

l r=

lr×(

1−it

erma

x_it

er)p

ower

lr=lr\times (1-\frac)^

lr=lr×

(1−m

ax_i

teri

ter​

)pow

er

init_lr =5e-

4max_iter =
15000
optimizer = sgd(params=net.parameters(
), lr=init_lr, momentum=
0.9, weight_decay=
0.0005
)lambda_func =
lambda step:(1
-step/max_iter)
**0.9
scheduler = torch.optim.lr_scheduler.lambdalr(optimizer, lr_lambda=lambda_func)
lrs =
for epoch in
range(15
):for i in
range
(1000):
...        optimizer.step()0
]["lr"])
scheduler.step(
)plt.plot(
range
(15000
), lrs)
plt.show(
)

pytorch框架學習（12） 學習率調整策略

主要方法 1.steplr 調整方式 lr lr gamma 2.multisteplr 調整方式 lr lr gamma milestones 50 125 160 scheduler lr optim.lr scheduler.multisteplr optimizer,milestones,g...

pytorch使用 （十二）學習率調整策略

梯度下降 lr 學習率 learning rate 控制更新的步伐 class lrscheduler object def init self,optimizer,last epoch 1 def get lr self raise notimplementederror 調整方式 lr lr g...

tensorflow中學習率的調參策略

learning rate decay type 調參策略 1 exponential decay learning rate,global step,decay steps,decay rate,staircase false,name none learning rate 初始值 global ...