在二分類任務中應用最廣泛的啟用函式是sigmoid，而在多分類任務中便是softmax函式，用於將實值隱射到(0-1)的區間內，通過交叉熵或者說是負對數極大似然估計作為損失函式，通過隨機梯度下降、牛頓法、adagrad、adadelta等優化函式進行迭代優化。
sigmoid函式公式如下：si
gmoi
d(x)
=11+
exp(
−x)
softmax函式公式如下so
ftma
x(x)
=exp
(x)∑
ni=1
exp(
x(i)
) ，其中x(i
) 代表
x 的第
i個分特徵。
在softmax函式的分子分母中都有exp函式的身影，當exp函式的輸入過大時，會導致在有限精度無法表示的情況，即常見的inf符號，在輸入資料沒有做過歸一化的情況下，某乙個特徵的取值範圍跨度很大時，其實極為容易發生上述情況。而變換處理方法其實也很簡單，只需要在分子分母的exp項輸入時減去某個值即可，最常用的是行方向上的最大值。即：so
ftma
x(x)
=exp
(x(i
)−ma
x(x(
i)))
∑ni=
1exp
(x(i
)−ma
x(x(
i)))
證明很簡單，通過上述方法變換後的值跟原始值是完全一致的，因為：ex
p(x(
i)+a
)∑ni
=1ex
p(x(
i)+a
)=ex
p(x(
i))e
xp(a
)∑ni
=1ex
p(x(
i))e
xp(a
)=ex
p(a)
exp(
x(i)
)exp
(a)∑
ni=1
exp(
x(i)
)=ex
p(x(
i))∑
ni=1
exp(
x(i)
) 下面是python**實現：
import numpy as np
defnormal_softmax
(x):
return (np.exp(x).t/np.exp(x).sum(axis=1)).t
deftransform_softmax
(x):
max_of_dim1 =np.max(x,axis=1,keepdims=true)
return (np.exp(x-max_of_dim1).t/np.exp(x-max_of_dim1).sum(axis=1,keepdims=true).t).t
data = np.random.randint(0,5,(10,5))
normal_data = normal_softmax(data)
normal_data
array([[ 0.20738626, 0.07629314, 0.07629314, 0.56373431, 0.07629314],
[ 0.23890664, 0.64941559, 0.08788884, 0.01189446, 0.01189446],
[ 0.35899605, 0.13206727, 0.01787336, 0.35899605, 0.13206727],
[ 0.39169577, 0.01950138, 0.14409682, 0.05301026, 0.39169577],
[ 0.01015357, 0.20393995, 0.20393995, 0.02760027, 0.55436626],
[ 0.09847516, 0.09847516, 0.26768323, 0.26768323, 0.26768323],
[ 0.56373431, 0.07629314, 0.20738626, 0.07629314, 0.07629314],
[ 0.43094948, 0.05832267, 0.05832267, 0.02145571, 0.43094948],
[ 0.52059439, 0.07045479, 0.19151597, 0.19151597, 0.02591887],
[ 0.46437643, 0.17083454, 0.17083454, 0.02311994, 0.17083454]])
transform_data = transform_softmax(data)
transform_data
array([[ 0.20738626, 0.07629314, 0.07629314, 0.56373431, 0.07629314],
[ 0.23890664, 0.64941559, 0.08788884, 0.01189446, 0.01189446],
[ 0.35899605, 0.13206727, 0.01787336, 0.35899605, 0.13206727],
[ 0.39169577, 0.01950138, 0.14409682, 0.05301026, 0.39169577],
[ 0.01015357, 0.20393995, 0.20393995, 0.02760027, 0.55436626],
[ 0.09847516, 0.09847516, 0.26768323, 0.26768323, 0.26768323],
[ 0.56373431, 0.07629314, 0.20738626, 0.07629314, 0.07629314],
[ 0.43094948, 0.05832267, 0.05832267, 0.02145571, 0.43094948],
[ 0.52059439, 0.07045479, 0.19151597, 0.19151597, 0.02591887],
[ 0.46437643, 0.17083454, 0.17083454, 0.02311994, 0.17083454]])
np.allclose(normal_data,transform_data)truedata = np.random.randint(10000000,10000005,(10,5))normal_data = normal_softmax(data)再檢視normal_data的值
array([[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan],
[ nan, nan, nan, nan, nan]])
transform_data = transform_softmax(data)
transform_data
array([[ 0.08015892, 0.08015892, 0.21789455, 0.02948882, 0.59229879],
[ 0.05832267, 0.02145571, 0.43094948, 0.05832267, 0.43094948],
[ 0.09501737, 0.0128592 , 0.09501737, 0.70208868, 0.09501737],
[ 0.00800164, 0.05912455, 0.43687463, 0.05912455, 0.43687463],
[ 0.14884758, 0.14884758, 0.14884758, 0.14884758, 0.40460968],
[ 0.20393995, 0.01015357, 0.55436626, 0.20393995, 0.02760027],
[ 0.44744543, 0.06055515, 0.44744543, 0.022277 , 0.022277 ],
[ 0.03106277, 0.08443737, 0.22952458, 0.6239125 , 0.03106277],
[ 0.01499127, 0.11077134, 0.01499127, 0.0407505 , 0.81849562],
[ 0.03875395, 0.77839397, 0.10534417, 0.03875395, 0.03875395]])
python實現 softmax啟用函式
softmax函式 以上版本有問題 a np.array 1010,1000,990 np.exp a np.sum np.exp a main 1 runtimewarning overflow encountered in exp main 1 runtimewarning invalid va...
python實現softmax函式 向量與矩陣形式
import numpy as np defsoftmax x 對輸入x的每一行計算softmax。該函式對於輸入是向量 將向量視為單獨的行 或者矩陣 m x n 均適用。利用softmax函式的性質 softmax x softmax x c 引數 x 乙個n維向量，或者m x n維numpy矩陣...
Softmax函式原理及Python實現
softmax函式用於將分類結果歸一化，形成乙個概率分布。作用類似於二分類中的sigmoid函式。對於乙個k維向量z，我們想把這個結果轉換為乙個k個類別的概率分布p z softmax可以用於實現上述結果，具體計算公式為 s of tmax xi exp xi jex p xj softmax x ...