[Math] Softmax Function 미분

Softmax function derivative

• 참고1: https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/
• 참고2: https://math.stackexchange.com/questions/2587001/understanding-step-in-derivation-of-softmax-function

계산에 활용되는 미분 공식

• 1.$\dfrac{d e^x}{d x}=e^x ​$
• 2.$f(x) = \dfrac{g(x)}{h(x)}\\ \dfrac{df(x)}{dx} = \dfrac{g'(x)h(x)-h'(x)g(x)}{(h(x))^2} \\ ​$
• 1. $g(a_i) = e^{a_i}, h = \Sigma_{k=1}^N e^{a_k}$

     계산

• $S_i, a_j$ 모두 벡터
• 결과값은 $i \neq j, i = j$의 경우가 각각 다름

$$S_j = \dfrac{e^{a_j}}{\Sigma^{N}_{k=1}e^{a_k}} \\ D_jS_i = \dfrac{\partial S_i}{\partial a_j} \\ \begin{align} DS = \begin{bmatrix} D_1 S_1 & \cdots & D_n S_1 \\ \vdots & \cdots & \vdots \\ D_1 S_n & \cdots & D_n S_n \end{bmatrix} \end{align}$$

case1. $i=j$

$$i = j \\ \dfrac{\partial}{\partial a_i} (\dfrac{e^{a_i}}{\Sigma_{k=1}^N e^{a_k}}) \\ =\dfrac{e^{a_i}\Sigma - e^{a_i}e^{a_i}}{\Sigma^2} \\ = \dfrac{e^{a_i}}{\Sigma} - (\dfrac{e^{a_i}}{\Sigma})^2 \\ = (1-p_i)p_i \\ \text{where } p_i = S_i$$

case2. $i \neq j$

$$i \neq j \\ \dfrac{\partial}{\partial a_i} (\dfrac{e^{a_j}}{\Sigma_{k=1}^N e^{a_k}}) \\ = \dfrac{0 - e^{a_j}e^{a_i}}{\Sigma^2} \\ = -\dfrac{e^{a_j}}{\Sigma} \times \dfrac{e^{a_i}}{\Sigma} = -p_j p_i$$