Swish function

The swish function is a family of mathematical function defined as follows:

The swish function

${\displaystyle \operatorname {swish} _{\beta }(x)=x\operatorname {sigmoid} (\beta x)={\frac {x}{1+e^{-\beta x}}}.}$ ^[1]

where ${\displaystyle \beta }$ can be constant (usually set to 1) or trainable.

The swish family was designed to smoothly interpolate between a linear function and the ReLU function.

When considering positive values, Swish is a particular case of doubly parameterized sigmoid shrinkage function defined in ^{[2] : Eq 3}. Variants of the swish function include Mish. ^[3]

Special values

For β = 0, the function is linear: f(x) = x/2.

For β = 1, the function is the Sigmoid Linear Unit (SiLU).

With β → ∞, the function converges to ReLU.

Thus, the swish family smoothly interpolates between a linear function and the ReLU function. ^[1]

Since ${\displaystyle \operatorname {swish} _{\beta }(x)=\operatorname {swish} _{1}(\beta x)/\beta }$, all instances of swish have the same shape as the default ${\displaystyle \operatorname {swish} _{1}}$, zoomed by ${\displaystyle \beta }$. One usually sets ${\displaystyle \beta >0}$. When ${\displaystyle \beta }$ is trainable, this constraint can be enforced by ${\displaystyle \beta =e^{b}}$, where ${\displaystyle b}$ is trainable.

$${\displaystyle \operatorname {swish} _{1}(x)={\frac {x}{2}}+{\frac {x^{2}}{4}}-{\frac {x^{4}}{48}}+{\frac {x^{6}}{480}}+O\left(x^{8}\right)}$$

$${\displaystyle {\begin{aligned}\operatorname {swish} _{1}(x)&={\frac {x}{2}}\tanh \left({\frac {x}{2}}\right)+{\frac {x}{2}}\\\operatorname {swish} _{1}(x)+\operatorname {swish} _{-1}(x)&=x\tanh \left({\frac {x}{2}}\right)\\\operatorname {swish} _{1}(x)-\operatorname {swish} _{-1}(x)&=x\end{aligned}}}$$

Derivatives

Because ${\displaystyle \operatorname {swish} _{\beta }(x)=\operatorname {swish} _{1}(\beta x)/\beta }$, it suffices to calculate its derivatives for the default case.$${\displaystyle \operatorname {swish} _{1}'(x)={\frac {x+\sinh(x)}{4\cosh ^{2}\left({\frac {x}{2}}\right)}}+{\frac {1}{2}}}$$so ${\displaystyle \operatorname {swish} _{1}'(x)-{\frac {1}{2}}}$ is odd.$${\displaystyle \operatorname {swish} _{1}''(x)={\frac {1-{\frac {x}{2}}\tanh \left({\frac {x}{2}}\right)}{2\cosh ^{2}\left({\frac {x}{2}}\right)}}}$$so ${\displaystyle \operatorname {swish} _{1}''(x)}$ is even.

History

SiLU was first proposed alongside the GELU in 2016, ^[4] then again proposed in 2017 as the Sigmoid-weighted Linear Unit (SiL) in reinforcement learning. ^{[5] [1]} The SiLU/SiL was then again proposed as the SWISH over a year after its initial discovery, originally proposed without the learnable parameter β, so that β implicitly equaled 1. The swish paper was then updated to propose the activation with the learnable parameter β.

In 2017, after performing analysis on ImageNet data, researchers from Google indicated that using this function as an activation function in artificial neural networks improves the performance, compared to ReLU and sigmoid functions. ^[1] It is believed that one reason for the improvement is that the swish function helps alleviate the vanishing gradient problem during backpropagation. ^[6]

See also

• Activation function
• Gating mechanism

References

1. Ramachandran, Prajit; Zoph, Barret; Le, Quoc V. (2017-10-27). "Searching for Activation Functions". arXiv: 1710.05941v2 [cs.NE].
2. Atto, Abdourrahmane M.; Pastor, Dominique; Mercier, Gregoire (March 2008). "Smooth sigmoid wavelet shrinkage for non-parametric estimation". 2008 IEEE International Conference on Acoustics, Speech and Signal Processing (PDF). pp. 3265–3268. doi:10.1109/ICASSP.2008.4518347. ISBN   978-1-4244-1483-3. S2CID   9959057.
3. Misra, Diganta (2019). "Mish: A Self Regularized Non-Monotonic Neural Activation Function". arXiv: 1908.08681 [cs.LG].
4. Hendrycks, Dan; Gimpel, Kevin (2016). "Gaussian Error Linear Units (GELUs)". arXiv: 1606.08415 [cs.LG].
5. Elfwing, Stefan; Uchibe, Eiji; Doya, Kenji (2017-11-02). "Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning". arXiv: 1702.03118v3 [cs.LG].
6. Serengil, Sefik Ilkin (2018-08-21). "Swish as Neural Networks Activation Function". Machine Learning, Math. Archived from the original on 2020-06-18. Retrieved 2020-06-18.