In the field of deep learning, people often use the term function saturation. There are many examples, but here is a one related to sigmoid:
The initial stage of growth is approximately exponential; then, as saturation begins, the growth slows, and at maturity, growth stops
None of the books/tutorials/papers explain what is the definition of the saturation. I also have not found anything by googling the term.
It is not hard to guess that it means that with the growth of $x$, the $y$ grows significantly slower (almost does not grow at all). My another idea is that this is something similar to horizontal asymptote.
So my questions are:
- Is there such a term
function saturationor something similar to it? If so, what is the definion - can I say that $\arctan$ saturates (what about $\tan$ or $arcsec$)?
2 Answers
$\begingroup$Certain functions have limits. As output values approach this limit the resulting curve starts to level out (becomes horizontal). Another way of thinking about this is that the function starts to lose its "power" to transform inputs.
Similarly in chemistry, if you keep dissolving salt in water, you will reach a point where the water becomes saturated, and its not possible to dissolve any more salt in it. The resulting rate of salination will start to slow down as the solution becomes saturated.
Here is an example of a function which starts with exponential growth and then slows as it becomes saturated.
$\endgroup$ $\begingroup$In the context of "deep learning", a "saturation function" is the common type of transfer function found in Multilayer Perceptrons (MLPs), supervised Neural Networks (NN). Hidden layer nodes and output nodes in MLPs process the data entering via the input layer using activation and transfer functions. Each processing node receives the weighted input from an array of upstream nodes. This weighted input is summed and passed as a scalar (the activation of the node) to the transfer function. (An MLP weight is similar in concept to a regression coefficient). So, every NN transfer function is a function of one argument (the scalar input), that argument being constructed by the activation function out of several inputs.
As already stated, the MLP transfer function is usually a "saturation type", sometimes called a squashing function. A saturation function is one with finite upper and lower bounds. As you noted, the usual choice of transfer function is a sigmoidal (S-shaped) response function, such as the logistic and hyperbolic tan functions, which transform an input in the range (−∞, +∞) to the range (0, 1) and (−1, 1) respectively. They thus keep the response of the MLP bounded. Alternatives are the Heaviside step function and the piecewise linear (ramp) function. (So, one type of combined activation/transfer function is functionally analogous to a Generalized Linear logistic regression.)
As a useful reference, I would suggest section "13.4 The Multilayer Perceptron" (pp 228-246) in Natural Computing Algorithms by A. Brabazon, M. O'Neill and S. McGarraghy (2015, Springer-Verlag).
$\endgroup$
