The sigmoid activation function (squeezing function)

So as discussed earlier, we want our artificial neurons to output a continuous real value between 0 and 1 instead of the binary 0 or 1. In the threshold activation function we were proposing the following strategy:

if (sum(Wi * Xi) < threshold) {
	return 0;
} else {
	return 1;
}

Now we must choose an activation function that can replace the above function so that it will output a continuous value between 0 and 1 as output. Let us use the following function for this:

This is called the sigmoid activation function.

Note that instead of the threshold or sigmoid functions, we can select any similar function of our choice to serve the purpose of an activation function. But there are many desirable features for an activation function. We selected the sigmoid function because of its simplicity and also because of some other reasons that will be discussed in later chapters.

So in effect our neuron output function will look like:

double Neuron::Output(double * Input, int Num_Inputs)
{
	double Sum=0;
	for(int i=1;i<=Num_Inputs;i++)
		Sum += Weight[i] * Input[i-1];
	return (1/ (1 + exp(-Sum)));
}

Note that in the above output function we did not use the value of Weight[0]. More on that later.

Next Page >> The artificial neuron

Introduction to neural networks << Previous Page