多斜率自适应卷积神经网络激活函数

针对卷积神经网络中激活函数无法有效为处于不同激活程度的像素点提供特定梯度响应的问题,设计了一种由多个分段线性函数组成的自适应激活函数.首先依据像素激活值的取值范围,自适应地生成多个独立的激活域,各个激活域的并集包含激活图中全体像素点的激活值;随后在每个激活域中学习一个特定的线性函数,为处于该激活域中的像素点提供特定的梯度响应;最后以NIN网络和ResNet18网络为例,在CIFAR 10和CIFAR 100数据集上,验证所提激活函数的性能.实验结果表明,与现有激活函数相比,本文提出的激活函数,由于能够更好地为处于不同激活程度的像素点提供适当的梯度响应,使分类准确率在NIN网络上分别达到87.96%和69.01%,在ResNet18网络上分别达到88.56%和73.54%.

Multi Slope Adaptive Convolutional Neural Network Activation Function

Aiming at the problem that the activation function in Convolutional Neural Networks can’t effectively provide a specific gradient response for pixels at different activation levels, an adaptive activation function is designed, which is composed of multiple piecewise linear functions. Firstly, according to the ranges of pixel activation value, multiple independent activation domains are adaptively generated, and the union of each activation domain contains the activation values of all pixels in the activation map; then a specific linear function is learned in each activation domain in order to provide a specific gradient response for the pixels in the activation domain; finally taking the NIN and ResNet18 as examples, on the CIFAR 10 and CIFAR 100 dataset, the performance of the proposed activation function is verified. Experimental results show that, compared with the existing activation function, the activation function proposed in this paper can provide a suitable gradient response for pixels with different activation levels, so that the classification accuracies reach 87.96% and 69.01% on the NIN respectively. The accuracies reach 88.56% and 73.54% on the ResNet18, respectively.