Feature Representations Using the Reflected Rectified Linear Unit (RReLU) Activation

As noted in Ref. [ 14 ], a notable property of concatenated ReLU is information preservation: Concatenated ReLU preserves both the negative and the positive linear responses after convolution (thus if the response is positive, then it is preserved through $\max \{0,x\}$ ; while if the response is negative, then it is preserved through $\max \{0,-x\}$ ). A direct consequence of information preservation is the ability to reconstruct the input to the convolutional layers, which are given the output of CReLU. Reconstruction property of a CNN implies that the features it computes are representative of the input data and this aspect of convolutional neural networks is important in order to understand the inner working of deep convolutional networks^{[ 14 , 45 ]}. Note that in order to be able to reconstruct the output of the convolutional layer after application of the activation function, the activation should preserve the features computed by the convolutional layers and hence the reconstruction property should apply to the output of the activation function. Intuitively, if an activation function satisfies the reconstruction property, then it is "lossless" and hence conducive to computation of the best possible features using the output of the convolutional layers. Next, we show that the reflected ReLU activation satisfies the reconstruction property. The proof is simple and follows from a similar result for the concatenated ReLU as stated in Ref. [ 14 ]. We state the result in the context of reconstruction from the features computed using a single convolutional layer without max-pooling. The result for the case when max-pooling is used is similar and we omit it here as it also follows through the application of a similar result stated in Ref. [ 14 ].

Theorem 1. Let $x\in {ℝ}^n$ be the input vector and let w be the $d\times k$ weight matrix such that the columns of the matrix correspond to $k$ convolutional filters. Let $x=x^{\prime }+(x-x^{\prime })$ where $x^{\prime }\in \mathrm{range}(w)$ and $(x-x^{\prime })\in$ $\ker (w)$ . Then we can reconstruct $x^{\prime }$ with $f_{\mathrm{CNN}}(x)=\mathrm{RReLU}(w^{\text{T}}x)$ .

Proof. The proof of the result follows using the reconstruction algorithm as given in Ref. [ 14 ] (replicated as Algorithm 1). Note that the RReLU is defined as

$$\sigma (x)=(\max \{0,x\},\max \{0,-x\},\min \{0,x\},\min \{0,-x\}).$$

The first part of the activation $(\max \{0,x\},$ $\max \{0,-x\})$ is nothing but the CReLU and hence by Algorithm 1, this satisfies the reconstruction property. Now referring back to Fig. 2 , we see that when applied to a single convolutional layer, RReLU consists of two parts, of which the first part is nothing but CReLU which is concatenated with its negation to get the RReLU activation. Hence in order to reconstruct the input from the output of RReLU, we simply apply Algorithm 1 to the output of the first part of the RReLU activation thereby generating the input. This completes the proof.

We are now ready to discuss the results using the reflected ReLU activation for the task of classification with the MNIST and CIFAR-10 datasets.