Gradient Amplification: An Efficient Way to Train Deep Neural Networks

Our experiments are performed on CIFAR 10 dataset which consists of 60 000 colored images of 10 classes with 6000 images per class and each image has 32  $\times$  32 resolution. We implement our algorithms using python and pytorch^{[ 29 ]} libraries. In our experiments, we employ several standard deep learning models and train them for 150 epochs. The number of epochs, combination of number of epochs, and learning rates can be chosen as one thinks best. In this work, the first 100 epochs have learning rate of 0.1 and the next 50 epochs have the learning rate of 0.01 (as shown in Fig. 1 ). The first 50 epochs are trained with learning rate of 0.1 without gradient amplification. This is because for the first few epochs, the model is considered to be in transient phase and the network parameters undergo significant changes. This initial transient can be considered for any number of epochs and in this work, we set it to 50 epochs. The next 50 epochs have the same learning rate of 0.1 but has gradient amplification applied during backpropagation while training the model (as shown in Fig. 2a ). After identifying the best params with gradient amplification for epochs 51-100, using those params for those epochs, we extend amplification for epochs 101-130 to identify the best params and with no amplification for epochs 131-150, as shown Fig. 2b . There are mainly three important parameters while applying gradient amplification method, namely, the type of the layers to be employed for amplification, the ratio of layers ( $\beta$ ) to be chosen from selected layers to perform amplification, and gradient amplification factor. The effects of varying each of these parameters are explained in detail in the subsections below. We run our experiments on Resnet and VGG models with different architectures.

10.26599/BDMA.2020.9020004.F001 Figure 1Experiment setting showing the number of epochs and learning rates corresponding to epochs for training all the models.

10.26599/BDMA.2020.9020004.F002 Figure 2Two-step training process carried out during performance analysis of deep learning models. Experiments are firstly executed on the models with training steps shown in Step 1. For Step 2, ratio parameters for gradient amplification which have better performance of the models in Step 1 are considered as the parameters for epochs 51-100 and experiments are performed by analyzing ratio parameters for epochs 101-130, with no amplification from epochs 131-150. These settings show the number of epochs and the learning rates corresponding to these epochs while training these models.

Here we perform three phase analysis while evaluating our model.

Phase 1 In this phase, we choose the type of layers to be considered for amplification. There are several types of layers at which amplification can be applied such as activation function layers, pooling layers, batch normalization layers, and convolution layers. Convolution layers apply kernel functions and extract important features from the data and pooling layers perform accumulation of features over a grid using several strategies, such as retrieving maximum values, minimum values, averaging, fractional pooling, and so on. Since the network parameter tuning while training can be sensitive to these values, in this work, we do not perform amplification on these layers. Batch normalization layers normalize data over a batch of inputs, and activation function layers transform data non-linearly before forwarding it to the succeeding layers. In our work, we perform gradient amplification on batch normalization and activation function layers. ReLU is the activation function used in Resnet and VGG models. From these two types of layers, either one or both of them can be considered for amplification. Once the type of the layers is selected, we now tag all the layers of the selected type to belong to the group $G$ . We now move to the next phase to determine the final amplification layers amp.

Phase 2 Once the set of layers $G$ is determined, the next task is to find the subset of layers, which gives better performance. It requires identifying subset size and selection of those many layers from $G$ . Since the size is unknown, experiments are performed by selecting the size to be a ratio of size of $G$ . This ratio, $\beta$ , is chosen from the set $\beta \in \{0,0.1,0.2,\mathrm{\dots },0.9,1\}$ . The actual size of amp is determined by the value $\beta \times \text{size}\mathrm{\_}\text{of}(G)$ . When the value is $0$ , no layers are chosen and gradient amplification is not performed. When the value is $1$ , then all the layers in $G$ are considered for amplification. $0$ is included to verify whether the model performs better without gradient amplification or vice versa. Random selection is employed to select amp subset of layers from $G$ . We perform experiments with all these sizes and select the model with the best performance.

Phase 3 In this phase, the layers amp on which gradient amplification can be applied is known. The only parameter left to explore is $\mathrm{\Gamma }$ , the factor with which gradient needs to be amplified. To reduce computation complexity in testing all the combinations of parameter values amp, $\beta$ , and $\mathrm{\Gamma }$ , firstly, experiments are performed on all combinations of amp and $\beta$ , i.e., until Phase 2, then the best models are chosen from Phase 2 and analyzed by varying $\mathrm{\Gamma }$ . The value of $\mathrm{\Gamma }$ is firstly varied from $\{1,2,3,\mathrm{\dots },10\}$ to analyze the impact of amplification and then fine-tuned by varying from $\{1.1,1.2,\mathrm{\dots },2.9,3.0\}$ to determine the value that works best during training.

Our experiments are performed on CIFAR 10 dataset which consists of 60 000 colored images of 10 classes with 6000 images per class and each image has 32  $\times$  32 resolution. We implement our algorithms using python and pytorch^{[ 29 ]} libraries. In our experiments, we employ several standard deep learning models and train them for 150 epochs. The number of epochs, combination of number of epochs, and learning rates can be chosen as one thinks best. In this work, the first 100 epochs have learning rate of 0.1 and the next 50 epochs have the learning rate of 0.01 (as shown in Fig. 1 ). The first 50 epochs are trained with learning rate of 0.1 without gradient amplification. This is because for the first few epochs, the model is considered to be in transient phase and the network parameters undergo significant changes. This initial transient can be considered for any number of epochs and in this work, we set it to 50 epochs. The next 50 epochs have the same learning rate of 0.1 but has gradient amplification applied during backpropagation while training the model (as shown in Fig. 2a ). After identifying the best params with gradient amplification for epochs 51-100, using those params for those epochs, we extend amplification for epochs 101-130 to identify the best params and with no amplification for epochs 131-150, as shown Fig. 2b . There are mainly three important parameters while applying gradient amplification method, namely, the type of the layers to be employed for amplification, the ratio of layers ( $\beta$ ) to be chosen from selected layers to perform amplification, and gradient amplification factor. The effects of varying each of these parameters are explained in detail in the subsections below. We run our experiments on Resnet and VGG models with different architectures.

10.26599/BDMA.2020.9020004.F001 Figure 1Experiment setting showing the number of epochs and learning rates corresponding to epochs for training all the models.

10.26599/BDMA.2020.9020004.F002 Figure 2Two-step training process carried out during performance analysis of deep learning models. Experiments are firstly executed on the models with training steps shown in Step 1. For Step 2, ratio parameters for gradient amplification which have better performance of the models in Step 1 are considered as the parameters for epochs 51-100 and experiments are performed by analyzing ratio parameters for epochs 101-130, with no amplification from epochs 131-150. These settings show the number of epochs and the learning rates corresponding to these epochs while training these models.

Here we perform three phase analysis while evaluating our model.

Phase 1 In this phase, we choose the type of layers to be considered for amplification. There are several types of layers at which amplification can be applied such as activation function layers, pooling layers, batch normalization layers, and convolution layers. Convolution layers apply kernel functions and extract important features from the data and pooling layers perform accumulation of features over a grid using several strategies, such as retrieving maximum values, minimum values, averaging, fractional pooling, and so on. Since the network parameter tuning while training can be sensitive to these values, in this work, we do not perform amplification on these layers. Batch normalization layers normalize data over a batch of inputs, and activation function layers transform data non-linearly before forwarding it to the succeeding layers. In our work, we perform gradient amplification on batch normalization and activation function layers. ReLU is the activation function used in Resnet and VGG models. From these two types of layers, either one or both of them can be considered for amplification. Once the type of the layers is selected, we now tag all the layers of the selected type to belong to the group $G$ . We now move to the next phase to determine the final amplification layers amp.

Phase 2 Once the set of layers $G$ is determined, the next task is to find the subset of layers, which gives better performance. It requires identifying subset size and selection of those many layers from $G$ . Since the size is unknown, experiments are performed by selecting the size to be a ratio of size of $G$ . This ratio, $\beta$ , is chosen from the set $\beta \in \{0,0.1,0.2,\mathrm{\dots },0.9,1\}$ . The actual size of amp is determined by the value $\beta \times \text{size}\mathrm{\_}\text{of}(G)$ . When the value is $0$ , no layers are chosen and gradient amplification is not performed. When the value is $1$ , then all the layers in $G$ are considered for amplification. $0$ is included to verify whether the model performs better without gradient amplification or vice versa. Random selection is employed to select amp subset of layers from $G$ . We perform experiments with all these sizes and select the model with the best performance.

Phase 3 In this phase, the layers amp on which gradient amplification can be applied is known. The only parameter left to explore is $\mathrm{\Gamma }$ , the factor with which gradient needs to be amplified. To reduce computation complexity in testing all the combinations of parameter values amp, $\beta$ , and $\mathrm{\Gamma }$ , firstly, experiments are performed on all combinations of amp and $\beta$ , i.e., until Phase 2, then the best models are chosen from Phase 2 and analyzed by varying $\mathrm{\Gamma }$ . The value of $\mathrm{\Gamma }$ is firstly varied from $\{1,2,3,\mathrm{\dots },10\}$ to analyze the impact of amplification and then fine-tuned by varying from $\{1.1,1.2,\mathrm{\dots },2.9,3.0\}$ to determine the value that works best during training.