Increasing Momentum-Like Factors: A Method for Reducing Training Errors on Multiple GPUs

SGD^{[ 18 ]} is one of the simplest first-order algorithms, but it introduces noise into the gradient and blocks optimization in the training progress^{[ 29 ]}. SGD randomly selects one sample or a random sample set at one time for parameter updates so that updating the parameters does not create redundancy. When the data size is large, SGD can effectively accelerate the training process.

Smith and Le^{[ 30 ]} stated that SGD should be interpreted as integrating stochastic equations. They also presented the scale of random fluctuations in the SGD dynamics as

(1) $$g=\xi \left(\frac{N}{M}-1\right)$$

where $\xi$ is the learning rate, $N$ is the size of the training set, and $M$ is the batch size. If we reduce the learning rate $\xi$ , the noise scale $g$ drops, thus leading to an improved training performance. If we keep the learning rate $\xi$ constant, then we could also increase batch size $M$ to reduce the negative impact of noise scale. By contrast, a small batch size increases the noise scale and adversely affects the training performance. According to Ref. [ 31 ], calculating the mean and variance values over a batch makes the loss calculated for a particular example dependent on other examples in the same batch. Therefore, if the batch size is large, a high dependency between samples in the batch limits the training performance. This fluctuation scale $g$ can also be considered as a noise factor. When $M\ll N$ , applying a linear scaling rule^{[ 32 ]} maintains the mean SGD weight update constant per training sample. A specific description about the linear scaling rule^{[ 32 ]} is shown in Section 2.2.

SGD has several variants^{[ 33 , 34 , 35 ]}. A common one is SGD with momentum^{[ 36 ]}; specifically, Smith and Le^{[ 30 ]} extended the traditional SGD to include momentum, and found that the noise factor $g$ changes into

(2) $$g={\displaystyle \frac{\xi }{1-m}}\left({\displaystyle \frac{N}{M}}-1\right)\approx {\displaystyle \frac{\xi N}{M(1-m)}}$$

where $m$ is the momentum. Equation ( 2 ) degenerates into Eq. ( 1 ) when $m=0$ . If a linear scaling rule is adopted, $\xi /M$ is constant. Then, we obtain $g\propto 1/(1-m)$ . Increasing $m$ results as $g$ rises may cause a drop in generalization performance. Adam^{[ 37 ]} combines the advantages of two optimization algorithms, namely, AdaGrad^{[ 38 ]} and RMSProp^{[ 39 ]}. Adam evaluates the first moment estimation of a gradient and the second moment estimation and then calculates the update step. This algorithm is a second-order optimization method, which adjusts the learning rate for each parameter by performing small updates for frequently used parameters and large updates for seldomly used parameters. In the vanilla Adam algorithm^{[ 37 ]}, ${\beta }_1$ and ${\beta }_2$ are set to control the influence of gradients and the square of gradients on the parameter update, respectively. They play a similar role to momentum in the SGD method with momentum. NAG^{[ 40 ]} is an improved method of momentum^{[ 36 ]} and a first-order optimizer. It updates with the gradient by “looking ahead” instead of using the current gradient. Moreover, it calculates the variance of gradients with respect to the last one. By utilizing these values, NAG updates the parameters in the training process.

SGD and its variants, known as stochastic gradient algorithms, can be regarded as first-order or second-order stochastic gradient algorithms, as shown in Algorithms 1 and 2, respectively.

SGD^{[ 18 ]} is one of the simplest first-order algorithms, but it introduces noise into the gradient and blocks optimization in the training progress^{[ 29 ]}. SGD randomly selects one sample or a random sample set at one time for parameter updates so that updating the parameters does not create redundancy. When the data size is large, SGD can effectively accelerate the training process.

Smith and Le^{[ 30 ]} stated that SGD should be interpreted as integrating stochastic equations. They also presented the scale of random fluctuations in the SGD dynamics as

(1) $$g=\xi \left(\frac{N}{M}-1\right)$$

where $\xi$ is the learning rate, $N$ is the size of the training set, and $M$ is the batch size. If we reduce the learning rate $\xi$ , the noise scale $g$ drops, thus leading to an improved training performance. If we keep the learning rate $\xi$ constant, then we could also increase batch size $M$ to reduce the negative impact of noise scale. By contrast, a small batch size increases the noise scale and adversely affects the training performance. According to Ref. [ 31 ], calculating the mean and variance values over a batch makes the loss calculated for a particular example dependent on other examples in the same batch. Therefore, if the batch size is large, a high dependency between samples in the batch limits the training performance. This fluctuation scale $g$ can also be considered as a noise factor. When $M\ll N$ , applying a linear scaling rule^{[ 32 ]} maintains the mean SGD weight update constant per training sample. A specific description about the linear scaling rule^{[ 32 ]} is shown in Section 2.2.

SGD has several variants^{[ 33 , 34 , 35 ]}. A common one is SGD with momentum^{[ 36 ]}; specifically, Smith and Le^{[ 30 ]} extended the traditional SGD to include momentum, and found that the noise factor $g$ changes into

(2) $$g={\displaystyle \frac{\xi }{1-m}}\left({\displaystyle \frac{N}{M}}-1\right)\approx {\displaystyle \frac{\xi N}{M(1-m)}}$$

where $m$ is the momentum. Equation ( 2 ) degenerates into Eq. ( 1 ) when $m=0$ . If a linear scaling rule is adopted, $\xi /M$ is constant. Then, we obtain $g\propto 1/(1-m)$ . Increasing $m$ results as $g$ rises may cause a drop in generalization performance. Adam^{[ 37 ]} combines the advantages of two optimization algorithms, namely, AdaGrad^{[ 38 ]} and RMSProp^{[ 39 ]}. Adam evaluates the first moment estimation of a gradient and the second moment estimation and then calculates the update step. This algorithm is a second-order optimization method, which adjusts the learning rate for each parameter by performing small updates for frequently used parameters and large updates for seldomly used parameters. In the vanilla Adam algorithm^{[ 37 ]}, ${\beta }_1$ and ${\beta }_2$ are set to control the influence of gradients and the square of gradients on the parameter update, respectively. They play a similar role to momentum in the SGD method with momentum. NAG^{[ 40 ]} is an improved method of momentum^{[ 36 ]} and a first-order optimizer. It updates with the gradient by “looking ahead” instead of using the current gradient. Moreover, it calculates the variance of gradients with respect to the last one. By utilizing these values, NAG updates the parameters in the training process.

SGD and its variants, known as stochastic gradient algorithms, can be regarded as first-order or second-order stochastic gradient algorithms, as shown in Algorithms 1 and 2, respectively.

In this section, we present distributed algorithms. As we use the parameter server architecture in our experiments, we show the training algorithms of workers and servers in Algorithms 3 and 4, respectively.

Algorithm 3 refers to the training algorithms on workers in the parameter server. In Algorithm 3, we input all the hyperparameters needed by the experiments and allocate $P$ GPUs as workers in the initialization phase. In the t-th iteration, the workers give the servers the signals of Pull trigger and Pull last saved parameter ( ${\theta }_{t-1}$ ). These parameters are fed into Algorithms 1 or 2 according to OP. Finally, workers send the Push trigger and Push ${\theta }_t$ to the servers. Algorithm 4 shows the training algorithms on servers in the parameter server architecture. It has the same initialization phase as Algorithm 3 but a different process. In the t-th iteration, when receiving the Pull trigger from the workers, the servers Push ${\theta }_{t-1}$ to the workers. After receiving the signal of Push trigger, the servers are responsible for Pull ${\theta }_t$ and save it locally.

Algorithms 1 and 2 are performed on the basis of the first- and second-order stochastic gradient. All of our experiments are based on these algorithms.

In this section, we present distributed algorithms. As we use the parameter server architecture in our experiments, we show the training algorithms of workers and servers in Algorithms 3 and 4, respectively.

Algorithm 3 refers to the training algorithms on workers in the parameter server. In Algorithm 3, we input all the hyperparameters needed by the experiments and allocate $P$ GPUs as workers in the initialization phase. In the t-th iteration, the workers give the servers the signals of Pull trigger and Pull last saved parameter ( ${\theta }_{t-1}$ ). These parameters are fed into Algorithms 1 or 2 according to OP. Finally, workers send the Push trigger and Push ${\theta }_t$ to the servers. Algorithm 4 shows the training algorithms on servers in the parameter server architecture. It has the same initialization phase as Algorithm 3 but a different process. In the t-th iteration, when receiving the Pull trigger from the workers, the servers Push ${\theta }_{t-1}$ to the workers. After receiving the signal of Push trigger, the servers are responsible for Pull ${\theta }_t$ and save it locally.

Algorithms 1 and 2 are performed on the basis of the first- and second-order stochastic gradient. All of our experiments are based on these algorithms.