Batch Normalization

Understanding Batch Normalization with Codes Explained

What is Batch Normalization?

Normalization means normalizing the data dimensions so that they are of approximately the same scale. Batch normalization is a technique for training very deep neural networks that normalizes the contributions to a layer for every mini-batch. Batch Norm is an essential part of the most of deep learning implementation.

Batch Normalization is normalizing the hidden units activation values so that the distribution of these activations remains same during training. Batch Normalization has the impact of settling the learning process and drastically decreasing the number of training epochs required to train deep neural networks.

We will assume that X is of size [N x D]. N is the number of data and D is their dimensionality. To get variance, we need to subtract the mean across every individual feature in the data.

Gamma and Beta are learned along with the other parameters of the network. There would be update rule for gamma and beta and this update rule would depend upon the derivative of the loss function with respect to gamma and beta.

Advantages of Batch Normalization

Here are some benefits of Batch Normalization:

• The model is less delicate to hyperparameter tuning. That is, though bigger learning rates prompted non-valuable models already, bigger LRs are satisfactory at this point.
• Shrinks internal covariant shift.
• Diminishes the reliance of gradients on the scale of the parameters or their underlying values.
• Weight initialization is a smidgen less significant at this point.
• Dropout can be evacuated for regularization.

Batch Normalization Layer

Batch Normalization normalizes batch data (output of Affine or Conv) before activation. Batch Normalization Layer is consist of forward pass and backward pass. The codes are from the book ‘Deep Learning From Scratch’, published in September 2019.

Initialization of Batch Normalization Layer

Temporary variables in forward() and backward() defined in the initialization part.

class BatchNormalization:

def __init__(self, gamma, beta, momentum=0.9, running_mean=None, running_var=None): 
        self.gamma = gamma # scale after normalization 
        self.beta = beta # shift after normalization 
        self.momentum = momentum # smoothing factor of EMA, momentum * newest value 
         
				# temporary variables in forward() 
				self.input_shape = None # Conv:4dim, Affine:2dim

# mean and variance for test session  
        self.running_mean = running_mean  
        self.running_var = running_var   
         
        # temporary variables in backward() 
        self.batch_size = None 
        self.xc = None # centered batch data 
        self.std = None 
        self.dgamma = None 
        self.dbeta = None

Forward Pass of Batch Normalization Layer

Forward Pass has two parts. The second one(__forward) is the actual forward process of batch normalization.

def forward(self, x, train_flg=True): #train_flg=False when testing  
        self.input_shape = x.shape # hold original shape here 
        if x.ndim != 2: # If the previous layer is not Affine(ndim=2) 
            N, C, H, W = x.shape 
            x = x.reshape(N, -1)  
        out = self.__forward(x, train_flg) 
        return out.reshape(*self.input_shape) # recover the shape of x after normalization

def __forward(self, x, train_flg): 
        if self.running_mean is None: 
            N, D = x.shape 
						# In the first iteration,setting moving average and var as 0. 
            self.running_mean = np.zeros(D) 
            self.running_var = np.zeros(D) 
                         
        if train_flg: # When training 
            mu = x.mean(axis=0) # find mean for each column 
            xc = x - mu # mean subtraction  
            var = np.mean(xc**2, axis=0) 
            std = np.sqrt(var + 10e-7) # + 10e-7 protects the vale from 0 
            xn = xc / std #normalize 
             
            self.batch_size = x.shape[0] 
            self.xc = xc 
            self.xn = xn 
            self.std = std 
            self.running_mean = self.momentum * self.running_mean + (1-self.momentum) * mu 
            self.running_var = self.momentum * self.running_var + (1-self.momentum) * var             
        else: # When testing 
            xc = x - self.running_mean 
            xn = xc / ((np.sqrt(self.running_var + 10e-7))) 
             
        out = self.gamma * xn + self.beta # gamma and beta are defined after training 
        return out

Backward Pass of Batch Normalization Layer

This process is to find dgamma and dbeta, which is used to update beta and gamma.

def backward(self, dout): 
        if dout.ndim != 2: 
            N, C, H, W = dout.shape 
            dout = dout.reshape(N, -1)

dx = self.__backward(dout)

dx = dx.reshape(*self.input_shape) 
        return dx

def __backward(self, dout): 
        dbeta = dout.sum(axis=0) 
        dgamma = np.sum(self.xn * dout, axis=0) # Hadamard product 
        dxn = self.gamma * dout 
        dxc = dxn / self.std  
        dstd = -np.sum((dxn * self.xc) / (self.std * self.std), axis=0) 
        dvar = 0.5 * dstd / self.std 
        dxc += (2.0 / self.batch_size) * self.xc * dvar 
        dmu = np.sum(dxc, axis=0) 
        dx = dxc - dmu / self.batch_size 
         
        self.dgamma = dgamma 
        self.dbeta = dbeta 
         
        return dx

Batch Norm is a very useful layer. If you are interested in Deep Learning, you will for sure have to get familiar with this method. I hope this post gives you a good understanding of how Batch Norm works.