Deep convolutional neural network architecture for facial emotion recognition

Resizing and standardization

Each individual model is capable of adopting a unique target size for its input tensor. The size 48×48 is chosen to enhance the versatility and obtain a substantial amount of information. Additionally, the pixels were normalized by dividing the pixel values by 255 (Gal & Rubinfeld, 2018). By adjusting each pixel’s range to span from 0 to 1 the model’s convergence rate is accelerated. This strategic normalization step aligns with the objective of optimizing the training process.

Colour space conversion

The Fer2013Plus dataset contains Y’UV images. However, when they are loaded into the model, it is loaded as a three channel tensor (red, green, blue). These tensors are then converted into grayscale as it helps to better form relations between the images (Chavolla et al., 2018). It reduces complexity, reduces dimensionality(the images only have a single color channel), reduces noise, and increases computational efficiency as they take up less space in the memory.

Label encoding

The labels of the dataset have been loaded into the model as a one-hot encoded tensor. Since, one-hot encoding doesn’t assume any magnitude, which basic integer values do, the model doesn’t have any assumptions on the values of ones and zeroes. Additionally, under normal encoding circumstances the model has a higher probability of assuming a relation between the encoded integer values, which is mitigated in one-hot encoding.

Data augmentation

To prevent overlearning, increase variation and for better generalization, the training images have been picked at random and have been flipped horizontally. It was determined that this level of augmentation is optimal and further augmentation leads to loss in semantics which can potentially cause unrealistic samples, resulting in the loss of original information.

Input layer

Input layer is used to define the tensor shape the model takes in as input. This layer processes the inputs which are 2D arrays of pixel values. It converts the arrays into 1D tensors, by multiplying the input shapes to get single values that are compatible with all the other convolutional layers in the model.

Convolutional layer

Convolutional layers are the building blocks of a CNN model. They extract features and distinguish between images. They have kernels with a specified size that slide with a predefined stride value over the image pixel and obtain a dot product upon multiplication (Albawi, Mohammed & Al-Zawi, 2017). This produces a feature map which is unique to the filter. As an input to the next layer, the previous layers feature maps are added using a bias to get the input for the next convolution neuron. Initializing the weights of a neural network is crucial because it can influence the convergence speed, optimization landscape, and overall generalization of the model. Poor initialization can lead to slow convergence, vanishing gradients, and hinders the network’s ability to learn meaningful features. The kernel initializer is a parameter in neural network layers that determines how the weights of the layer should be initialized at the beginning of training. It specifies a method for setting initial values for the weights in order to help the network converge more effectively and potentially improve its performance (Xu & Wang, 2022). A kernel initializer “He Normal”(used when exponential linear unit activation functions are used) is used in this model.

Activation function

In the convolutional layers, the ELU activation function is used. It doesn’t have the “dying relu problem” (Lu, 2020) which means the gradients for the remaining neurons become very small hindering further learning. To solve this problem ELU allows a small negative slope for negative inputs. The ELU activation function can be expressed: (1)${\displaystyle f\left(x\right)=\left\{x\leftrightarrow x>0a\left(e^x-1\right)\text{otherwise}\right.}$

Equation (1) represents a piecewise function that operates differently for positive and non-positive values of x. When x is greater than 0, f(x) returns the input x itself. For non-positive x values, it computes a value using the formula a(e^x − 1), where ’a’ is a constant and ’e’ is the base of the natural logarithm.

At the output layer of the model the Softmax activation function (Nwankpa et al., 2018) is used since it is a multiclass classification. (2)${\displaystyle s\left(x_i\right)=\frac{e^{x_i}}{\sum _{j=1}^n{e}^{x_j}}}$

Equation (2) defines a softmax function s(x_i) applied to a vector ’x’ with ’n’ elements. It calculates the exponential of each element x_j in the vector, sums up all the exponential values, and then divides each individual exponential value by the sum to normalize the vector elements into probabilities.

It is designed to convert raw scores, or logits into probability distributions over multiple classes. During training the softmax function’s output probabilities are compared to the label’s using the loss function (categorical crossentropy; Bessel & Bradley, 1818) to update the model’s parameters and improve its predictions.

Normalization layer

A normalization layer is placed after a convolutional layer. During the forward pass in training, this layer takes in the feature map (output) of the previous layer as its input and gives the normalized version of it as an output for the next layer. It normalizes the data by calculating the mean and standard deviation of the feature map (mini-batch) of the previous layer and it subtracts the input by the mean and divides the difference by the standard deviation. (3)${\displaystyle {\mu }_i=\frac{1}{K}\sum _{k=1}^K{\left(x_{i,k}\right)}^2}$ (4)${\displaystyle {\sigma }_i^2=\frac{1}{K}\sum _{k=1}^K{\left(x_{i,k}-{\mu }_i\right)}^2}$ (5)${\displaystyle x_i^{\stackrel{ˆ}{}}=\frac{x_i-{\mu }_i}{\sqrt{{\sigma }_i^2+ϵ}}}$ (6)${\displaystyle y_i=\gamma \cdot x_i^{\stackrel{ˆ}{}}+\beta .}$

Equation (3) calculates the mean μ_i of values of a vector x_i,k across a set of K elements. It sums each element’s value in x_i,k for all K instances and then divides the sum by K to obtain the average value of the vector.

Equation (4) measures the variance ${\sigma }_i^2$ of a vector x_i across a batch by calculating the average squared deviation of each value from the batch mean μ_i. It sums the squared differences for all K instances and divides by K, indicating the overall variability of the data around the mean.

Equation (5) calculates the normalized value x_i of an input x_i,k by subtracting the batch mean and dividing by the batch standard deviation $\sqrt{{\sigma }_i^2+ϵ}$, ensuring the input has a mean of 0 and a standard deviation of 1.

Equation (6) transforms the normalized value of $x_i^{\stackrel{ˆ}{}}$ by scaling with the parameter and shifting with the parameter β, resulting in the final output y_i. This step provides flexibility to the model, allowing it to adjust the normalized data during training.

This implies that at the output there is a zero mean and unit deviation, this results in lesser sensitivity to the initialized kernels weights. Crucially, it reduces the variance to zero and hence eliminates internal covariate shift in the model, which leads to lesser chances of vanishing gradients. It gives a regularized data which adds noise into the model which may help generalization, but lengthens training time. Overall, this is necessary due its advantage of eliminating internal covariate shift (Ioffe & Szegedy, 2015).

Pooling layer

Pooling layers are generally placed before a convolutional layer in the model. It systematically reduces the dimensionality of the feature map of the previous layer using the hyperparameters specified like stride and pooling window. By doing so, it helps the model retain only the prominent features by selecting the maximum value which has the most significant activation (Nirthika et al., 2022). This gives an added benefit that unnecessary noise is not retained helping the model train on the most prominent features, this in turn increases generalization as it learns the model can now identify the features in different positions in the image. This can also help increase the depth of the model as it reduces the number of parameters to be trained for the layers after it, making it possible to add more convolutional layers with greater neuron count. As the model is able to generalize more and it can identify prominent features at differing positions and orientations, it helps prevent overfitting on the training data.

Dropout layer

The dropout layer is a regularization technique that temporarily deactivates a randomly chosen fraction of neurons when training to prevent overfitting. It introduces a form of model averaging and a certain amount of noise in the network which encourages the model to learn more robust features. This layer also allows for training different subsets of neurons at different iterations of training, helping to improve generalization. While a dropout layer may slow down the convergence, it results in a more accurate model. The dropout layer has a hyperparameter known as dropout rate which is used to specify the number of neurons to be deactivated at the training iteration (Park & Kwak, 2017).

Table 1 gives a comprehensive description of the proposed DCNN model depicting the hierarchical arrangement of the layers along with the crucial information about the number of parameters the model is training on and the dimension of the images at each layer.

It is structured to acquire hierarchical patterns through sequences of convolutional and pooling layers, followed by fully connected layers and a class prediction output.

Essential components and design decisions include:

1. Input layer: The network starts with an input layer, shaping it to accept designated image data of shape (48,48,1).

2. Convolutional layers: The architecture features three sets of 2-dimensional convolutional layers, each with two successive convolutions. These layers extract features through the exponential linear unit (ELU) activation function, mitigating gradient issues.

3. The first set (conv2d_1 and conv2d_2) employs 64 filters of size (3, 3) with ’same’ padding. The second set (conv2d_3 and conv2d_4) enhances feature representation with 128 filters of size (3,3) with ‘same’ padding. The third set (conv2d_5 and conv2d_6) performs further feature extraction using 256 filters of size (3,3) with ‘same’ padding.

4. Batch normalisation layers: A batch normalization layer follows every convolutional layer and the final dense layer. Totaling to 7 batch normalization layers, they increase the model’s training stability and increase convergence.

5. Max pooling layers: Max-pooling layers of pool size (2,2) follow each convolutional set (maxpool2d_1, maxpool2d_2, maxpool2d_3), minimizing feature map dimensions while conserving key information.

6. Dropout: Dropout layers with a dropout rate of 0.3 (dropout_1, dropout_2, dropout_3) and 0.4 (dropout_4) are integrated to mitigate overfitting by randomly deactivating neuron outputs during training.

7. Flatten layer: The ultimate max-pooled feature maps are compressed into a 1D vector, fed into fully connected layers.

8. Dense layers: Two fully connected layers are employed, with the initial dense layer (dense_1) encompassing 128 neurons and ELU activation.

9. Output layer: The final layer (out_layer) is a dense layer with neurons equivalent to the classification task’s class count. It uses the softmax activation for class probabilities.

10. Model creation: The Keras functional API (Lu, 2020) amalgamates the entire structure, producing the “DCNN” model.

This architecture in Fig. 1 strives to balance feature extraction capacity and model complexity. Utilizing multiple convolutional layers, upping filter counts, and applying dropout and batch normalization, the model aims to acquire intricate hierarchical features from input images while averting overfitting. It’s a suitable DCNN for image classification tasks—categorizing input images into distinct classes.

Early stopping

Early Stopping is a model call back technique in machine learning that halts the training process when a predefined performance metric ceases to improve or starts deteriorating (Prechelt, 2000). This call back has three main hyperparameters, monitor which has been set to monitor the validation loss, delta checks for the minimum change in the monitored metric, patience value is set to wait for a specified amount of epochs (a cycle through the training data) before the model training is stopped. This call back is mainly used to prevent overfitting of the model by stopping the model training process.

Learning rate scheduler

A learning rate scheduler is a technique in machine learning that dynamically adjusts the learning rate during training. It helps optimize the training process by decreasing the learning rate over time, allowing the model to converge faster initially and then fine-tune as it approaches convergence. This aids in finding an optimal balance between rapid progress and stable refinement, resulting in better convergence and improved model performance (Kim et al., 2021). Specifically using Reduce LROnPlateau API from tensorflow keras callbacks (Lu, 2020) provides hyperparameters such as monitor, min_delta, mode, factor, patience, min_lr, etc. The hyperparameter mode is used to specify whether the learning rate should decrease when the quantity being monitored has stopped improving. Figure 2 shows the decrease in learning rate with epochs.