Research contributions

• Developed an automated brain tumor classification system using DL and MRI images.
• Employed a pretrained VGG16 CNN architecture, achieving an accuracy of 99.24%, higher than comparable models.
• Utilized a large and diverse dataset with 17,136 brain MRI images, including glioma, meningioma, pituitary tumors, and normal brain scans.
• Implemented data augmentation and fine-tuning strategies to enhance classification performance.
• Created a user-friendly web application for easy image upload and real-time tumor prediction.
• Demonstrated the potential of the system to reduce human error and improve diagnostic accuracy in clinical settings.

Process overview

This research comprised three primary phases: training and validation, testing, and implementation of the brain tumor classification model. Fig 1 provides a detailed flowchart outlining the fundamental steps of the proposed methodology.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of the proposed methodology.

https://doi.org/10.1371/journal.pone.0322624.g001

First, all necessary libraries were identified and imported, as shown in Fig 1. These libraries included Keras, numpy, matplotlib, sklearn, and os. Following the import of libraries, the dataset was prepared, organizing training images, validation images, and testing images. The dataset was then split into training, validation, and testing sets in the ratio of 8:1:1. A pretrained VGG16 model was employed as the base model, with all its layers frozen to retain pre-learned features. The DL model was then compiled, trained, evaluated, and implemented. If the model’s accuracy did not exceed 90%, the number of epochs was adjusted, and the model was retrained to enhance performance. Once the model was fully trained, a web application was developed to predict and classify different types of brain tumors.

Development tools

The development tools used for model training and testing in this research project included the Python programming language. The laptop was equipped with an Intel Core i5-1135G7 processor with a base frequency of 2.40 GHz. The graphics were handled by Intel(R) Iris(R) Xe Graphics. The laptop had 8 GB of memory and 476 GB of storage. The operating system used was Windows 11. These hardware specifications were utilized to carry out the necessary computations and run the Python-based development environment for training and testing the models.

Dataset

The dataset used in this study for brain tumor classification was collected by combining multiple selected public MRI datasets, as shown in Table 2. Relevant datasets from various sources were sought to gather a comprehensive collection of brain tumor images.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Selected MRI datasets for training, validation, and testing.

https://doi.org/10.1371/journal.pone.0322624.t002

The data collection process involved utilizing three primary datasets. The first dataset was obtained from Figshare, specifically “Version 5” posted on April 3, 2017 [17]. This dataset comprised 3064 T1-weighted contrast-enhanced images featuring three different types of brain tumors. The second dataset, known as the SARTAJ dataset, contributed a portion of the data. The “no tumor” class data were originally sourced from the Kaggle dataset [18]. To ensure data quality, Sartaj Bhuvaji and his team conducted a manual examination of the images. They identified certain images in the “no tumor” class that exhibited visible tumors. These images were subsequently removed from the dataset to maintain accuracy. To further validate the dataset, they sought the expertise of a doctor who thoroughly examined all the MRIs and provided a certificate supporting the project’s objectives [18]. In addition, the SARTAJ dataset had a problem in that the “glioma tumor” class images were not categorized correctly [20]; thus, the “glioma tumor” class images on the Figshare site were used instead of the images from SARTAJ. The third dataset utilized in this study was the Br35H dataset, specifically the “Brain Tumor Detection 2020” collection. The images from this dataset were incorporated to augment their custom dataset, specifically for the “no tumor” class.

By amalgamating the data from these sources, a custom dataset was created, comprising brain tumor images categorized into four different classes: pituitary tumor, glioma tumor, meningioma tumor, and no tumor. The dataset contained a significant number of images, with the “pituitary tumor”, “glioma tumor”, “meningioma tumor”, and “no tumor” classes consisting of 1757, 1621, 1645, and 2000 images, respectively. The overall data collection process involved sourcing publicly available MRI datasets. This carefully curated custom dataset served as the foundation for training and evaluating the DL model utilized in the study. Figshare, SARTAJ, and Br35H datasets were combined in an approximate ratio of 4:3:2. The number of images for each tumor or no tumor category from each of the datasets is shown in Table 2. The combined dataset is then split based on class into an 8:1:1 ratio for training, validation, and testing processes.

Preprocessing

Preprocessing played a crucial role in preparing the dataset for training a DL model. It involved a series of steps that were designed to enhance the performance and generalization ability of the model. In this study, several preprocessing techniques were applied to the brain tumor dataset, aiming to optimize the training process and improve the model’s accuracy.

The grayscale conversion step ensured that the images were represented in a single channel, containing only intensity information. By converting the images to grayscale, color information was discarded, simplifying the data and reducing computational complexity. This simplification is suitable for brain MRI images, where the intensity variations hold crucial information for analysis and classification.

To increase the diversity of the training data and improve the model’s ability to generalize to unseen examples, data augmentation techniques were applied. These techniques introduce slight modifications to the original images, generating additional training samples with variations. In this study, the data augmentation techniques employed included rescaling, shearing, zooming, and horizontal flipping. Each augmentation parameter served a specific purpose:

1. 1.. Rescaling: The rescaling parameter normalized the pixel values by dividing them by 255. This normalization step ensures that the pixel values fall within the range of 0–1 and aids in the convergence of the training process. Normalizing the data to smaller values is a common practice in DL to enable the model to learn more effectively and speed up convergence.
2. 2.. Shear Range: Augmentation with a range of shear transformations introduced variations in the orientation of the images. This technique helps the model become more resilient to different angles and geometric distortions that may occur in brain MRI images, improving its ability to accurately recognize and classify structures.
3. 3.. Zoom Range: Applying a range of zoom values during augmentation simulated variations in the scale or magnification of the images. By training on images at different levels of magnification, the model learns to handle images with varying sizes and zoom levels, enhancing its ability to generalize and classify unseen images effectively.
4. 4.. Horizontal Flip: The horizontal flip augmentation horizontally mirrored the images. This transformation introduces variations that occur naturally, such as differences in orientation or anatomical structures from left to right. By training on both the original and flipped versions of the images, the model becomes more robust to these variations.

The augmentation techniques were specifically applied to the training set, while the validation and test sets remained unaltered. This separation ensures that the model’s performance is evaluated on the original, unaltered data, providing a reliable assessment of its generalization ability. The data augmentation and normalization steps were implemented using the Keras library, which provides convenient tools for image preprocessing. The “ImageDataGenerator” class in Keras allows on-the-fly data augmentation during the training process. It generates batches of augmented images based on the specified augmentation techniques and batch size. To efficiently generate augmented image batches during training, the “flow_from_directory” method from the Keras library was utilized. This method reads and augments images from the directory structure, reducing memory consumption and enabling the training of models with large datasets. By incorporating these augmented images into the training dataset, the model becomes more robust to variations in the test data, leading to improved generalization and performance.

Furthermore, concatenation was implemented in the preprocessing to concatenate the original and augmented images for the training data. A custom function called “concatenate_images” was implemented to iterate through each image in the training directory, apply augmentation transformations, and save the augmented images. For each original image, two augmented images were generated using the data generator. This augmentation strategy resulted in a threefold enhancement of the dataset size, increasing it from 5,712–17,136 samples. This expanded dataset not only provided a larger pool of training examples but also introduced greater diversity in tumor patterns and variations, thereby enhancing the model’s exposure to various scenarios and improving its ability to generalize to unseen data.

Besides, the number of images in the training, validation, and testing directories was counted. This count provides an overview of the dataset distribution, enabling researchers to assess the balance of the dataset and identify any potential data imbalances or biases that might affect the model’s training and evaluation. To prevent overfitting and to ensure reliable evaluation of the model, the dataset was split into training, validation, and testing sets. An 8:1:1 ratio was employed, where 80% of the data were allocated for training and the remaining 20% were evenly divided between the validation and testing sets. This partitioning helps assess the model’s performance on unseen data and provides a basis for adjusting hyperparameters and fine-tuning the model.

The preprocessing steps applied in this study aimed to optimize the brain tumor dataset for DL model training. The model’s performance and generalization ability were enhanced by organizing the dataset, splitting it into appropriate sets, applying data augmentation techniques, and normalizing the data. The augmentation diversified the training data, making the model more robust to variations in the test data. The normalization ensured consistent data representation, and the dataset organization facilitated efficient data management. These preprocessing steps collectively contributed to improved model accuracy and reliability in brain tumor detection and classification tasks. The generated dataset is explained in further detail in the Results section of this paper.

Deep learning architecture

The chosen DL architecture for the brain tumor classification model was a CNN with a pretrained VGG16 model as the base model. The VGG16 model is a popular CNN architecture that has demonstrated strong performance on various computer vision tasks.

The network structure consisted of the following:

1. Pretrained VGG16 Model: This model included an Input Layer that accepted input images with dimensions of (224, 224, 3). It comprised 13 convolutional layers organized into five blocks, with each block followed by a max-pooling layer for down-sampling. This architecture effectively learned hierarchical representations of the input images, extracting relevant features for classification tasks.
2. Flatten Layer: This layer flattened the output from the previous layer into a 1D vector, preparing it for the subsequent dense layers.
3. Dense Layer (256 neurons, ReLU activation): This fully connected layer with 256 neurons used the rectified linear unit (ReLU) activation function to introduce nonlinearity and learn complex patterns and features.
4. Dense Layer (4 neurons, softmax activation): This final fully connected layer had 4 neurons, representing the four different classes of brain tumors. It used the softmax activation function to produce class probabilities for multiclass classification.

The model was trained using the Adam (Adaptive Moment Estimation) optimization algorithm. In the initial training phase, a learning rate of 0.001 was used, and in the subsequent training phase, a learning rate of 0.0001 was used. The Adam optimizer helped optimize the model’s weights and biases, improving its ability to accurately classify brain tumors.

Modifications and fine-tuning

Transfer learning was applied using the pretrained VGG16 model. This strategy allows storing and leveraging the knowledge gained from the pretrained VGG16 model on a large-scale dataset and applying that knowledge to the brain tumor classification task [21]. All layers of the model, except the last two dense layers, were frozen or made nontrainable. This allowed the network to leverage the pretrained model’s learned features while adapting the last two layers specifically for the brain tumor classification task.

To perform a technique called fine-tuning [22], the model was trained twice. During the first stage of training, the nontrainable parameters from the VGG16 model remained fixed, while only the parameters in the last two dense layers were updated with a learning rate of 0.001. This fine-tuning strategy enabled the model to benefit from the knowledge gained from a large-scale dataset on which the VGG16 model was originally trained while specializing in its learning to the specific brain tumor classification problem.

After the initial training stage, the top few layers of the pretrained model were unfrozen to perform further fine-tuning. The first ten layers are set as nontrainable, while the remaining layers starting from the 11th layer are set as trainable. This selective unfreezing allows specific parts of the model to adapt to the new dataset. After unfreezing the layers, the entire model was retrained with a smaller learning rate of 0.0001. This allowed the unfrozen layers to adapt to the specific features of the new dataset while still retaining the knowledge from the pretrained layers. The lower learning rate would help in fine-tuning the newly unfrozen layers without causing significant changes to the already trained layers. By using this two-stage training approach with frozen and unfrozen layers, the fine-tuning strategy allows the model to learn more specialized features from the new dataset, improving its performance on the brain tumor classification task.

Network training and evaluation

The model was trained using the training set with a batch size of 32 and the Adam optimizer. The training process involved feeding the augmented images generated from the training set through the CNN model and updating the trainable parameters using backpropagation. Both the first and second stages underwent 50 epochs of training with 179 steps per epoch and 21 validation steps. The number of epochs was adjusted as needed to improve the model’s performance.

The model’s performance was evaluated using various metrics, including the confusion matrix, sensitivity, specificity, and per-class accuracy. A confusion matrix is a type of performance metric commonly used to assess the performance of a classification model for which the true values are known. It is a table with four different combinations of predicted and actual values: true-positive (TP), false-positive (FP), true-negative (TN), and false-negative (FN) [23]. These variables are fundamental in evaluating the performance of a classification model and understanding the accuracy of its predictions with respect to positive and negative cases.

TP represents the number of instances that are correctly identified as positive by the model. This means that the predicted positive outcomes align with the actual positive labels. TN, on the other hand, indicates the number of instances correctly identified as negative by the model, where the predicted negatives match the actual negative labels. FN represents the instances that are incorrectly classified as negative by the model, meaning that the predicted negatives do not match the actual positive labels. Finally, FP refers to the instances that are incorrectly classified as positive, where the predicted positives do not match the actual negative labels.

The validation set, which was separated from the training set, was used to monitor the model’s performance during training. Early stopping was employed to halt the training process if the model’s performance did not improve for a certain number of epochs. Once the training was completed, the model’s performance was tested using a separate testing set for each class. The trained model was applied to the testing set, and the performance metrics were calculated based on the model’s predictions. The following are the scoring metrics employed in this study. These metrics are denoted by Equations (1–5) as follows:

Sensitivity or recall was the measure of how much the correct prediction was from all the positive classes. The higher the sensitivity was, the better the performance of the trained model.

(1)

Precision was the measure of how much the actual positive from all classes the model had predicted as positive. The higher the precision was, the better the performance of the trained model.

(2)

Specificity was the measure of the ability of a classification model to correctly predict the negative cases. A high specificity would indicate that the model has a low rate of false positives and was effective at identifying negative cases.

(3)

The F1-score was also known as the F score or the F measure. The F1-score gave a measurement based on a balance between precision and recall. The higher the F1-score was, the better the performance of the trained model.

(4)

Accuracy was the measure of the correct prediction out of all classes. The higher the accuracy was, the better the performance of the trained model.

(5)

Generated dataset

The input image processing phase successfully transformed the dataset for training the model. The application of grayscale conversion and various augmentation techniques resulted in an expanded and diverse training dataset, which was essential for training a robust and accurate brain MRI image classification model. After the augmentation process, the training directory contained a total of 17,136 images belonging to four classes of brain MRI images. The validation and test directories also contained a substantial number of images, totaling 656 and 655, respectively, all belonging to the same four classes. Fig 2 illustrates sample MRI brain images before and after augmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Sample images before and after augmentation.

https://doi.org/10.1371/journal.pone.0322624.g002

The input image processing techniques employed in this study successfully increased the variance and diversity of the training dataset. By incorporating grayscale conversion and applying augmentation parameters, the model was equipped to handle different variations, orientations, and scales commonly found in brain MRI images. These preprocessing steps were essential for training a model that can generalize well to unseen data and accurately classify different types of brain MRI images.

Training and validation results

The constructed network was compiled and trained using a total of 3,963 glioma MRI images, 4,017 meningioma MRI images, 4,371 pituitary MRI images, and 4,785 normal MRI images. The model’s performance was evaluated by feeding the validation data into the network and calculating the accuracy and loss. For validation, 150 glioma MRI images, 153 meningioma MRI images, 150 pituitary MRI images, and 203 normal MRI images were used. Fig 3 illustrates the accuracy and loss plots against the number of epochs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Accuracy and loss plots of the model.

https://doi.org/10.1371/journal.pone.0322624.g003

The observed behavior, where the accuracy of the model increased as the number of epochs increased while the loss decreased, indicated a convergence of the model during the training process. Convergence referred to the point at which the model had learned the underlying patterns and relationships present in the training data, leading to improved performance.

As the model underwent training over multiple epochs, it progressively adjusted its parameters to minimize the loss function. The loss function quantified the discrepancy between the model’s predicted outputs and the actual labels. By iteratively updating the parameters based on the computed gradients, the model aimed to find the optimal set of parameters that minimize prediction errors. The increase in accuracy signified that the model’s predictions align more closely with the true labels as the training progresses. The model became better at correctly classifying or predicting the target variable, reflecting an improved understanding of the underlying patterns in the data.

Simultaneously, the decrease in loss indicated that the model was reducing its prediction errors. As the model updated its parameters to minimize the loss function, it gradually aligned its predictions with the actual labels, resulting in a lower overall loss. This decrease in loss demonstrated that the model was converging toward a state where its predictions were more accurate. However, it is important to consider the potential for overfitting during the convergence process. Overfitting could occur when the model becomes too specific to the training data and fails to generalize well to unseen data [24]. To mitigate overfitting, it was common to monitor the performance of the model on a separate validation set. If the accuracy on the validation set started to decrease or plateau while the training accuracy continued to improve, it may indicate that the model was overfitting.The observed convergence of the model, with increasing accuracy and decreasing loss over the course of training, suggested that the model was learning and improving its predictions. Careful monitoring and evaluation of the model’s performance on both training and validation data helped ensure that the convergence was indicative of generalization and not overfitting.

Network performance

The trained model was assessed using a separate test dataset consisting of 150 glioma MRI images, 153 meningioma MRI images, 150 pituitary MRI images, and 202 normal MRI images. The “classification_report” function was employed to compute precision, recall, and F1-score for each class, as shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Precision, recall, and F1-score for each class.

https://doi.org/10.1371/journal.pone.0322624.t003

Based on the data in Table 3, the model demonstrated high performance in classifying different types of brain tumors. For glioma tumors, the precision was 0.99, the recall was 1.00, and the F1-score was 1.00. This indicates that the model was highly effective in identifying glioma tumors, with very few false positives, and correctly identified all true glioma cases. The “meningioma tumor” class showed a precision of 0.99, a recall of 0.97, and an F1-score of 0.98. While the precision was high, indicating few false positives, the slightly lower recall suggests that a small number of true meningioma cases were not identified by the model. However, the overall F1-score of 0.98 reflects a strong balance between precision and recall. For the “no tumor” class, the precision, recall, and F1-score were all 1.00. This perfect score indicates that the model flawlessly identified the absence of tumors without any false positives or false negatives. The “pituitary tumor” class had a precision of 0.98, a recall of 0.99, and an F1-score of 0.99. This demonstrates the model’s robustness in detecting pituitary tumors with minimal false positives and a high rate of correctly identified true cases. Overall, the analysis of precision, recall, and F1-scores across different classes shows that the model performs exceptionally well in distinguishing between various types of brain tumors and the absence of tumors. The high scores across these metrics underscore the model’s reliability and effectiveness in medical imaging classification tasks.

Moreover, a confusion matrix was computed to provide a detailed breakdown of the model’s classification performance across different brain tumor types. The confusion matrix was computed using the “confusion_matrix” function, and a heatmap of the confusion matrix was plotted, as shown in Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Confusion matrix of the brain tumor classification.

https://doi.org/10.1371/journal.pone.0322624.g004

Analyzing the confusion matrix reveals that the model demonstrates strong accuracy across all categories, with minor misclassifications primarily between meningioma tumors and pituitary tumors. Specifically, for glioma tumors, out of 150 cases, the model accurately predicted all 150, achieving a perfect classification accuracy of 100%. In the case of meningioma tumors, the model correctly identified 149 out of 153 cases, resulting in an accuracy of approximately 97.39%. However, there were some minor misclassifications: one case was misclassified as glioma, and three cases as pituitary tumors. These results suggest that while the model performs well, it occasionally struggles to distinguish between certain characteristics of meningioma and other tumor types, leading to a few misclassification errors.

For cases with no tumor, the model’s performance was exceptional, achieving a perfect classification accuracy of 100%. All 202 instances of no tumor were correctly identified, indicating the model’s proficiency in recognizing normal brain scans. In the classification of pituitary tumors, the model accurately predicted 149 out of 150 cases, resulting in an accuracy of approximately 99.33%. There was one misclassification where a pituitary tumor was incorrectly identified as a meningioma tumor. Overall, the confusion matrix highlights the model’s high accuracy and reliability in classifying brain tumor types.

Furthermore, the sensitivity, specificity, and accuracy for each class were also calculated and tabulated in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Sensitivity, specificity, and accuracy for each class.

https://doi.org/10.1371/journal.pone.0322624.t004

Based on Table 4, it is observed that the system exhibited remarkable performance, with sensitivities ranging from 97.39% to 100.00%, specificities from 99.34% to 100.00%, and accuracies from 97.39% to 100.00%. Specifically, the sensitivity for glioma tumors was 100%, and the specificity was 99.80%, indicating the model’s excellent ability to correctly identify glioma tumors without falsely labeling other tumors as glioma. For meningioma tumors, the sensitivity was 97.39% and the specificity was 99.80%, reflecting the model’s strong performance in identifying meningioma tumors while minimizing false positives. For no tumor cases, both sensitivity and specificity were 100%, showcasing the model’s flawless detection of the absence of tumors. For pituitary tumors, the model achieved a sensitivity of 99.33% and a specificity of 99.41%, demonstrating its robustness in detecting this specific tumor type.

As a result, the overall sensitivity of the model was 99.18%, and the overall specificity was 99.75%, highlighting the model’s high reliability in distinguishing between different classes. The per-class accuracy rates were 100% for glioma tumors, 97.39% for meningioma tumors, 100% for no tumor, and 99.33% for pituitary tumors. The model achieved an overall classification accuracy of 99.24%, indicating excellent performance on the test dataset.

In addition, the brain tumor classification system was also manually tested, verifying its accuracy. In order to predict and classify the brain tumor MRI images into four target classes of brain conditions (normal, glioma, meningioma, and pituitary), MRI images were randomly chosen and loaded into the system. For example, when a random brain MRI image was chosen and loaded from the pituitary folder, the word “pituitary” was printed out together with the image loaded, which indicated that correct outputs were obtained. The process of manual testing was repeated by loading MRI images from different classes. As a result, the designed system had correct predictions on randomly chosen MRI images.

Web application

A web application was developed using the Dash framework to create a user-friendly GUI that allows users to upload MRI brain images and obtain predictions regarding the presence and type of tumor. The GUI was designed using HTML and Dash components, incorporating features such as an image upload button, an image display area, and a section to present prediction results and interesting facts about the predicted tumor type.

The GUI interacted with the classification model by processing the uploaded image, passing it through the model, and displaying the results. It enhanced the user experience by guiding users through the image upload process and providing informative details about the predicted tumor type based on the model’s prediction. As a result, the web application successfully performed accurate predictions of tumor types on randomly selected and uploaded MRI images. Fig 5 displays the output when a glioma MRI image was selected and uploaded, showcasing the GUI’s functionality and the corresponding prediction.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Web application output for uploaded glioma MRI image.

https://doi.org/10.1371/journal.pone.0322624.g005

Overall, the proposed approach demonstrated effective brain tumor classification performance, as evidenced by the statistical analyses, performance metrics, and visualizations presented.