A. Computer-Assisted Leukocytes Segmentation and Classification Applications

Leukocytes segmentation is used in medical imaging to study the anatomical structure, diagnosis, treatment planning, and locate tumours. Ravikumar [12] presented a segmentation method using thresholding and ellipse-curve fitting. They used morphological properties for feature extraction, followed by classification of leukocytes using a naïve Bayes classifier. They utilized both geometrical and statistical features for classification purpose. The method is tested on two different datasets and the results were compared with other WBCs classification techniques. Rezatofighi et al. [8] presented framework to automatically segment five types of WBCs using a gran-Schmidt ortogonalization method in combination with snake algorithm. For this purpose, they extracted texture features from blood smear images. Two classifiers including artificial neural network (ANN) and SVM were tested and a comparative analysis was performed based on their classification performance.

Ko et al. [13] leukocyte from images using merging rules based on mean shift clustering and boundary remover rules. Gradient vector flow (GVF) [14] was used for segmenting nuclei and cytoplasm. For nuclei segmentation, probability density function was estimated from WBC nuclei. Mean shift clustering was used to segment the nuclei while cytoplasm was segmented using morphological opening of the green channel of RGB. Boundary edges and noise were removed by the GVF method to detect cytoplasm boundaries. Sholeh [15] used different algorithms for WBC enhancement and segmentation. Platelets and noise were removed from blood smear images using morphological operations. To smooth the cell shape, opening and closing operations were performed. Segmentation of WBC was done by obtaining the blue channel. Next, the five types of WBCs were segmented, counted, and classified. Then, the segmented image was labelled for leukocytes counting, followed by features extraction such as major axis length, minor axis length, and average axis length. Finally, ANN was used to automatically classify the five types of WBCs.

Prinyakupt and Pluempitiwiriyawej [3] have used thresholding and mathematical morphology for segmentation of cell nucleus. Morphology is a mathematical operation that applies addition/subtraction on blood smear images to separate WBCs, RBCs, and platelets. Threshold segmentation was done to partition the image into background and foreground and the optimal threshold value for the segmentation of WBCs was then selected. Geometrical features were extracted and SVM classifier was used for classification of leukocytes. In another study [16], self-dual multi-scale morphological toggle method was used to segment the nucleus and cytoplasm of WBCs. Watershed transform and level set methods were used to identify the cytoplasm regions. For identification of cytoplasm, there are two different techniques based on granulometric analysis and morphological transformations. This method extracted geometrical features such as area, solidity, eccentricity, perimeter area of convex part of the nucleus, ellipse, and its major axis length. Authors used KNN classifier for the classification of WBCs.

In [17], different segmentation techniques have been used such as global thresholding and FCM, calculating the blue channel of blood smear images for segmenting WBCs. Next, stain colour analysis is performed to get the feature vector of a particular region of interest. Finally, leukocytes are classified as infected or non-infected according to the set threshold value of the dataset. Bikhet et al. [17] enhanced the input image by removing noise and subtracted the foreground. Next, different features such as area of the cell, area of the cytoplasm, area of the nucleus and average colour of the cell were extracted for classification.

Ravikumar [12] conducted a comparative study of two techniques known as standard extreme learning machine (ELM) classification technique and fast relevance vector machine (FRVM) for segmentation. In comparison to FRVM, ELM classification technique performs well in local maxima. However, overall performance of ELM is not satisfactory [12]. To overcome the limitations of ELM and FRVM, authors modified the FRVM, which effectively detected leukocytes. Sedat Nazlibilek et al. [18] presented another approach which applies image enhancement on blood smear images prior to segmentation of WBCs using Otsu segmentation algorithm. Then, a set of morphological operations were used for removing unwanted objects and filling the holes. Next, the size of each segmented cell was calculated by using geometric features such as major axis length, minor axis length, and average axis length for each segmented object followed by classification of leukocytes into different classes using neural network. Piuri and Scotti [19] followed a similar approach as devised by Sedat Nazlibilek et al. [18]. However, they used feed-forward neural networks classifier for classification.

The aforementioned methods have two major issues: they are computationally expensive and their accuracy rate decrease drastically when tested on real datasets having noise and light reflectance. Keeping in view the above challenges, we have presented a resource-aware leukocyte classification and segmentation framework, offloading computationally complex tasks from smartphones to cloud server. However, analysis of blood sample images with mobile-assisted health monitoring systems is challenging due to limited resources of smart devices for data processing and transmission. To overcome this issue, we proposed an efficient and adaptive offloading framework for utilizing the available resources considering users’ preferences.

B. Mobile-Cloud Computing in Medical Applications

In recent years, mobile-cloud computing has become a valuable area for researchers [2], [20], mainly focusing to minimize the computational burden on smart devices by offloading computational tasks to cloud server, extending lifespan of smart devices. Guet et al. [21] have done extensive trace-driven assessments that presented efficient offloading extrapolation machines that can effectively reduce resource limitations of smartphone devices to minimize computational burden than other common schemes. Yang et al. [22] showed a resource-aware offloading scheme for a resource constrained smartphone devices. They focused the overall resources of the system including computational power, storage, and communication cost. Main purpose of the system was to save valuable resources up to greater extent. Miettinen and Nurminen [23] described energy consumption as a secondary source for smartphones. Hashem et al. [24] have given an extensive review of mobile-cloud computing in the field of mobile-cloud based healthcare. They aided various state-of-the-art methods dictating the constraints of the currently developed approaches. Utilization of cloud computing is mandatory, because most of the image processing techniques required high computational power, storage space, and network bandwidth. Liu et al. [25] presented an adaptive resource discovery based energy-efficient technique for mobile cloud computing which works independently with different network environments.

A. Mobile-Cloud Based Leukocytes Classification

Current mobiles have limited potential and are not applicable to process complex tasks. Thus, the leukocytes classification and segmentation from microscopic blood smear images is a computationally challenging task and such kind of tasks can be transferred to cloud server for processing. For offloading, a virtual machine (VM) based learning technique is used that can ensure the ability by transferring computationally heavy tasks partially or entirely from a mobile to more prevailing servers such as cloud server [27]. For leukocytes classification and segmentation, a multi-class classifier is trained in the cloud.

B. Offloading-Based Learning Automata

Learning automata in context of mobile-cloud offloading is an adaptive decision making model, which interacts with surrounding environments in discrete time instants. At each time instant, the proposed automata chooses a suitable threshold value for partitioning data and its processing tasks into local and cloud server. Data collected from environment works as an input for the learning automata and therefore it is known as response from the environment. In the underlying scenario, automata gets responses from surrounding environment, i.e., computational power, storage, bandwidth, and battery strength of a mobile phone, and then it shows special reaction (threshold calculation) in that specific situation. The process of threshold selection based on environment parameters is called reinforcement. The variation in the performance of the mobile-cloud adaptive loading framework is termed as “learning”, and, therefore, the learning system enhances the performance with respect to time in the process of achieving ultimate goal i.e., ideal threshold value selection. There are various internal and external environmental conditions that can increase or decrease the performance of the proposed learning technique. The proposed learning automata can be represented as LE = {A, B, C}, where A is input set, B is output set and C represent the environment as shown in Fig. 2.

FIGURE 2.

Illustrating the relationship between learning algorithm and environment.

Show All

C. Mobile Device Resource Monitoring

Prior to data offloading, it is required to analyse the resources consumption of mobile device. Learning automata-based offloading continuously tracks device resources such as computation power, storage, battery consumption, and internet bandwidth. In mobile-cloud based healthcare, smartphone devices send and receive the data/information over continuously varying network bandwidths. Variation in the internet bandwidth can affect overall performance of the system by disrupting the smooth transmission of patient data to cloud server. In the proposed framework, communication is the main element that needs to adjust according to the situation in order to save energy consumption of the computational method [28]. Optimal data offloading between mobile device and cloud sever is fully dependent on the bandwidth of the network. The device resource monitoring unit of the proposed framework evaluates the available communication resources and the resources required to offload data for further processing in the cloud.

D. Training of Multi-Class Classifier in the Cloud

Machine learning algorithms generally require large amounts of system resources such as computation power and storage. Machine learning techniques need large amount of training datasets for getting better accuracy [29]. However, large training dataset cannot be loaded into mobile memory in the training phase, due to limited storage space and limited computational resources. Therefore, it is required to offload the diagnostic data to cloud, reducing computational and storage burdens on a mobile device. In machine learning field, some classifiers offer more robust and accurate classification procedures due to their global properties such as ANN [30] and SVM [31], [32]. These techniques become more complex and expensive to process large datasets on smartphone [33]. In current technological era, there are various distributed machine learning methods which can be trained on multiple local computers or on single powerful cloud server. Once trained, these models can be used for testing purposes on both local as well as remote locations. Hence, we propose a resource-aware training model to train a classifier on ubiquitous cloud server.

Cloud-based classifier training mechanism with MapReduce technique is shown in Fig. 3. MapReduce is a programming model build up on the concept of functional programming called map and reduce function [34]. The MapReduce model is widely used to reduce classifier training time for processing heavy datasets, dealing both heterogeneity and large scale data. Similarly, a multi-class classifier can be trained on cloud server. Performance of this model is important, particularly in computational expensive tasks such as machine learning applications [29].

FIGURE 3.

Training a multi-class classifier on a cloud server.

Show All

E. Cloud Service

In mobile-cloud based leukocytes segmentation and classification from microscopic blood smear images, hematologists are provided detailed and broad evidence to correctly diagnosis different diseases. Cloud provides global access for storing, retrieving data, and performing data analysis in more easy and secure way.

For instance, in WBCs classification based on mobile-cloud, we only need to train a multi-class classifier in the cloud in order to classify WBCs, producing precise results for medical specialists and hematologists. There are number of cloud service providers such as Microsoft, Apple, Amazon, and Google [35]. For simulation purpose, we selected Google App Engine¹ (GAP), because it provides definite features to their clients in a cost effective manner. GAP is more suitable for Android based mobile devices as it uses secure plug layer to provide high-level security for encoding its different services [36]. It also facilitates users by providing an option to set up their privileges based on the interfaces assigned to them. Therefore, the security and privacy of the client’s data and application is ensured by Google [37]. Moreover, it provides free services to its user for prototyping.

A. Data Acquisition and Pre-Processing

The study consists of 1030 blood smear WBC samples which were collected from Hayatabad Medical Complex (HMC²) Peshawar, Pakistan. These blood smears were captured with Head Nikon DS-Fi2 ³ having high-definition color. The digital images were taken with approximately $100\times$ magnification factor. All the images were saved in JPG format of dimension $960\times1080$ pixels.

The size of the images was adjusted to $256\times256$ pixels during processing in MATLB as per the requirements of simulation. The detail of the dataset is given in Table 1. Blood smear enhancement includes noise removal, contrast adjustment, and image sharpening. In the underlying work, the acquired images are sharpened using a Gaussian un-sharp mask because sharp images can be easily segmented [38]. In the proposed method, the sharper images are converted from RGB to HSI colour space for applying the colour K-means clustering algorithm [15]. The main purpose of converting RGB colour space to HSI is to minimize the number of colours, helping in easy segmentation of WBCs using the K-means clustering algorithm [39].

TABLE 1 Image Dataset Obtained From HMC Hospital

B. Blood Cells Nuclei Segmentation

In image processing and machine learning applications, image segmentation presents division of areas having similar properties. Through image segmentation, a given image can be converted into a meaningful form for further analysis [10]. Image segmentation has numerous practical applications in different fields, especially in medical imagining such as studying the anatomical structures, diagnosis, treatment planning, localizing tumours, counting leukocytes, classifying WBCs, and other pathologies. Image segmentation refers to the partitions of an image into a set of disjointed and similar regions, which are meaningful to a particular application [40]. Thus, the segmentation process is based on global thresholding, mathematical morphology, FCM clustering, watershed, Otsu binarization, and colour contrast. Global thresholding is a better option to segment microscopic blood smear images. However, cytoplasm, nucleus, and background have their own unique grey levels. Thus, global thresholding can perform worse when the lighting level varies from one image to another image. Colour based segmentation of WBCs includes five different techniques to segment them from other cells of the image. The user can select one of these methods and check the precision and accuracy of different techniques to decide the correct algorithm for a particular application and disease. In order to select the suitable segmentation method for the proposed framework, we considered K-means cluster based segmentation, FCM, active contours, watershed method, OTSO and simple thresholding method. Through experiments, we found colour K-means to be the best candidate for the underlying tasks as it is simple and comparatively more accurate for colour based segmentation as shown in Fig. 5.

FIGURE 5.

Applying K-mean clustering algorithm on blood smear image.

Show All

C. Features Extraction and Feature Reduction Using PCA

After the segmentation process, next step is to extract the features, which is critical step towards classification accuracy. Features are descriptors of an image, representing their natural similarities. These features along with their labels are then used by the classifier for matching different images and classifying them into certain classes.

In the proposed work, we have extracted three different sets of features including:

1. Geometric features of an image, i.e., area, circularity, perimeter, and centroid.

2. Statistical properties such as arithmetic mean, variance, standard deviation, regression, correlation, skewness, kurtosis, root mean square, and histogram.

3. Textural features such as correlation, energy, entropy, contrast, and inverse difference movement.

These features are extracted from frequency domain, which is comparatively more suitable to extract strong features for leukocytes classification than spatial domain. To this end, DWT is applied on each dimension of the 2D blood images, producing four sub-bands LL, LH, HH, and HL. The process is repeated two times for LL band. The proposed method uses level-3 decomposition for feature extraction due to statistically rich features of LL band at level-3 of DWT. After features extraction, it is necessary to reduce its size to minimize the computation time and storage requirement. To achieve this goal, a method known as principle component analysis (PCA) is used. PCA is simple and more suitable for our framework compared to other dimensionality reduction methods such as auto encoder, which is its main motivational reason of its usage.

D. Multi-Class SVM-Driven Leukocytes Classification

The next step after feature extraction is selection of the best classifier, considering the input and its expected result. SVM is a well-known supervised learning technique in machine learning, which is based on statistical learning theory. This technique is robust and accurate even if we have a small amount of training data. SVM is a binary classifier but it can be extended further for multi-class classification. In the proposed work, we have used an ensemble multi-class SVM (EMC-SVM) for classification of leukocytes into its five classes. This is due to the diversity of blood smear images for which training a single classifier is impractical because of limited performance [11], [13]. It has been experimentally proven that ensemble SVM performs well compared to traditional SVM [41]. Therefore, the proposed EMC-SVM was devised to classify the WBCs into five different classes. For training purposes, 70% of the whole data is utilized. The remaining 30% of the data is used for testing the accuracy of the proposed classifier. To test a new blood smear image, same procedure of pre-processing, segmentation, and feature extraction is performed. The extracted feature vector is then passed through EMC-SVM, which assigns a class label to the given test image among the available five classes.

A. Subjective Evaluation of the Proposed Segmentation Method

The subjective analyses are carried out to evaluate the performance of the proposed WBC boundary detection scheme by comparing with the boundary of WBCs marked by a group of medical specialists. Four haematologists were requested from HMC to join digital image processing laboratory for evaluation purposes. These haematologists having experience of more than 10 years in analysing haematological problems. The leukocytes were particularly segmented for diagnostic purpose (having a clear view of their nuclei structure and color). The proposed technique is compared with the ground truth. This algorithm can find and segment the five classes of leukocytes correctly. Evaluation of the proposed segmentation technique is based on three metrics including false positive rate (FPR), false negative rate (FNR) and F- measure which can be computed as follows:

View Source\begin{align} FPR=&\frac {FP}{TN+FP} \\ FNR=&\frac {FN}{TP+FN} \\ F-Measure=&2\ast \left ({ {\frac {FPR\ast FNR}{FPR\ast FNR}} }\right ) \end{align}

• True positive (TP) is the number of WBC pixels correctly identified.

• False positive (FP) is the number of non-WBC cells pixels that are marked as WBC pixels.

• True negative (TN) refers to the number of non-WBC pixels that are correctly marked.

• False negative (FN) is the number of WBC pixels incorrectly labelled as non-WBC

• F-Measure is the weighted harmonic mean of FPR and FNR that provides a more realistic performance scenario of any subjective evaluation method.

In our case, subjective evaluation of the proposed segmentation method in terms of FPR, FNR, and F-measure is given in Table 2. It can be seen that F-measure of the proposed technique for 1030 blood smear images with all types of WBCs in both nucleus and cell segmentations has above 82% value, indicating efficient segmentation results closer to manual segmentation.

TABLE 2 Comparative Analysis of Different Segmentation Methods

B. Comparative Analysis of the Proposed Segmentation Method

In this sub-section, we evaluated the performance of the K-mean with different state-of-the-art segmentation methods. In the assessment process, we applied the proposed technique individually to each of 1030 blood smear images and results are compared with state-of-the-art segmentation techniques. The proposed segmentation method achieved an average accuracy of 95.7% for nuclei and 91.3% for cytoplasm. Table 3 shows the experimental observations, where correctly detected leukocytes ratio is represented by A₁ as given in Eq. 4. $\text{A}_{1} = 1$ indicates that all the leukocytes are correctly detected in the microscopic blood smear images. In the evaluation setup, the ratio of the detected leukocytes to the total detected leukocytes is denoted by parameter A₂ as given in Eq. 5. $\text{A}_{2} = 1$ shows that no incorrect detection occurred. Thus, by analysing A₁ and A₂, we can evaluate the overall accuracy of the proposed technique. Overall evaluation shows that the accuracy of the proposed technique is much better than other methods. Table 3 shows the effect of colour adjustment on the segmentation performance. The performance of the proposed method is compared with three state-of-the-art methods as shown in Table 4. In case of eosinophil segmentation, accuracy of adaptive threshold-based method is 87.2 which is higher than accuracy of active contour-based method. However, average accuracy of adaptive threshold-based method is lower than active contour-based segmentation method. On other perspective, accuracy of the proposed segmentation method surpasses the accuracy of both active contour-based and adaptive threshold-based segmentation methods in all five categories. Average accuracy of the proposed segmentation method is 94.3 which is greater than the average accuracy of other two underlying comparison methods.

View Source\begin{align} A_{1}=&\frac {The~number~of~currectly~segmented~WBCs}{Totlatl~number~of~segmented~WBCs}X 100\%\notag \\ \\ A_{2}=&\frac {Totlatl~number~of~segmented~WBCs}{Total~number~of~WBCs~that~exists~in~all~images}X 100\notag \\ {}\end{align}

TABLE 3 Performance Evaluation with and Without Colour Adjustment in Percentage

TABLE 4 Performance Analysis of the Proposed Method, Adaptive Threshold and Active Contour-Based Segmentation Method for Each Type of Leukocyte

C. Qualitative Analysis Based on Visual Results

Comparative analysis of visual results of the proposed segmentation technique with two state-of-the-art methods including adaptive thresholding [9] and active contour technique [42] is presented in this section. For this purpose, three parameters were considered to check the visual quality of different leukocytes. The parameters are shape, size, color, and texture. The segmentation method must retain size and shape of the nuclei in the blood smear image. In terms of colors, leukocyte’s nuclei and cytoplasm luminance information must be well-preserved in blood smear image. Segmentation technique must have the ability to maintain textures from input blood smear images. Texture regions must not be shown clear by the segmentation technique because the haematologist differentiates some of the leukocytes classes through their texture. Sample qualitative evaluation results are shown in Fig. 6, where first column shows the original blood smear images, second column represents adaptive threshold, third column refers to active contour, and the last column illustrates the visual results for the proposed method.

FIGURE 6.

Comparative analysis of the proposed method for leukocytes segmentation with adaptive thresholding and active contour model.

Show All

D. Energy Consumption Estimation Model

Energy consumption of a system is defined as the total resource usage by the device from start to the completion of the required task. Energy consumed during a particular task depends on computational complexity of the underlying tasks [43]. In the proposed framework, energy consumption can be analysed in two different scenarios, i.e., energy consumption via local computation and offloading the task fully or partially to cloud sever.

We determined that cloud-assisted applications consume most of the energy in communication process. For smart devices, major factor affecting energy consumption includes energy consumption for process execution (energy consumption per-operation) [44]. In the communication perspective, energy consumption refers to energy required for data transmission (either full-offloading or partial offloading to cloud). Moreover, for smart devices, reliable network connection and transmission energy is required for offloading patient data for testing and receiving classification/decision results from cloud server.

Each of these energy consumption metrics $\textit{E}_{\mathrm {\mathbf {no{}off}}}$ and $\textit{E}_{\mathrm {\mathbf {off}}}$ can be calculated as follows:

View Source\begin{align} E_{\mathrm {no{}off{}}}=&\left ({ \frac {D_{out}\mathrm {}}{\mathrm {b}^{\mathrm {'}}\mathrm {send}}\ast \mathrm {e send} }\right )+ \left ({ e_{Mcomp}\ast I }\right ) \\ E_{\mathrm {off}}=&\left ({ \frac {D_{inp}\mathrm {}}{b_{send}}\ast \mathrm {e send} }\right )+ \left ({ \frac {D_{out}\mathrm {}}{b_{receive}}\ast e_{receive} }\right ) \end{align}

Details of each parameter is given in Table 5. Two scenarios are evaluated that whether $\textbf{E}_{\mathrm {\mathbf {total}}}$ , is positive or negative as given in Eq. 8. This helped in deciding about the execution of the program on mobile device or on cloud server, considering the minimal consumption of energy.

View Source\begin{equation} E_{\mathrm {total}} = E_{\mathrm {off}} - E_{\mathrm {no{}off}} \end{equation}

TABLE 5 Variables Used in the Equations

E. Energy and Computational Complexity Analysis

In this section, we analyze computational time and energy consumption of the proposed method to highlight its computation- and communication-saving feature. For this purpose, we incorporated the experimental practices of [37] and [38]. According to framework presented in [39], if the size of data transmission packet is $8\times8$ , then energy required for its transmission is $88.48~\mu \text{J}$ . In RGB image, each pixel consists of 24 bits (3 bytes). In gray scale image, each pixel consists of 8 bits (1 byte) while in binary image, each pixel consists of 1 bit. In the proposed framework, image size is adjusted to $256\times256$ pixels and offloaded in patches of size $8\times8$ to cloud server for further processing. Table 6 shows the approximate energy consumption both for smartphone and cloud server of mobile-cloud-based leucocytes segmentation and classification for three different computational scenarios. The full-offloading (transmission of complete blood smear image) approach consumes more energy than local and learning based adaptive-offloading (offloading image feature) approaches because of the high communication cost. Local processing approach minimizes the transmission cost, performing computationally complex operations i.e., pre-processing, segmentation, and classification at local device. However, the execution of such energy starving tasks on smartphone is not a practical approach. For such types of computationally heavy tasks, cloud servers provide a superior environment over smartphones, which is evident from the computation costs listed in Table 6.

TABLE 6 Approximate Energy Consumption (in Joules) for Both Smartphone and Cloud Server During our Mobile-Cloud Based Leukocytes Segmentation and Classification for Three Computational Scenarios: (1) Local Computation; (2) Entire Image-Offloading, and (3) Offloading Image Features

F. Performance Analysis of the Proposed Framework

In this section, several experiments were conducted to evaluate the performance of the proposed classification with other state-of-the-art schemes. The comparison is based on four metrics including accuracy, sensitivity, specificity, and precision as given below:

View Source\begin{align} Accuracy=&\frac {(TP+TN)}{TP+TN+FP+FN} \\ Sensitivity=&\frac {TP}{TP+FN} \\ Specificity=&\frac {TN}{TP+FN} \\ Precision=&\frac {TN}{TP+FN} \end{align}

FIGURE 7.

Comparative analysis of the proposed method based on accuracy, specificity, sensitivity, and precision for leukocytes classification with linear classifier.

Show All