Interpretable noninvasive diagnosis of tuberculous pleural effusion using LGBM and SHAP: development and clinical application of a machine learning model

Data collection protocol

Data was collected from January 1, 2021, to September 30, 2024, based on criteria such as TPE diagnosis and laboratory results (ADA, TB-Ab). The data was collected by trained medical staff to ensure accuracy and consistency. Standardized procedures, including specific steps for sample collection, processing, and analysis, were developed by the hospital’s Ethics Committee and Laboratory Quality Management Department. These procedures were strictly followed throughout the study to ensure the reliability of the collected data.

Data sources

Total participants: This study included 963, 424 patients diagnosed with TPE and 539 non-TPE control patients. It is a prospective, multicenter, observational study designed to develop and validate a machine-learning model based on routine laboratory data for noninvasive TPE diagnosis. Data collection spanned from January 2021 to September 2024 at two medical centers in China: Zhejiang Provincial People’s Hospital and Jiashan County First People’s Hospital.

Data collection: Data were extracted from the hospital (HIS) and laboratory (LIS) information systems at Zhejiang Provincial People’s Hospital and The First People’s Hospital of Jiashan. The study included 20 laboratory parameters, including protein, lymphocyte, tuberculosis antibody (TB Ab), erythrocyte sedimentation rate (ESR), adenosine deaminase (ADA), lactate dehydrogenase (LDH), nucleated cell count (NCC), eosinophil, sex, carcinoembryonic antigen (CEA), Cancer Antigen 15-3 (CA153), Cancer Antigen 12-5 (CA125), Cytokeratin 19 (CK19), alpha-fetoprotein (AFP), Cancer Antigen 199 (CA199), chloride, squamous cell carcinoma (SCC), glucose, age, and macrophage.

Data preprocessing: Missing data were handled using mean imputation. Categorical variables were encoded for processing, and features with >25% missing data were excluded. A Spearman correlation analysis removed highly correlated features (r > 0.6) to minimize redundancy and overfitting.

Feature selection: Lymphocytes and macrophages were excluded from the model due to their potential to cause overfitting, despite being clinically relevant. The remaining 18 laboratory parameters were selected based on their clinical importance and association with TPE, as demonstrated in previous studies. Parameters such as ADA and TB-Ab, which are well-established biomarkers for TPE and routinely measured in clinical practice, were included. This selection ensures both diagnostic accuracy and practical applicability.

Inclusion criteria: Patients aged ≥18 years with suspected pleural effusion and a confirmed diagnosis of tuberculous pleural effusion (TPE), based on pleural biopsy, positive Mycobacterium tuberculosis culture, or documented tuberculosis history, were included.

Exclusion criteria: Patients were excluded if they had incomplete clinical data, unclassified pleural effusion, no follow-up, or major health conditions such as active infections or unrelated malignancies. The non-TPE group included patients with other respiratory diseases, such as chronic obstructive pulmonary disorder (COPD), pneumonia, and non-tuberculous effusions (e.g., malignancy, heart failure).

Data application and processing

Data cohorts: Datasets are typically divided into training, testing, and validation sets, as models tend to perform best on their training data (James et al., 2013; Hastie, 2009; Molnar, 2020). The training cohort (Cohort 1, n = 610) and the testing cohort (Cohort 2, n = 153) were randomly assigned, using data collected between January 2021 and January 2024 for model development and internal validation. The external validation cohort (Cohort 3, n = 200) comprised data collected between February 2024 and September 2024 to assess the model’s generalizability. The training cohort consists of data from 2021 to 2024, while the validation cohort includes data from 2024. This time difference allows us to evaluate the model’s generalizability and performance on more recent data, ensuring its robustness across evolving patient demographics and clinical practices.

Data preprocessing: To ensure optimal data quality and model performance, preprocessing steps were implemented. Missing values were addressed using mean imputation for randomly distributed gaps in the data. Categorical variables, such as TB-Ab, were numerically encoded (0 for negative, 1 for weak positive, and 2 for positive) to facilitate processing. A Spearman correlation analysis was performed to eliminate features with high collinearity (correlation coefficient > 0.6), reducing redundancy and enhancing model generalizability. Despite their clinical significance, lymphocytes and macrophages were excluded to prevent overfitting. The missing data were handled using mean imputation, and variables with more than 25% missing data were excluded. While mean imputation was used, more advanced techniques, such as multiple imputation or model-based imputation, could be explored in future work, especially when the missing data are not random.

Model development: Ten machine learning models, including LGBM, gradient boosting decision trees (GBDT), and support vector machines (SVM), were developed using data from the training and testing cohorts. The LGBM model achieved the highest predictive accuracy. SHAP analysis enhanced interpretability, identifying 11 key features (e.g., ADA, TB-Ab, protein) crucial to the model’s performance.

External validation: The LGBM model was validated using Cohort 3, demonstrating superior diagnostic performance compared to conventional clinical methods. The model accurately identified patients with TPE, highlighting its potential as an effective diagnostic tool.

Feature selection

Clinical expert screening: Feature selection was grounded in clinical relevance and supported by existing literature. Based on prior studies and clinical experience, clinical experts initially selected laboratory parameters strongly associated with TPE, such as protein levels, ADA activity, and glucose levels. These parameters served as the foundation for model development.

Feature importance evaluation: The LGBM model assessed feature importance, ranking features by their contribution to the model’s predictive capabilities. Following this evaluation, 11 key features were retained, substantially enhancing the model’s accuracy.

Final feature set: The streamlined LGBM model incorporated these 11 key features, demonstrating robust predictive power for diagnosing TPE. This final model offers reliable support for clinical decision-making, improving diagnostic efficiency in practice.

Machine learning model construction

Model performance evaluation

The model’s performance was assessed using ROC curves and critical metrics, including specificity, sensitivity, negative predictive value (NPV), positive predictive value (PPV), F1 score, and accuracy. These metrics were calculated for internal (Cohort 2) and external (Cohort 3) validation cohorts to evaluate diagnostic accuracy comprehensively. The LGBM model achieved the highest AUC of 0.9454 during internal validation, indicating superior performance.

Statistical analysis

Data handling and feature selection: Data were initially extracted from electronic medical records, including 20 routine laboratory parameters such as pleural fluid cell classification, ADA, ESR, pleural fluid glucose, and protein levels. The feature selection process first employed Spearman correlation analysis to address multicollinearity. When the correlation coefficient between the two features exceeded 0.6, the feature with a weaker correlation to the outcome was removed. Following this, based on feature importance analysis and model performance, 18 key features were retained (e.g., pleural fluid cell classification, ADA), while lymphocyte and macrophage proportions were excluded to avoid overfitting. Features with a missing rate above 25% were discarded, and for those with minimal missing data, median imputation was applied to minimize bias.

All analyses were conducted using Python 3.6 and R 4.1.0. Categorical variables are reported as frequencies and percentages, and continuous variables as medians with interquartile ranges. Group comparisons used the chi-square or Fisher’s exact test for categorical variables and the Mann-Whitney U or Kruskal-Wallis H test for continuous variables. Model performance comparisons were carried out using the DeLong non-parametric method, and the significance of differences in AUC values was assessed using MedCalc software. A P-value < 0.05 was considered statistically significant.

Baseline analysis

This study evaluated a machine learning-based predictive diagnostic model for identifying TPE. The clinical characteristics and outcomes for the 963 patients, categorized as TPE and non-TPE, are summarized in Table 1. A total of 763 patients were randomly divided into training and internal validation sets in a 4:1 ratio. In comparison, an external cohort of 200 patients with pleural effusion was used for external validation (Table 2 compares clinical variables across these cohorts). The study design is detailed in Fig. 1.

The Z-value in non-parametric tests quantifies how much the distributions of the two groups differ from the null hypothesis. Its magnitude and direction indicate the significance and trend of the difference.

Model development and performance comparison

Data collected within the first 24 h of patient admission with pleural and peritoneal effusion were used to develop 10 ML models to predict tuberculous effusion. The LGBM model outperformed others with an AUC of 0.9557, followed closely by GBDT (AUC = 0.9545) and SVM (AUC = 0.9538). Table 3 summarizes the model performances, and Fig. 2A presents the ROC curves for models, all utilizing 18 features. During feature reduction, LGBM maintained superior predictive ability, as shown in Fig. 2B. Consequently, LGBM demonstrated the highest predictive accuracy among all models. The LGBM model significantly outperformed other machine learning models in this study, as summarized in Table 3. During internal validation, LGBM achieved the highest AUC of 0.9557, surpassing models like random forest (AUC = 0.9494) and GBDT (AUC = 0.9545), demonstrating superior diagnostic accuracy. Additionally, LGBM exhibited improved sensitivity and specificity, particularly when handling high-dimensional and imbalanced datasets, indicating greater robustness. The comparative performances of these models are visually depicted in Figs. 2A and 2B, further highlighting LGBM’s suitability for noninvasive clinical diagnostics.

LGBM model performance by feature set

Figure 2C and Table 4 detail the performance of the LGBM model across different feature sets. Specificity, sensitivity, NPV, PPV, F1 score, and accuracy were calculated at the optimal cutoff, determined by maximizing the Youden index. Table 4 details the LGBM model’s performance across different feature sets, showing how the number of features affects its diagnostic accuracy for TPE. Additionally, Table 5 provides the P-value calculations corresponding to the feature value changes and their impact on the LGBM algorithm’s performance.

Final model identification

The final LGBM model was constructed based on selected feature parameters (see Figs. 2C and 3). The 18-feature model showed improved performance over the 10-feature (ΔAUC = 0.0223, P = 0.4559), 3-feature (ΔAUC = 0.0654, P = 0.9013), and 12-feature models (ΔAUC = 0.0137, P = 0.4559). However, its performance was not statistically superior to the 11-feature model (ΔAUC = 0.0103, P = 0.2621). As shown in Table 4, the 11-feature model exhibited similar net benefit and threshold probability as the 18-feature model, indicating equivalent clinical utility. Consequently, the 11-feature LGBM model was selected for further analysis. This model included age, eosinophil, TB-Ab, ADA, glucose, chloride, LDH, protein, CA153, CK19, and SCC. The final LGBM model achieved an accuracy of 0.8682, AUC of 0.9454, F1 score of 0.8794, sensitivity of 0.9394, specificity of 0.7937, PPV of 0.9259, and NPV of 0.8267, as detailed in Table 4. These metrics demonstrate the model’s predictive solid performance in diagnosing TPE and underscore its utility as an interpretable predictive tool for clinical application.

In the final evaluation of the LGBM model, we analyzed false positive and false negative cases, finding that false positives were mainly caused by elevated ADA and TB-Ab levels, while false negatives were common in mild TPE patients with ADA levels below the diagnostic threshold. SHAP analysis further confirmed the role of these features in misclassification. Despite the overall good performance of the model, improvements are needed in the recognition of low-expression TPE cases. To address this, we plan to enhance sensitivity for low-expression TPE cases, introduce more TPE-related features, especially for early or asymptomatic cases, and use more complex models or ensemble methods to improve performance in complex cases. These optimization measures will enhance the model’s diagnostic ability in clinical settings. Additionally, to assess the stability and reliability of the model, we calculated the confidence intervals for the AUC. The results showed that in both internal and external validation, the AUC of the LGBM model was 0.9454 and 0.9262, respectively (95% confidence intervals: [0.9400–0.9500] and [0.9200–0.9300]), further validating the model’s stability and predictive ability. To validate the advantages of the LGBM model, we compared it with traditional diagnostic methods, such as clinical evaluation, imaging, and routine laboratory tests. The LGBM model demonstrated significantly higher diagnostic sensitivity in early or asymptomatic TPE cases compared to clinical evaluation; it provided more consistent and efficient diagnoses than imaging, especially in cases with unclear images; additionally, by integrating multiple laboratory indicators, LGBM offered a more comprehensive assessment of TPE diagnosis, improving accuracy. Overall, the LGBM model excels in enhancing diagnostic sensitivity and accuracy, and is highly adaptable, especially in resource-limited areas. The LGBM model has broad clinical applicability, providing efficient diagnostic support in resource-limited hospitals, particularly improving diagnostic sensitivity in early and mild TPE cases, and it can further enhance its performance in low-expression TPE cases through feature optimization and ensemble methods in the future.

External validation and performance evaluation of feature parameters

To ensure sample size adequacy and assess the impact of different sample sizes, 5-fold and 10-fold cross-validation were conducted, as presented in Figs. 4A and 4B. The final model yielded a mean AUC of 0.9262 in the 5-fold cross-validation and 0.9337 in the 10-fold cross-validation, confirming its stability and generalizability. Additionally, individual feature performance, including age, eosinophil, TB-Ab, ADA, glucose, chloride, LDH, protein, CA153, CK19, and SCC, was evaluated compared to the 11-feature model. As demonstrated in Fig. 5, the individual performance of features such as age (ΔAUC = 0.2885), eosinophil (ΔAUC = 0.2599), TB-Ab (ΔAUC = 0.2273), ADA (ΔAUC = 0.0642), glucose (ΔAUC = 0.3094), and others consistently underperformed relative to the full 11-feature model during internal validation.

Model interpretability

This study evaluated the performance of multiple machine learning models, including random forest, gradient boosting decision trees (GBDT), and support vector machines (SVM). LGBM demonstrated clear advantages in managing high-dimensional data (Ke et al., 2017), addressing class imbalances, and minimizing training time. By employing gradient-based one-sided sampling (GOSS) and exclusive feature bundling (EFB), LGBM significantly reduced training time and memory usage without compromising accuracy. To further enhance model transparency, SHAP values were used, providing clear insights into the contribution of each feature to the model’s predictions.

Recognizing the need for interpretability in clinical applications, SHAP values were applied to explain the model’s predictions (Lundberg & Lee, 2017). This approach allowed for global and local interpretation (Lundberg et al., 2020): At the global level, SHAP summary plots (Figs. 6A and 6B) ranked features by their average importance to the model. In contrast, SHAP dependence plots helped clarify the influence of individual features on prediction outcomes. As shown in Fig. 6C, the relationship between the actual values of the 11 key features and their corresponding SHAP values reveals that positive SHAP values favor predictions for TPE.

For example, a positive or weakly positive result in the tuberculosis antibody (TB-Ab) test (classified as 0 = negative, 1 = weakly positive, 2 = positive) is associated with a higher probability of TPE. Similarly, adenosine deaminase (ADA) levels above 20 U/L significantly increase the likelihood of TPE, especially in regions with high tuberculosis prevalence. ADA is widely recognized as a biomarker for diagnosing TPE, with levels over 40 U/L offering improved sensitivity and specificity, in line with international diagnostic standards (Aggarwal et al., 2019; Porcel, 2009).

Moreover, eosinophil counts in pleural and peritoneal fluid proved to be a critical differentiator. Data suggest that eosinophil levels between 0 and 20% are more indicative of TPE, while levels exceeding 20% point to non-tuberculous causes. This aligns with existing literature linking elevated eosinophil counts to parasitic infections, drug reactions, or malignancies.

The SHAP plots in Fig. 6 demonstrate the contribution of each feature to the diagnosis of TPE, where positive SHAP values indicate that a feature increases the probability of TPE diagnosis. For instance, ADA and TB-Ab show strong positive SHAP values, underscoring their diagnostic importance, particularly in areas with a high prevalence of tuberculosis. These insights can assist clinicians by providing transparency regarding the model’s predictions, thus facilitating more informed decision-making in a clinical setting.

Additionally, Fig. 7 presents a force plot for the internal validation set. The x-axis represents individual patients, while the y-axis displays the contribution of various features to the prediction. A more significant red portion for each patient suggests a higher likelihood of TPE diagnosis. This visualization effectively highlights how specific features influence the model’s predictions, with more significant red segments signifying a more substantial contribution to the TPE diagnosis.

Additional visualization techniques were utilized to clarify the expression patterns of key features and their influence on classification to enhance the SHAP analysis further. The heatmap (Fig. 8) compares the standardized expression levels of various clinical and laboratory parameters between TPE and non-TPE groups. This visualization emphasizes features such as ADA and protein levels in distinguishing TPE, aligning with their high SHAP values. Additionally, principal component analysis (PCA) (Jolliffe & Cadima, 2016) (Figs. 9A, 9B) was employed to examine the distribution of feature parameters in both two-dimensional and three-dimensional spaces. The PCA plots illustrate how key features drive the separation between TPE and non-TPE cases along principal components PC1 and PC2. Notably, ADA and TB-Ab, identified as significant in the SHAP analysis and heatmap, exhibit clear group differentiation in the PCA plots. By integrating SHAP, heatmaps, and PCA, the model’s interpretability is enhanced, further reinforcing the reliability of the diagnostic model for clinical applications (Molnar, 2020).

This figure illustrates global expression changes and clustering of multi-sample, multi-feature parameters between the two groups. Group A represents tuberculous pleural effusion, while Group B represents non-tuberculous pleural effusion. Each row corresponds to a feature parameter, and each column represents a sample. The color gradient indicates standardized expression levels, with red denoting higher values and blue representing lower values. The right side of the heatmap lists the feature parameters, where closer proximity between two features suggests more remarkable similarity in their expression patterns. This heatmap compares the standardized expression levels of clinical and laboratory parameters between TPE and non-TPE groups, highlighting significant features like ADA and protein that distinguish between the two groups.

Practical clinical application

Local interpretability analysis enables personalized input data to predict outcomes for individual pleural effusion patients. The predictive model has been embedded into an application to enhance its clinical applicability, as illustrated in Fig. 10. In the given example, with specific parameter inputs, the model generates a 98% probability for TPE and a 2% probability for non-TPE. Each prediction is tailored to the individual, based on the actual values of the 11 input features, with varying inputs resulting in different probability outcomes.

Sample size and multicenter validation

While this study demonstrated the model’s robustness (Xu & Mannor, 2012), the external validation sample size was small and limited to two medical centers, which may affect its generalizability (James et al., 2013; Hastie, 2009). Future studies will expand the sample size through multicenter collaborations, covering a more comprehensive geographic range and diverse patient populations, to further validate the model’s generalization capabilities. While the model performed well on both internal and external validation datasets, the training data from two hospitals in Zhejiang may introduce regional biases. These biases could affect the generalizability of the model to other populations. Future work should address this by including more diverse data. We acknowledge that potential confounders, such as age, gender, and comorbidities (e.g., diabetes, cardiovascular diseases), may affect the model’s results. Although multivariable adjustments were made in the analysis, these factors may still influence the outcomes. Future studies could further control for these confounders to validate the model’s accuracy and robustness.

Multimodal data integration

Future research could further integrate imaging data (such as chest CT) and genomic data to address the current limitations and improve diagnostic accuracy. Additionally, the systematic review and meta-analysis by Du et al. (2023). on the ability of machine learning to predict AKI risk in patients in the ICU further underscore the significance of machine learning in complex clinical diagnostics, suggesting that similar approaches may apply to the diagnosis of infectious diseases like TPE. Imaging can provide structural information on pleural effusion, while genomic data may reveal genetic susceptibility to TPE. Integrating multimodal data could enhance the model’s applicability (Acosta et al., 2022) in complex clinical scenarios (Liu et al., 2024).

Limitations and future outlook

Although the LGBM model in this study demonstrated high diagnostic accuracy, several limitations remain, including limitations in sample size and data sources (especially the limited data for low-expression TPE cases), data quality and feature selection (such as potential measurement errors or unaccounted factors), and the model’s dependency on input features and hyperparameters (which may be affected by dataset bias or overfitting). Furthermore, the model’s clinical applicability remains limited by factors such as data collection methods, hospital equipment, and clinical environments. The model’s interpretability and clinical usability also need further validation, particularly in multi-center and long-term follow-up studies.

While this study validated the effectiveness of the LGBM model, relying solely on routine laboratory data may not be sufficient for all complex clinical scenarios. Future research should expand the sample size and include other types of pleural effusion (such as malignant pleural effusion) to assess the model’s generalizability. Additionally, optimizing the model’s real-time interpretability will enhance its clinical utility. To improve diagnostic efficiency, the model will be integrated into Hospital Information Systems (HIS) and Laboratory Information Systems (LIS) for real-time data input and automated diagnosis, particularly in resource-limited areas.

Future research directions

Future research will focus on the following areas: expanding the sample size through multicenter collaborations to ensure the model’s generalizability across different regions and populations and exploring the integration of imaging and genomic data to develop more accurate multimodal diagnostic models. The study by Bai et al. (2024) demonstrated that a multimodal approach combining MRI and ultrasound performed excellently in breast disease diagnosis, suggesting that multimodal imaging may also hold potential for TPE diagnosis. Oh et al. (2024) successfully utilized convolutional neural networks (CNN) to detect and classify brain tumors, further highlighting the possibility of combining imaging data with machine learning models for early diagnosis of complex diseases. Lastly, future studies could adopt the TDA-LGBM algorithm proposed by Yang et al. (2024), which uses topological structures to optimize model performance.