Disease diagnosis using symptoms and patient data
Abstract
This project developed a machine learning diagnostic system to predict diseases based on patient-reported symptoms, aiming to support healthcare workers in low-resource settings. By utilizing a unified dataset of over 32,000 patient records, the study addressed challenges such as overlapping symptoms and class imbalance. Using a tree-based modeling strategy, including models like Random Forest, XGBoost, and LightGBM combined into a soft voting ensemble, the system achieved high performance with a balanced F1-score across disease classes. The resulting tool, deployed via Streamlit, facilitates faster and more accurate triage decisions, demonstrating the practical application of data science in enhancing healthcare outcomes.
Problem statement
Many infectious diseases such as malaria, typhoid, and tuberculosis share overlapping symptoms like fever, headache, and fatigue, often leading to misdiagnosis and delayed treatment. Additionally, demographic variables in most healthcare datasets are often inconsistent, missing, or incomplete, which reduces model reliability.
Objectives
- Build a data-driven diagnostic system capable of predicting common illnesses using only symptom data.
- Design a model capable of handling class imbalance to ensure rare diseases like tuberculosis and diabetes are not overlooked.
- Support healthcare workers by providing faster and fairer diagnosis, especially in low-resource settings.
Methodology
The project involved sourcing and merging multiple datasets from Kaggle into a unified dataset of over 32,000 patient records where symptoms were binary-encoded (1 for present, 0 for absent). Preprocessing steps included converting textual entries to lowercase, standardizing symptom and disease names, handling missing values, and removing duplicates. Exploratory Data Analysis (EDA) included calculating class distribution, using Chi-square tests for feature ranking, and UMAP projection for high-dimensional visualization. A tree-based, tabular modeling strategy was adopted. Models trained included Decision Tree (baseline), Random Forest, XGBoost, and LightGBM. To address class imbalance, strategies included using 'class_weight='balanced'' for Decision Tree and Random Forest, 'scale_pos_weight' for XGBoost, and 'is_unbalance=True' for LightGBM. Finally, a Soft Voting Ensemble was created combining the top three models (Random Forest, XGBoost, and LightGBM) to aggregate probability scores.
Results
Evaluation prioritized fairness and class-level metrics over overall accuracy. Decision Tree (Tuned) achieved 0.536 accuracy, 0.936 precision, 0.536 recall, and 0.661 F1-Score. Random Forest (Tuned) achieved 0.948 accuracy, 0.951 precision, 0.948 recall, and 0.948 F1-Score. XGBoost (Tuned) achieved 0.949 accuracy, 0.951 precision, 0.949 recall, and 0.949 F1-Score. LightGBM (Tuned) achieved 0.944 accuracy, 0.948 precision, 0.944 recall, and 0.945 F1-Score. The ensemble model improved minority disease recognition and prediction stability. The final model was deployed via Streamlit for user interaction.
Conclusion
The project successfully demonstrated that symptom-based disease prediction is achievable using machine learning. The ensemble model effectively improved prediction confidence and fairness across multiple diseases. Future work aims to incorporate demographic and clinical features, validate the model with hospital datasets, and explore deep learning architectures.