What a I learnt about outcome definitions, patient-grouped splits and honest metrics while
predicting hospital readmission with scikit-learn.
The problem
Hospital readmission is expensive for health systems and harmful for patients. For people with diabetes, a
return to hospital often signals gaps in glycaemic control, medication reconciliation or follow-up. If a
hospital could identify, before discharge, which patients are most likely to return, it could target scarce
follow-up resources where they matter most. I set out to build such a model end to end, using the public
Diabetes 130-US Hospitals dataset from the UCI Machine Learning Repository: 101,766 inpatient encounters from
130 US hospitals between 1999 and 2008.
My clinical training made one thing clear early. The hard part was not fitting a model. It was deciding what
exactly to predict, for whom and when, and then making sure the evaluation reflected how the model would
really be used.
Step 1: Know your rows
Each row is a hospital encounter, not a patient. The 101,766 encounters belong to 71,518 patients, and
patients with repeat admissions contribute almost half of the rows (46.2%). That single fact changes the
evaluation design. If encounters from the same person land in both the training and test sets, the model can
partly recognise the person instead of learning generalisable patterns. In my check, a naive random split
placed 6,791 patients in both sets, and 41.6% of test encounters belonged to someone the model had
already seen.
Step 2: Load the data as the documentation intends
Missing values are coded as ?. But in the HbA1c and glucose columns, the text None means the test was
not performed, which is a real clinical category. pandas silently converts None to a missing value by
default. In my data that erased all 96,420 "not performed" glucose results. Loading with
na_values=['?'] and keep_default_na=False kept the clinical meaning intact. Further "missingness" hid
inside integer codes (admission type 5, 6 and 8 mean not available, NULL and not mapped, covering 10.2% of
encounters). I mapped these to missing using the dataset's own ID mapping file.
Step 3: Define the outcome and the prediction moment
The readmitted column has three values: <30 (within 30 days), >30 (later) and NO. I made two separate
binary outcomes and never mixed them: any readmission (primary, 47.1% prevalence) and 30-day readmission
(secondary, 11.4%). I fixed the prediction time at the end of the stay, before discharge. I removed the 2,423
encounters ending in death or hospice. The data confirmed that all 1,652 encounters ending in death were
labelled NO, which would have taught the model a meaningless rule.
Step 4: A leakage review, column by column
Target leakage is the quiet killer of healthcare models. I wrote a short justification for every variable.
Discharge disposition was used only to define the cohort, not as a predictor, because it is decided at the
very end of the stay. Payer code (billing data, 40% missing) and weight (97% missing) were excluded. Primary
diagnosis codes were grouped into nine broad clinical categories, under an explicit assumption that the
working diagnosis is known before discharge. Secondary diagnosis codes, which are often finalised by coders
after discharge, were left out of the first model. An assertion stops the notebook if any forbidden column
ever enters the feature matrix.
Step 5: Pipelines and a patient-grouped split
I used StratifiedGroupKFold to assign whole patients to an 80/20 train/test split while keeping the
readmission rate balanced (0.471 in both), and asserted zero patient overlap. All learned preprocessing lived
inside a scikit-learn Pipeline with a ColumnTransformer: median imputation, standard scaling, and one-hot
encoding with rare-category pooling (min_frequency=100, handle_unknown='infrequent_if_exist'). Nothing
was ever fitted on test data, even during cross-validation.
outer_split = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
train_idx, test_idx = next(outer_split.split(X, y, groups))
assert not set(groups.iloc[train_idx]) & set(groups.iloc[test_idx]) # zero shared patients
Step 6: Baselines, then models
A majority-class baseline reached about 52.9% accuracy while identifying zero readmitted patients. That
is a useful reminder that accuracy is not a clinical metric. Logistic regression achieved a test ROC-AUC
of 0.679 (95% patient-level bootstrap CI 0.670–0.687) and a PR-AUC of 0.651 against a prevalence of
0.471. Cross-validation on the training set was very stable (0.672 ± 0.003). A histogram gradient-boosting
model reached ROC-AUC 0.691 (95% CI 0.683–0.700) at a training cost of about 10 seconds.
Did boosting meaningfully improve on logistic regression? Not really. The CIs overlap, the gain is 0.013 in
AUC, and at the same sensitivity it improved specificity by only 2.3 percentage points. The signal is
dominated by one, roughly monotonic feature: the number of inpatient admissions in the previous year.
Shuffling it drops ROC-AUC by 0.075, about four times more than any other feature. I kept logistic
regression for its transparency.
Step 7: Thresholds and calibration
I chose the decision threshold on out-of-fold training predictions, not on the test set, using a stated
rule: the highest threshold reaching at least 70% sensitivity. That gave 0.428. At that threshold, test
sensitivity was 0.697, specificity 0.547 and precision 0.579, flagging 56.8% of patients. The rule
transferred almost perfectly from training (70.0%) to new patients (69.7%). The honest reading is that to
catch 7 in 10 readmissions, a team would follow up more than half of all discharges.
The calibration curve hugged the diagonal. Mean predicted risk was 0.469 against 0.471 observed, the
calibration slope was 1.05, and every decile was within 5 percentage points of its observed rate. For a
discharge team, calibration matters: a "60% risk" label should mean roughly six in ten such patients return.
Here it does, at least in this population.
What I would tell another beginner
- Spend the first hours on the data and the question, not the algorithm.
- Write down the prediction time before choosing features.
- Split by patient whenever patients repeat.
- Put every learned transformation inside a pipeline.
- Report PR-AUC, calibration and a threshold table, not accuracy alone.
- Be explicit about what the model cannot do. Mine was trained on US data from 1999–2008 and must not be used for patient care in Kenya without local validation.
The full code, figures and report are on GitHub: https://github.com/Mwaivictor/diabetes-readmission-prediction.
Dataset: Clore, Cios, DeShazo and Strack (2014), UCI Machine Learning Repository, CC BY 4.0
(DOI 10.24432/C5230J). Original study: Strack et al. (2014), BioMed Research International.
