July 2, 2015

Who am I?

  • Background
    • Bachelor in Mathematics
    • Master in Statistics
    • PhD in Biostatistics

  • Job
    • Statistical consultant for Open Analytics NV

The research discussed in this presentation was conducted during my PhD at the KU Leuven.

Diagnosis of ovarian tumors

A good accurate specific pre-operative diagnosis is crucial for:

  • correct triaging
    • referral to specialized centers
    • referral to gynecological oncologist
  • correct choice of optimal management
    • avoiding rupture and residual disease in malignant tumors
    • avoiding overly invasive management of benign tumors

Accuracy/Reliability of predictions

Current approaches

Before analysis

During analysis

Multinomial/Multiclass calibration

First framework: Parametric

  • Quantification
    • Calibration intercept
      • < 0: overestimation
      • > 0: underestimation
    • Calibration slope
      • < 1: overfitting
      • > 1: underfitting
    • Test perfect calibration
  • Visualization
Note: specific for multinomial logistic regression.

Second framework: Generic

  • Quantification
    • Estimated Calibration Index (ECI)
      • rescaled average squared difference between predicted and observed probabilities
      • ranges between 0 and 100
  • Visualization

Note: can be used irrespective of the adopted algorithm or used risk prediction model.

multiCalibration package

Wrapper for multinomial logistic regression model

multiCalibration <- function(outcome, k, p = NULL, LP, r = 1, 
    estimates = FALSE, dfr = 2, parametric = TRUE, generic = TRUE, 
    plotseparate = TRUE, plotoverall = TRUE, datapoints = TRUE, 
    smoothing = TRUE, smoothpar = 1, eci = TRUE, intercept = FALSE, 
    slope = FALSE, test = FALSE, legendOutcome = NULL, 
    pathGraphs = "./"){...}

Wrapper for generic multiclass risk prediction model

multiCalibrationGeneric <- function(outcome, k, p, r = 1, dfr = 2, 
    generic = TRUE, plotseparate = TRUE, plotoverall = TRUE, 
    datapoints = TRUE, smoothing = TRUE, smoothpar = 1, eci = TRUE, 
    legendOutcome = NULL, pathGraphs = "./"){...}

Example

Code

    multiCalibration(outcomeTest, k = 5, p = pTest, LP = lpTest, 
            r = 1, estimates = FALSE, dfr = 2, parametric = TRUE, 
            generic = TRUE, plotseparate = TRUE, 
            plotoverall = TRUE, datapoints = FALSE, smoothing = TRUE, 
            smoothpar = 1.5, eci = TRUE, intercept = TRUE, 
            slope = TRUE, test = TRUE, legendOutcome = c("benign",
            "borderline", "stage I invasive", "stage II-IV invasive", 
            "metastatic"), pathGraphs = "./graphs/test_")

Note: the calibration is expected to be 'not perfect' in the considered case study, since the data were split in a specific way which enforced differences in training and validation data.

Quantification

## $ECI
## [1] 0.3432562
## 
## $calibrationIntercepts
##    calInt.1  calIntLL.1  calIntUL.1    calInt.2  calIntLL.2  calIntUL.2    calInt.3 
## -0.60132379 -0.82962752 -0.37302006 -0.54280519 -0.77736772 -0.30824265 -0.49632892 
##  calIntLL.3  calIntUL.3    calInt.4  calIntLL.4  calIntUL.4 
## -0.72634453 -0.26631331 -0.18817938 -0.46252389  0.08616514 
## 
## $calibrationSlopes
##   calSlopeLp.1 calSlopeLpLL.1 calSlopeLpUL.1   calSlopeLp.2 calSlopeLpLL.2 calSlopeLpUL.2 
##      1.0183461      0.7380275      1.2986646      0.9987075      0.7344830      1.2629320 
##   calSlopeLp.3 calSlopeLpLL.3 calSlopeLpUL.3   calSlopeLp.4 calSlopeLpLL.4 calSlopeLpUL.4 
##      0.8816050      0.6476059      1.1156042      1.0048464      0.6924594      1.3172334 
## 
## $Deviances
##  devianceOriginal devianceIntercept    devianceSlopes 
##          2130.282          2082.359          2070.403 
## 
## $PValues
##     pOverall         pInt      pSlopes 
## 4.922852e-10 9.795137e-10 1.767830e-02

Visualization

Visualization

References

  • Van Hoorde, K. et al. Assessing calibration of multinomial risk prediction models. Statistics in Medicine, 2014 Jul 10; 33(15): 2585-2596.
  • Van Hoorde, K. et al. A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics, 2015 Apr; 54: 283-293.
  • Van Calster, B., Van Hoorde K. et al. Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study. BMJ, 2014 Oct 15; 349:g5920.

Acknowledgments