Brier Score and Integrated Brier Score

Brier Score

The Brier score is used to evaluate the accuracy of a predicted survival function at a given time $t$; it represents the average squared distances between the observed survival status and the predicted survival probability and is always a number between 0 and 1, with 0 being the best possible value.

Given a dataset of $N$ samples, $\forall i \in [\![1, N ]\!], \left(\vec{x}_i, \delta_i, T_i \right)$ is the format of a datapoint, and the predicted survival function is $\hat{S}(t, \vec{x}_i), \forall t \in \mathbb{R^+}$:

In the absence of right censoring, the Brier score can be calculated such that:

However, if the dataset contains samples that are right censored, then it is necessary to adjust the score by weighting the squared distances using the inverse probability of censoring weights method. Let $\hat{G}(t) = P[C > t ]$ be the estimator of the conditional survival function of the censoring times calculated using the Kaplan-Meier method, where $C$ is the censoring time.

In terms of benchmarks, a useful model will have a Brier score below $0.25$. Indeed, it is easy to see that if $\forall i \in [\![1, N]\!], \hat{S}(t, \vec{x}_i) = 0.5$, then $BS(t) = 0.25$.

Location

The function can be found at pysurvival.utils.metrics.brier_score.

API

brier_score - Brier score computations

    brier_score(model, X, T, E, t_max=None)


Parameters:

• model : Pysurvival object -- Pysurvival model

• X : array-like -- input samples; where the rows correspond to an individual sample and the columns represent the features (shape=[n_samples, n_features]).

• T : array-like -- target values describing the time when the event of interest or censoring occurred.

• E : array-like -- values that indicate if the event of interest occurred i.e.: E[i]=1 corresponds to an event, and E[i] = 0 means censoring, for all i.

• t_max: float ( default=None ) Maximal time for estimating the prediction error curves. If missing the largest value of the response variable is used.

Returns:

• times: array-like. A vector of timepoints. At each timepoint the brier score is estimated

• brier_scores: array-like. A vector of brier scores

Integrated Brier Score

The Integrated Brier Score (IBS) provides an overall calculation of the model performance at all available times.

Location

The function can be found at pysurvival.utils.metrics.integrated_brier_score to output the values and pysurvival.utils.display.integrated_brier_score to display the predictive error curve.

API

integrated_brier_score - Integrated Brier score computations

integrated_brier_score(model, X, T, E, t_max=None, figure_size=(20, 10))


Parameters:

• model : Pysurvival object -- Pysurvival model

• X : array-like -- input samples; where the rows correspond to an individual sample and the columns represent the features (shape=[n_samples, n_features]).

• T : array-like -- target values describing the time when the event of interest or censoring occurred.

• E : array-like -- values that indicate if the event of interest occurred i.e.: E[i]=1 corresponds to an event, and E[i] = 0 means censoring, for all i.

• t_max: float ( default=None ) Maximal time for estimating the prediction error curves. If missing the largest value of the response time variable is used.

• figure_size: tuple of double ( default=(20, 10) ) width, height in inches representing the size of the chart of the survival function. Option available if the function is being called from pysurvival.utils.display

Returns:

• ibs: double The integrated brier score