Model Evaluation¶

Goals¶

1. Minimize bias
2. Minimize variance
3. Minimize generalization gap

Guidelines¶

• Metrics computed from validation set may not be representative of the true population
• Never trust a single summary metric
• Always look at all the individual metrics
• false positives and false negatives are seldom equivalent
• understand the problem to known the right tradeoff

Model Selection¶

1. Fit multiple models \(g_i\) on the training data
2. Use interval validation data for hyper parameter tuning of each model \(g_i\)
3. Use external validation data for model selection and obtain \(g^*\)
4. Combine the training and validation data. Refit \(g^*\) on this set to obtain \(g^{**}\)
5. Assess the performance of \(g^{**}\) on the test data

Finally, train \(g^{**}\) on the entire data to obtain \(\hat f\)

Baseline¶

Always establish a baseline

• Basic/Naive/Dummy predictions

• Regression

  • Mean
  • Max
  • Min

• Classification

  • Most frequent value
  • Random

• Time series specific

  • Lag/Seasonal Lag
  • Last value available

• Human Level Performance

• Literature Review
• Performance of older system

Significance¶

All the evaluation should be performed relative to the baseline.

For eg: Relative RMSE = RMSE(model)/RMSE(baseline), with “good” threshold as 1

Probabilistic Evaluation¶

Now, we need to see if any increase or decrease in accuracy due to hyper-parameter tuning is statistically-significant, or just a matter of chance.

Regression Evaluation¶

The reason this works is because, here \(\chi^2\) is just sum of normally-distributed error terms

Probabilistic Evaluation¶

You can model the error such as MAE as a \(\chi^2\) distribution with dof = \(n-k\)

The uncertainty can be obtained from the distribution

Spurious Regression¶

Misleading statistical evidence of a relationship that does not truly exist

Occurs when we perform regression between

• 2 independent variables

and/or

• 2 non-stationary variables

(Refer econometrics)

You may get high \(R^2\) and \(t\) values, but \(u_t\) is not white noise (it is non-stationary)

Variance of error term becomes infinite as we go further in time

Classification Evaluation¶

Graphs¶

Graph		Preferred
ROC Curve		As high as possible At least higher than 45deg line
Calibration Graph		Along 45deg line

Probabilistic Evaluation¶

You can model accuracy as a binomial distribution with

• \(n=\) Validation set size
• = No of predictions
• = No of k folds * Validation Set Size
• \(p=\) Obtained Accuracy of classifier

The uncertainty can be obtained from the distribution

for n in [100, 1_000, 10_000, 100_000]:
  dist = stats.binom(n, 0.7)

  alpha = 0.025

  interval_width = dist.isf(alpha) - dist.isf(1-0.975)
  print(f"Size: {interval_width/n * 100}")
  # returns alpha % of observed accuracy that fall outside the inverval --> This is the maximum further accuracy that is theoretically achievable

Confusion Matrix¶

\(n \times n\) matrix, where \(n\) is the number of classes

Binary Classification¶

Multi-Class Classification¶

Confusion Matrix with respect to A

Classifcation Accuracy Measures¶

Jacquard Index¶

F1 Score¶

Macro Average¶

Weighted Average¶

Summary Statistics¶

Don’t just look at the mean evaluation metric of the multiple splits

Also use standard deviation & standard error to also get the uncertainty associated with the validation process.

Residual Analysis¶

Perform all the plots for train and validation data

Numeric¶

• Series of \(u_i\) are random

• \(E(u_i | x_i) = 0\)

  1. Symmetric distribution for values of error terms for a given value \(x\)
  2. Not over time/different values of \(x\)
  3. This means that
  4. you have used up all the possible factors
  5. \(u_i\) only contains the non-systematic component

• Homoscedascity of variance

  1. \(\text{var}(u_i | x_i) = \sigma^2 (u_i|x_i) = \text{constant}\) should be same \(\forall i\)
  2. For the Variance of distribution of potential outcomes, the range of distribution stays same over time
  3. \(\sigma^2 (x) = \sigma^2(x-\bar x)\)

  else, the variable is volatile; hard to predict; we cannot use OLS

  • if variance decreases, value of \(y\) is more reliable as training data
  • if variance increases, value of \(y\) is less reliable as training data
  • We use volatility modelling (calculating variance) to predict the pattern in variance

Why is this important?¶

For eg: Running OLS on Anscombe’s quartet gives

• same curve fit for all
• Same \(R^2\) for all

But clearly it is not optimal

which is shown in the residual plot

Log Likelihood¶

def ll(X, y, pred):
    # return log likelihood

        mse = np.mean(
      (y - pred)
      **2
    )

    n = float(X.shape[0])
    n_2 = n/2

    return -n_2*np.log(2*np.pi) - n_2*np.log(mse) - n_2

def aic(X, y, pred):
    p = X.shape[1]

    return -2*ll(X, y, pred) + 2*p

print(aic(X, y, pred))

Evaluation Curves¶

Related to Interpretation

Always look at all curves with uncertainties wrt each epoch, train, hyper-parameter value. The uncertainty of the train set and test set should also be similar. If train set metric uncertainty is small and test set metric uncertainty is large, this is bad even if the average loss metric is same

Learning Curve¶

	Conclusion	Adding more data will help	Comment
	High Bias (Underfitting)	❌	Model can’t fit larger datasets
	High Variance (Overfitting)	✅
	High Bias High Variance

Loss Curve¶

Same Model, Variable Learning Rate¶

IDK¶

Comments