Model Tuning & Regularization¶

Methods that constrain the complexity of a model in order to reduce overfitting and improve out-of-sample error, by reducing variance and compromising on an increased bias

Reducing capacity¶

Reduce the number of

• parameters (weights)
• layers (neural net)
• units per layers (neural net)
• Bottle neck layers (neural net)

Bottleneck Layer(s) for neural nets¶

Let

• Let layer \(i\) and layer \(j\) be adjacent layers
• \(w_i\) be weights of layer \(i\)
• \(\vert w_i \vert\) be the number of units in a layer \(i\)
• \(w_b\) be the bottleneck layer
• effective only if \(\vert w_b \vert < \arg \min(\vert w_i \vert, \vert w_j \vert)\)

Example

flowchart LR
a[10<br/>units] -->|100<br/>connections| b[10<br/>units]
c[10<br/>units] -->|10<br/>connections| d[1<br/>unit] -->|10<br/>connections| e[10<br/>units]

Increase DOF¶

• Reduce \(k\)

• Feature Selection

• Dimensionality Reduction

• Increase \(n\)

• Data augmentation

Subsampling at each iteration¶

• columns
• rows

Weight Decay¶

Also called as Shrinkage Term/Regularization Penalty

Reduce errors by fitting the function appropriately on the given training set, to help reduce variance, and hence avoid overfitting, while minimally affecting bias.

This is done by adding a penalty term in the cost function.

Note:

• All input features must be standardized
• Intercept should not be penalized

This is similar to $$ E_\text{out} = E_\text{in} + O(D_{VC}) $$

where

• \(\mu_{\beta^*_j}\) is the prior-known most probable value of \(\beta_j\)
• \(\sigma^2_{\beta^*_j}\) is the prior-known standard deviation of \(\beta_j\)

These 2 incorporate desirable Bayesian aspects in our model.

Bayesian Interpretation¶

Regularization amounts to the use of informative priors, where we introduce our knowledge or belief about the target function in the form of priors, and use them to “regulate” the behavior of the hypothesis we choose

Incorporating \(\hat \beta\) into the regularization incorporates maximum likelihood estimate of the coefficients.

Didn’t understand: The standard deviation of the prior distribution corresponds to regularization strength \(\lambda\)

Example

IDK¶

Contours of Regularizers¶

Why is standardization required?¶

• magnitudes of parameters need to be comparable
• Penalized estimates are not scale equivariant: multiplying \(x_j\) by a constant \(c\) can cause a significant change in \(\hat \beta\) when using regularization

Feature Selection Paths¶

Penalty Coefficient¶

Frequentist Interpretation¶

Select coefficient one/two iterations after the optimal point, to reduce variance further

\(\lambda\) regulates the smoothness of the spline that solves $$ \min_h \left { J(\theta, x_i, y_i) + \lambda \int [h''(t)]^2 \cdot dt \right } $$ Penalizing the squared \(k\)th derivative leads to a natural spline of degree \(2k − 1\)

Multi-Stage Regularization¶

Intuition: Since vars in 2^nd stage have less "competition" from noise variables, estimating using selected variables could give better results

System Equation Penalty¶

Useful if you know the underlying systematic differential equation

$$ J'(\theta) = J(\theta) + \text{DE} \ \text{RHS(DE)} = 0 $$ Refer to PINNs for more information

IDK¶

Early-Stopping¶

Stop one/two iterations before the optimal point, to reduce variance further

Dropout¶

Dropout is applied on the output of hidden fully-connected layers

Training¶

Stochastically drop out units with probability \(p_\text{drop}\) and keep with \(p_\text{keep}=1-p_\text{drop}\)

Annealed dropout

Evaluation/Production¶

At inference time, dropout is inactive, as we should not predict stochastically

Ensembling¶

Noise Injection¶

• Add noise to inputs
• Add noise to outputs

Behaves similar to L2 regularization

Label Smoothing¶

where

• \(y_\text{true}\) is the true label
• \(m=\) total number of labels

Active Teaching¶

flowchart LR
c(( ))
m[ML]

d[(Dataset)] --> m & EDA

EDA -->
rb[Rule-Based System]

subgraph Active Teaching
  rb & m <-->|Compare & Update| c
end

Comments