Day 25 - Custom Metrics
Metrics and loss functions, although used interchangably sometimes, are conceptually different.
Loss Functions
- Used by an optimization algorithm (e.g. Gradient Descent) to train a model.
- Need to be differentiable so that training works.
- Can't have a gradient of 0 everywhere (then training won't happen).
- Don't necessarily have to be easily interpretable by humans.
Metrics
- Used to evaluate a model.
- They need to be interpretable by humans.
- It's okay if they are non-differentiable.
- It's okay if they have 0 gradients everywhere.
Defining a custom metric function is the same as defining a custom loss function in most cases.
A metric can either be stateless or stateful. The Keras documentation has examples of both.
Stateless Metrics
Like loss functions, stateless metrics are simple functions that take y_true and y_pred as inputs and return a metric value.
Keras keeps track of this metric and its mean over the entire epoch.
Stateful Metrics
(aka streaming metrics)
Some metrics like precision cannot be averaged across batches. You need to keep track of the count of true positives and false positives across all batches and then calculate the value of the metric at the end of the epoch.
This can be done by subclassing tf.keras.metrics.Metric and implementing the following methods:
- __init__ to create variables to keep track of the metric's state over multiple batches.
- update_state to update values of the metric variables given
y_trueandy_pred.
- result to return the final value of the metric.
- reset_states to reset all variables to 0.