TorchMetrics -- How do we use it, and what's the difference between .update() and .forward()?

Follow the full discussion on Reddit.
TorchMetrics is a really nice and convenient library that lets us compute the performance of models in an iterative fashion. It's designed with PyTorch (and PyTorch Lightning) in mind, but it is a general-purpose library compatible with other libraries and workflows. This iterative computation is useful if we want to track a model during iterative training or evaluation on minibatches (and optionally across on multiple GPUs). In deep learning, that's essentially all the time. However, when using TorchMetrics, one common question is whether we should use .update() or .forward()? (And that's also a question I certainly had when I started using it.). Here's a hands-on example and explanation.