Constant improvement in prediction metrics has been the motor behind machine learning success. However, finding metrics for probabilistic predictions is more challenging. In this talk we argue that it is crucial to develop metrics that are meaningful for a particular task, and review the theory of scoring rules and decision theory that underpins this.