Over the past decades, many papers have been written about computationally efficient Gaussian processes, and much progress has been made. So given this progress, do we have a clear answer to the question of “which approximate Gaussian process method should I use?” This, the literature is not clear on. We argue that this is because tuning approximation methods is difficult. As a consequence, we believe that creating an automatically tuning GP approximation is the current largest open problem. This brings another problem: the common benchmarking procedures do not adequately measure how “automatic” an approximation is. We highlight some problems, and make some suggestions on what could be done differently in the future. However, we also conclude that benchmarking really well, is difficult, and takes a lot of effort.