Gaussian processes are the gold standard method for solving regression problems where noise and uncertainty quantification are important. In recent years, high-quality approximations have been developed to reduce their computational cost on large datasets. While the fundamental approximations (e.g. sparse variational or conjugate gradient) have been around for over a decade, applying them is still cumbersome due to the human “folk knowledge” that is required. In this talk, I will discuss our work on eliminating the need for this cumbersome human intervention. We do this by mathematically analysing the methods, and using the insights gained to provide automated methods for setting parameters. The resulting procedures are more robust, computationally efficient, and avoid heuristic setting of many parameters.