Michalis Titsias: Functional Regularisation for Continual Learning with Gaussian Processes

Abstract

We introduce a framework for Continual Learning (CL) based on Bayesian inference over the function space rather than the parameters of a deep neural network. This method, referred to as functional regularisation for Continual Learning, avoids forgetting a previous task by constructing and memorising an approximate posterior belief over the underlying task-specific function. To achieve this we rely on a Gaussian process obtained by treating the weights of the last layer of a neural network as random and Gaussian distributed. Then, the training algorithm sequentially encounters tasks and constructs posterior beliefs over the task-specific functions by using inducing point sparse Gaussian process methods. At each step a new task is first learnt and then a summary is constructed consisting of (i) inducing inputs -- a fixed-size subset of the task inputs selected such that it optimally represents the task -- and (ii) a posterior distribution over the function values at these inputs. This summary then regularises learning of future tasks, through Kullback-Leibler regularisation terms. Our method thus unites approaches focused on (pseudo-)rehearsal with those derived from a sequential Bayesian inference perspective in a principled way, leading to strong results on accepted benchmarks.

Bio

Michalis Titsias received a Diploma and MSc degree in Informatics from the University of Ioannina, Greece. He obtained a PhD degree from the School of Informatics, University of Edinburgh, in 2005. From October 2007 to July 2011, he worked as a research associate at the School of Computer Science of the University of Manchester, while from November 2011 to September 2012 he worked as a postdoctoral research scientist in statistical cancer genomics at the Wellcome Trust Centre for Human Genetics and the Department of Statistics at the University of Oxford. Form 2012 to 2018 he was an Assistant Professor at the Department of Informatics in Athens University of Economics and Business. From 2018 up to present he works as a Research Scientist at DeepMind.