Department of Image Processing

Foundation Models are an emerging approach to AI where a single model (like GPT-3 or DALL-E) is trained on a wide range of data and then adapted to many different tasks. I personally view their popularity as unfortunate, due to the black-box nature of these models. However, that doesn't change the fact that understanding these developments might prove crucial --- both altruistically (to make their impact more beneficial) and for selfish reasons (to not be caught off guard should these models make prior approaches to one's research field obsolete). In this talk, I will focus on building the intuition for self-supervised learning and foundation models by giving various examples of their behaviour. I will mention scaling laws, which allow us to predict the future performance of these models. Finally, if time permits, I will also discuss the behaviour of self-supervised learning in the hypothetical limit of perfect performance.