Hybrid Talk

Abstract: 

Large Language Models (LLMs) such as Chat-GPT, GPT-4, Bard, PaLM have recently demonstrated an almost shocking level of language understanding and generation abilities, passing a wide variety of complex tests from GRE and SAT to Bar Examination.

Even more impressively, the latest of these models have demonstrated understanding (and ability to manipulate) complex artifacts of other modalities, such as images and code.Despite the fact that, as proprietary models, details of their neural architectures and training objectives are not disclosed, all evidence suggests that it is in fact the sheer scale of these models (e.g. GPT-4 is speculated to have tens of trillions of parameters) and the data on which they were trained the key factor to their unprecedented abilities. In fact, even in controlled experiments with smaller language models, certain abilities have been shown to emerge (hence dubbed »emerging abilities«) only at a certain scale.

In this talk, I will first cover the (known) technical details of LLMs and their training procedures. In the second part, I will focus on emerging abilities (at different scales) as well as cases on which LLMs still fail. Finally, I will conclude with a discussion of implications that the observation that »scale is all that matters« has on future AI research, and NLP research in particular.