Past events

The UMN Machine Learning Seminar Series brings together faculty, students, and local industrial partners who are interested in the theoretical, computational, and applied aspects of machine learning, to pose problems, exchange ideas, and foster collaborations. The talks are every Thursday from 12 p.m. - 1 p.m. during the Fall 2021 semester.

This week's speaker, Yihe Dong (Google), will be giving a talk titled "Attention is not all you need."

Abstract

I will be talking about our recent work on better understanding attention. Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. We show that self-attention possesses a strong inductive bias towards "token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Along the way, we develop a useful decomposition of attention architectures. This is joint work with Jean-Baptiste Cordonnier and Andreas Loukas.

Biography

Yihe Dong is a machine learning researcher and engineer at Google, with interests in geometric deep learning and natural language processing.

Welcome back! The spring 2022 semester begins on Tuesday, January 18.

View the full academic schedule on One Stop.
 

The University of Minnesota will be closed in observance of Martin Luther King, Jr. Day.

View the full schedule of University holidays.
 

The University of Minnesota will be closed in observance of New Year's Day.

View the full schedule of University holidays.
 

The application deadline for the data science majors is December 30.

Undergraduate students typically apply to a major while enrolled in fall semester courses during their sophomore year (third semester).

Submit your application here.

All applicants will be notified of their admission decision via email within three weeks of the application deadline.

The University of Minnesota will be closed (floating holiday).

View the full schedule of University holidays.
 

The University of Minnesota will be closed in observance of Christmas Day.

View the full schedule of University holidays.
 

The University of Minnesota will be closed (floating holiday).

View the full schedule of University holidays.
 

The last day of the fall 2021 semester is Wednesday, December 22.

View the full academic schedule on One Stop.
 

Final exams for fall 2021 will be held between Thursday, December 16 and Wednesday, December 22.

View the full academic schedule on One Stop.