Multimedia Event Knowledge Extraction: From Object-Centric to Event-Centric Understanding

Speaker: Manling Li (UIUC)

Date and Time: Friday, September 30 10am CT

Place: 3401 SC or Zoom (link below)


Human memories can be viewed as repositories of historical events. An event is a semantic structure encapsulating the fundamental questions of Who? What? Where? When? Why?, which are of primary concern to humans. The enormous volume of data requires machines to automatically obtain such factual knowledge including events and their arguments (participants) and perform reasoning by synthesizing a wide variety of unstructured data, such as text, images, and videos. However, current event understanding is text-only, local, and lacking in reasoning.

This talk focuses on constructing event graphs to deal with real-world events that are multimedia, interconnected, probabilistic, and span a long time period. We discover the local event structures (i.e., who, what, where, and when) by transferring the semantic understanding ability from text to vision during vision-language pretraining. To perform global reasoning across events (i.e., what is likely to occur, and why), we further investigate the interactions between events such as the temporal orders and the evolution patterns along a long period, and leverage historical events to discover such event schema knowledge globally. Our structural event graph modeling is able to represent the global inter-dependencies of events and long-distance interactions via arguments, leading to a comprehensive understanding of events and effective forecasting of future events.


Manling Li is a Ph.D. candidate at the Computer Science Department of University of Illinois Urbana-Champaign. Her work on multimedia knowledge extraction won the ACL'20 Best Demo Paper Award and NAACL'21 Best Demo Paper Award. She was selected as a DARPA Riser (nominated by DARPA) and a EE CS Rising Star in 2022. She was a recipient of Microsoft Research PhD Fellowship in 2021. She was awarded C.L. Dave and Jane W.S. Liu Award, and has been selected as Mavis Future Faculty Fellow. She has more than 30 publications on multimedia knowledge extraction and reasoning, and gave tutorials about multimedia IE at AAAI'21, ACL'21 and NAACL'22. Additional information is available at


Meeting ID: 816 789 9060

Password: kairos