CSE DSI Machine Learning Seminar with Jiawei (Joe) Zhou (CS, Stony Brook)

The Future of NLP (→ AI) Systems: Efficiency, Multimodality, and Trustworthiness

Technologies from natural language processing (NLP), particularly large language models (LLMs), have transformed how we approach and even define AI. NLP, once a subfield of AI, has now taken center stage as the foundation of generative AI. At the same time, many traditional notions and methods of NLP are becoming obsolete—“almost dead”—overtaken by the paradigm established by LLMs across a vast range of applications. This transformative moment brings uncertainty for classical NLP research, opens new opportunities for the next generation of AI, and poses challenges amid a rapidly shifting landscape.

In this talk, I will reflect on the trajectory from NLP to LLMs and toward broader AI, discuss the remaining challenges, and highlight promising research frontiers: pushing the limits of model and system efficiency, advancing multimodal grounding and reasoning, and building truly personalized and trustworthy AI systems with agency. I will share our recent work addressing these directions—through algorithmic innovations for accelerating LLM inference, integrating dynamic knowledge, reducing hallucination in vision-language models (VLMs), leveraging visual text inputs, and evaluating model fairness and personalization. I will conclude with reflections on what the future of NLP—or rather, AI systems—might look like.

Dr. Jiawei (Joe) Zhou is an assistant professor in Data Science, Applied Math and Statistics, and Computer Science at Stony Brook University. Previously he was a research assistant professor at the Toyota Technological Institute at Chicago (TTIC), and obtained a PhD degree in computer science from Harvard University. Dr. Zhou’s expertise spans in NLP and ML applications, especially generative models with Transformers like large language models (LLMs) and multimodal models such as vision language models (VLMs). His work encompasses technical research such as decoding algorithms and broader applications of trustworthy generation from different sources. His early contributions to executable semantic parsing for efficient conversation systems were recognized with an ACL’22 Outstanding Paper Award. He also received the Best Student Paper at IEEE ASRU’25, and the Amazon Research Award 2025 on efficient long-horizon agentic reasoning. He maintains broad research interests in NLP and related areas, such as evaluation, efficiency, knowledge representation, advanced reasoning and reinforcement learning, and relevant techniques in multimodal AI.