CRAY Colloquium: "On the Mythos of LLM Thinking Tokens"

This week's speaker, Subbarao Kambhampati from Arizona State University, will be giving a talk titled, "On the Mythos of LLM Thinking Tokens."

Abstract

Ever since DeepSeek R1, the intermediate tokens output by the Large Reasoning Models (LRMs) have been ascribed several magical properties-- that they provide an interpretable window into the "thinking" of the model, that their length is a reliable indicator of the computational complexity of the task, and that the main contribution of RL-based post-training is to improve model's thinking--as evidenced by the average length of the thinking tokens increasing during training. In this talk, I will discuss several recent efforts from our group that systematically dissect these claims and offer counterintuitive conclusions. I will also discuss implications of these efforts to agentic systems, as well as some alternate ways to get LLMs to provide verifiable traces of their problem solving--that can be used in safety critical applications.

Biography