Language and graph foundational models: Distillation and pretraining

Industrial Problems Seminar

Vasileios Ioannidis (Amazon Search AI)

Please note the 10:10am start time.

Abstract

Graph neural networks (GNNs) learn from complex graph data and have been remarkably successful in various applications and across industries. This presentation first introduces GNNs via the message passing framework and dives into popular GNN variants. Next, it explores the fusion of textual data with heterogeneous graph structures to improve semantic and behavioral representations. It introduces the Language Model GNN (LM-GNN), a framework that efficiently combines large language models and Graph Neural Networks (GNNs) through fine-tuning. LM-GNN supports various tasks like node classification and link prediction and demonstrates its effectiveness. Another aspect addressed is the challenge of effective node representation learning in textual graphs. The Graph-Aware Distillation (Grad) framework is proposed, which encodes graph structures into a Language Model (LM) to enable fast and scalable inference. Grad optimizes GNN and a graphless student model, resulting in superior performance in node classification tasks. Finally, the presentation discusses pre-training text and graph models on large, heterogeneous graphs with textual data using the Graph-Aware Language Model Pre-Training (GALM) framework. It highlights the framework's effectiveness through experiments on real datasets.