CSE DSI Machine Learning Seminar with Boran Han (AWS AI)

Towards efficient foundation models via context-awareness and continual pretraining

The development of efficient foundation models is crucial in advancing AI technologies. Our research introduces novel methodologies, including context-aware multimodal Large Language Models (LLMs), efficient domain-specific pretraining leveraging existing foundation models, and the efficient transfer of knowledge across multiple foundation models. Specifically, we unveil CaMML, an innovative module for integrating multimodal contexts through Retrieval Augmented Generation (RAG). To address domain-specific pretraining challenges, particularly in the geospatial domain, we employ a multi-objective continual pretraining paradigm that leverages existing foundation models, supported by a comprehensive dataset named GeoPile. Additionally, our Adaptive Feature Transfer (AFT) technique efficiently and adaptively utilizes the pre-trained features for downstream tasks enabling the cross-architecture transfer from multiple foundation models, validating in text, image, and multimodal domains. This thread of work enhances the capabilities of large models, improving their accessibility and applicability across various domains.

Boran Han is an Applied Scientist at AWS AI. She earned her Ph.D. from Harvard University. Since joining AWS, Boran has redirected her research focus towards the development of multimodal foundation models and its scientific applications. Her research objective is to enhance the accuracy and robustness of multimodal foundation models.