PROGRAM

Title: Domain Generalization in Complex Multi-modal Large Language and Action Models

Abstract: Multimodal large language and action models have enabled rapid progress in visual question answering, reasoning, and perception-conditioned decision-making agents. However, these systems remain brittle when adapted to downstream domains, especially through fine-tuning on custom datasets or deployment in robotics. This domain generalization problem is significantly more complex to solve, because there is a range of modules involved (e.g. vision, language, and action heads) in the same model with varying amounts of distribution shift relative to their respective pre-training, different capacities, and different amounts of both pre-training and fine-tuning data. This talk will examine robustness across the lifecycle of training multimodal action models, specifically through preserving generalization during fine-tuning, detecting failures during deployment, and recovering from them upon detection. I will first present methods that constrain fine-tuning to retain the out-of-distribution generalization of large pretrained models, with applications across computer vision, multimodal models, and vision-language-action models in robotics. I will then briefly discuss how post-training and inference can alleviate remaining brittleness to distribution shift through verification functions that can monitor trajectories and reasoning chains of decision-making agents to enable run-time failure detection, monitoring, and recovery. I will conclude with open problem in understanding and solving complex multi-modal distribution shifts.

Title: Targeted Human Verification for Robust Deployment

Abstract: Machine learning models are increasingly deployed to process real-world data, and while model capabilities are improving and expanding, these models still make systematic errors that, if left unchecked, can result in biased and sometimes actively harmful dataset-scale analysis. Expert verification remains expensive and difficult to access at scale, particularly in scientific domains. We look at optimizing the use of expertise along several dimensions: first, we explore how humans can efficiently select from a pool of models, such as those on Hugging Face, to identify which model is best-performing for their data of interest. Second, we show human verification directly compliments active learning systems for rare category discovery in new deployments (domains), and propose a new discovery-focused stopping criterion inspired by ecological rarefaction that consistently outperforms existing systems. Third, we show that strategic review guided by scientific inference uncertainty rather than classification uncertainty enables robust population-level inference with 2-4× fewer expert labels. Our approaches robustly integrate ML into scientific workflows, where nearly all new data represents a distribution shift from what was seen before. We demonstrate direct applications in ecology that enable scientists to spend review effort on what matters for conservation action and scientific understanding: reliable species discovery, distribution modeling, and estimating covariate effects.

Title: Sharpness-Aware Optimization for Domain Generalization

Abstract: Domain-generalization methods typically target the data, the architecture, or the loss; this talk instead considers the optimizer, focusing on sharpness-aware minimization (SAM). In feature learning, SAM adaptively suppresses features the network has already learned well, counteracting SGD's simplicity bias and yielding a more diverse feature set; this improves out-of-distribution performance on data with spurious or redundant features, including CelebA, Waterbirds, CIFAR-MNIST, and DomainBed. In pretraining, the same approach reduces catastrophic forgetting under subsequent parameter updates: it cuts forgetting by up to 80% after post-training across models from 20M to 150M parameters, and a brief SAM mid-training phase on a 1B-parameter OLMo-2 checkpoint reduces forgetting by 31% after MetaMath post-training and 40% after 4-bit quantization. The optimizer shapes both what a model learns and what it keeps making it a fundamental and still underexplored lever for domain generalization.

Title: Causal representation learning and causal generative AI

Abstract: Causality is a fundamental notion in science, engineering, and even machine learning and AI. Uncovering the causal process underlying observed data naturally helps answer 'why' and 'what-if' questions, informs optimal decisions, and enables adaptive prediction. In many scenarios, observed variables, such as image pixels and questionnaire responses, are often reflections of the underlying hidden causal variables rather than being causal variables themselves. Causal representation learning aims to reveal the underlying hidden causal variables and their relations. In this talk, we show how the modularity property of causal systems makes it possible to recover the underlying causal representations from observational data with identifiability guarantees: under suitable assumptions, the learned representations align with the underlying causal process. We further demonstrate how identifiable causal representation learning can directly benefit generative AI, using image generation / editing, text generation, and extrapolated data generation as illustrative examples.

Title: Seeing Domain Generalization in Foundation Models: A Nearest-Neighbor View

Abstract: Foundation models and vision-language models have changed the landscape of domain generalization. Trained on large and diverse web-scale data, they often appear robust across datasets, styles, and visual domains. But are they truly domain-general, or have they learned a broader set of correlations? In this talk, I will look at domain generalization through the geometry of learned representations. Using nearest-neighbor structure, clustering, and visualization, I will discuss when foundation-model embeddings organize images by semantic content, and when they remain entangled with domain, style, background, prompt vocabulary, or transformation-specific cues. The central message of the talk is that domain generalization in the foundation-model era is not only a question of better objectives or larger pretraining data. It is also a question of inspection: how can we visualize, cluster, query, and audit learned representation spaces before deploying models in open-world settings? A nearest-neighbor view provides a practical lens for understanding both the strengths and the hidden failure modes of modern domain-general models.

Title: Context-Aware Deepfake Data Generation for Domain Generalization

Abstract: Current deepfake detectors do not perform well in out-of-distribution tests. They fail under real-world domain shifts as they overfit to low-level pixel artifacts rather than understanding the scene context. In this talk, I will introduce a generative framework to solve this bottleneck. By combining the scale of AV-Deepfakes-1M with the reasoning-driven pipeline of MultiFakeVerse, we generate context-aware fakes with diverse gestures, human-object interactions, and scene backgrounds. This synthetic domain expansion induces detectors to learn beyond local texture shortcuts, while improving their domain generalization on unseen OOD data.