Publications

Please also check our google scholar pages and arxiv for latest works

2024

• Less: Selecting influential data for targeted instruction tuning. ICML 2024 (Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, Danqi Chen)
• Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving. NeurIPS 2024 (Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Rezende, Yoshua Bengio, Michael Mozer, Sanjeev Arora)
• Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates. NeurIPS 2024 (Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, Sanjeev Arora)
• CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs. NeurIPS 2024 (Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora, Danqi Chen)
• Language Models as Science Tutors. ICML 2024 (Alexis Chevalier, Jiayi Geng, Alexander Wettig, Howard Chen, Sebastian Mizera, Toni Annala, Max Jameson Aragon, Arturo Rodríguez Fanlo, Simon Frieder, Simon Machado, Akshara Prabhakar, Ellie Thieu, Jiachen T Wang, Zirui Wang, Xindi Wu, Mengzhou Xia, Wenhan Jia, Jiatong Yu, Jun-Jie Zhu, Zhiyong Jason Ren, Sanjeev Arora, Danqi Chen)
• AI-Assisted Generation of Difficult Math Questions. MATH-AI Workshop at NeurIPS 2024 (Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, Anirudh Goyal)
• Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning. Compositional Learning Workshop at NeurIPS 2024 (Oral), FITML Workshop at NeurIPS 2024 (Simran Kaur, Simon Park, Anirudh Goyal, Sanjeev Arora)
• ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty. NeurIPS 2024 Datasets Track (Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, Sanjeev Arora)
• Can Models Learn Skill Composition from Examples?. NeurIPS 2024 (Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, Sanjeev Arora)
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. FITML Workshop at NeurIPS 2024, ATTRIB Workshop at NeurIPS 2024 (Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, Boris Hanin)
• Progressive Distillation Induces an Implicit Curriculum. M3L Workshop at NeurIPS 2024, Theoretical Foundations of Foundation Models Workshop at ICML 2024, MI Workshop at ICML 2024 (Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, Surbhi Goel)
• Preference Learning Algorithms Do Not Learn Preference Rankings. NeurIPS 2024 (Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, Kyunghyun Cho)
• Provable Unlearning in Topic Modeling and Downstream Tasks. Preprint (Stanley Wei, Sadhika Malladi, Sanjeev Arora, Amartya Sanyal)

2023

• Fine-Tuning Language Models with Just Forward Passes. NeurIPS 2023 (Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, Sanjeev Arora)
• A Theory for Emergence of Complex Skills in Language Models. Preprint (Sanjeev Arora, Anirudh Goyal)
• Task-Specific Skill Localization in Fine-tuned Language Models. ICML 2023 (Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, Sanjeev Arora)
• A Kernel-Based View of Language Model Fine-Tuning. ICML 2023 (Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, Sanjeev Arora)
• Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models. ICLR 2024 (Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, Sanjeev Arora)
• The Marginal Value of Momentum for Small Learning Rate SGD. ICLR 2024 (Runzhe Wang, Sadhika Malladi, Tianhao Wang, Kaifeng Lyu, Zhiyuan Li)
• Why (and When) does Local SGD Generalize Better than SGD?. ICLR 2023 (Xinran Gu, Kaifeng Lyu, Longbo Huang, Sanjeev Arora)
• Trainable Transformer in Transformer. ICML 2024 (Abhishek Panigrahi, Sadhika Malladi, Mengzhou Xia, Sanjeev Arora)
• Unlearning via Sparse Representations. Preprint (Vedant Shah, Frederik Träuble, Ashish Malik, Hugo Larochelle, Michael Mozer, Sanjeev Arora, Yoshua Bengio, Anirudh Goyal)

2022

• Understanding Contrastive Learning Requires Incorporating Inductive Biases. ICML 2022 (Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy)
• Understanding Gradient Descent on the Edge of Stability in Deep Learning. ICML 2022 (Sanjeev Arora, Zhiyuan Li, Abhishek Panigrahi)
• Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction. NeurIPS 2022 (Kaifeng Lyu, Zhiyuan Li, Sanjeev Arora)
• On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. NeurIPS 2022 (Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, Sanjeev Arora)
• Understanding Influence Functions and Datamodels via Harmonic Analysis. ICLR 2023 (Nikunj Saunshi, Arushi Gupta, Mark Braverman, Sanjeev Arora)
• Adaptive Gradient Methods with Local Guarantees. Preprint (Zhou Lu, Wenhan Xia, Sanjeev Arora, Elad Hazan)
• New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound. NeurIPS 2022 (Arushi Gupta, Nikunj Saunshi, Dingli Yu, Kaifeng Lyu, Sanjeev Arora)

2021

• Evaluating Gradient Inversion Attacks and Defenses in Federated Learning. NeurIPS 2021 (Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, Sanjeev Arora)
• What Happens after SGD Reaches Zero Loss?--A Mathematical Framework. NeurIPS 2021, (Zhiyuan Li, Tianhao Wang, Sanjeev Arora)
• Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias. NeurIPS 2021 (Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, Sanjeev Arora)
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs). NeurIPS 2021 (Zhiyuan Li, Sadhika Malladi, Sanjeev Arora)
• On Predicting Generalization using GANs. ICLR 2022 (spotlight) (Yi Zhang, Arushi Gupta, Nikunj Saunshi, Sanjeev Arora)
• Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data. Preprint (Sanjeev Arora, Yi Zhang)
• Opening the Black Box of Deep Learning: Some Lessons and Take-aways. SIGMETRICS '21 (Sanjeev Arora)
• Technical perspective: Why don't today's deep nets overfit to their training data?. Communications of the ACM (Sanjeev Arora)

2020

• InstaHide: Instance-hiding Schemes for Private Distributed Learning. ICML 2020 (Yangsibo Huang, Zhao Song, Kai Li, Sanjeev Arora)
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks. ICLR 2021 (Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora)
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate. NeurIPS 2020 (Zhiyuan Li, Kaifeng Lyu, Sanjeev Arora)
• Provable Representation Learning for Imitation Learning via Bi-level Optimization. ICML 2020 (Sanjeev Arora, Simon Du, Sham Kakade, Yuping Luo, Nikunj Saunshi)
• Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?. ICLR 2021 (Zhiyuan Li, Yi Zhang, Sanjeev Arora)
• TextHide: Tackling Data Privacy in Language Understanding Tasks. EMNLP 2020 (Findings) (Yangsibo Huang, Zhao Song, Danqi Chen, Kai Li, Sanjeev Arora)
• Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality. NeurIPS 2020 (Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, Sanjeev Arora)
• A Sample Complexity Separation between Non-Convex and Convex Meta-Learning. ICML 2020 (Nikunj Saunshi, Yi Zhang, Mikhail Khodak, Sanjeev Arora)
• Privacy-preserving Learning via Deep Net Pruning. Preprint (Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, Kai Li)
• Theory of deep learning. Manuscript (Raman Arora, Sanjeev Arora, Joan Bruna, Nadav Cohen, Simon Du, Rong Ge, Suriya Gunasekar, Chi Jin, Jason Lee, Tengyu Ma, Behnam Neyshabur, Zhao Song)
• The Quest for Mathematical Understanding of Deep Learning. 40th IARCS Annual Conference on Foundations of Software Technology (Sanjeev Arora)

2019

• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks. ICML 2019 (Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Ruosong Wang)
• On Exact Computation with an Infinitely Wide Neural Net. NeurIPS 2019 (Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang)
• Implicit Regularization in Deep Matrix Factorization. NeurIPS 2019 (Sanjeev Arora, Nadav Cohen, Wei Hu, Yuping Luo)
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning. ICML 2019 (Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, Hrishikesh Khandeparkar)
• An Exponential Learning Rate Schedule for Deep Learning. ICLR 2021 (Zhiyuan Li, Sanjeev Arora)
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks. ICLR 2020 (Sanjeev Arora, Simon S Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, Dingli Yu)
• Enhanced Convolutional Neural Tangent Kernels. Preprint (Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, Sanjeev Arora)
• Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets. NeurIPS 2019 (Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Sanjeev Arora, Rong Ge)
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. ICLR 2019 (Sanjeev Arora, Nadav Cohen, Noah Golowich, Wei Hu)
• A Simple Saliency Method That Passes the Sanity Checks. Preprint (Arushi Gupta, Sanjeev Arora)
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks. ICLR 2020 (spotlight) (Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, Dingli Yu)

back