Research

We publish fundamental machine learning research on methods to help people improve the quality of their datasets and models for messy, real-world applications.

Featured publications by our team

• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

  Curtis Northcutt, Anish Athalye, and Jonas Mueller

  35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks

  Demo, Code, Blog Post, Press [1, 2, 3, 4, 5, 6, 7, 8]

• Confident Learning: Estimating Uncertainty in Dataset Labels

  Curtis Northcutt, Lu Jiang, and Isaac Chuang

  Journal of Artificial Intelligence Research (JAIR), Vol. 70 (2021)

  Code, Blog Post

• ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

  Hui Wen Goh and Jonas Mueller

  ICLR Workshop on Trustworthy ML, 2023

  Code (to run method), Code (to reproduce results), Blog Post, Press

• CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators

  Hui Wen Goh, Ulyana Tkachenko, and Jonas Mueller

  NeurIPS 2022 Human in the Loop Learning Workshop

  Code (to run method), Code (to reproduce results), Blog Post

• Detecting Errors in Numerical Data via any Regression Model

  Hang Zhou, Jonas Mueller, Mayank Kumar, Jane-Ling Wang, and Jing Lei

  ICML Workshop on Data-centric Machine Learning, 2023

  Code (to reproduce results)

• ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data

  Ulyana Tkachenko, Aditya Thyagarajan, and Jonas Mueller

  ICML Workshop on Data-centric Machine Learning, 2023

• Detecting Dataset Drift and Non-IID Sampling via k-Nearest Neighbors

  Jesse Cummings, Elías Snorrason, and Jonas Mueller

  ICML Workshop on Data-centric Machine Learning, 2023

  Blog Post, Code

• Estimating label quality and errors in semantic segmentation data via any model

  Vedang Lad and Jonas Mueller

  ICML Workshop on Data-centric Machine Learning, 2023

• DataPerf: Benchmarks for Data-Centric AI Development

  Mazumder et al.

  ICML DataPerf Workshop, 2022

  Benchmarks, Blog Post (Google), Blog Post (MLCommons)

• Detecting Label Errors in Token Classification Data

  Wei-Chen (Eric) Wang and Jonas Mueller

  NeurIPS 2022 Workshop on Interactive Learning for Natural Language Processing (InterNLP)

  Code (to run method), Code (to reproduce results), Blog Post

• Back to the Basics: Revisiting Out-of-Distribution Detection Baselines

  Johnson Kuan and Jonas Mueller

  ICML Workshop on Principles of Distribution Shift, 2022

  Code (to run method), Code (to reproduce results), Blog Post

• Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels

  Curtis Northcutt, Tailin Wu, and Isaac Chuang

  33rd Conference on Uncertainty in Artificial Intelligence (UAI 2017)

Additional publications by our team

• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

  Anish Athalye, Nicholas Carlini, and David Wagner

  35th International Conference on Machine Learning (ICML 2018)

  Best Paper Award, Code

• EgoCom: A Multi-person Multi-modal Egocentric Communications Dataset

  Curtis Northcutt, Zha Shengxin, Steven Lovegrove, and Richard Newcombe

  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

  Code

• Adaptive Interest for Emphatic Reinforcement Learning

  Martin Klissarov, Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Taesup Kim, and Alex Smola

  Advances in Neural Information Processing Systems (NeurIPS 2022)

• Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features

  Jiuhai Chen, Jonas Mueller, Vassilis Ioannidis, Soji Adeshina, Yangkun Wang, Tom Goldstein, and David Wipf

  International Conference on Learning Representations (ICLR 2022)

• Deep learning for the partially linear Cox model

  Qixian Zhong, Jonas Mueller, and Jane-Ling Wang

  Annals of Statistics, 2022

• Flexible Model Aggregation for Quantile Regression

  Rasool Fakoor, Taesup Kim, Jonas Mueller, Alexander Smola, and Ryan Tibshirani

  Journal of Machine Learning Research, 2023

• Benchmarking Multimodal AutoML for Tabular Data with Text Fields

  Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola

  35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and Benchmarks

  Code

• Overinterpretation reveals image classification model pathologies

  Brandon Carter, Siddhartha Jain, Jonas Mueller, and David Gifford

  Advances in Neural Information Processing Systems (NeurIPS 2021)

  Code

• Deep Extended Hazard Models for Survival Analysis

  Qixian Zhong, Jonas Mueller, and Jane-Ling Wang

  Advances in Neural Information Processing Systems (NeurIPS 2021)

• Continuous Doubly Constrained Batch Reinforcement Learning

  Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Pratik Chaudhari, and Alex Smola

  Advances in Neural Information Processing Systems (NeurIPS 2021)

  Code

• Verifying Hardware Security Modules with Information-Preserving Refinement

  Anish Athalye, M. Frans Kaashoek, and Nickolai Zeldovich

  16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2022)

  Code

• Notary: A Device for Secure Transaction Approval

  Anish Athalye, Adam Belay, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich

  27th ACM Symposium on Operating Systems Principles (SOSP 2019)

  Code

• Synthesizing Robust Adversarial Examples

  Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok

  35th International Conference on Machine Learning (ICML 2018)

  Blog Post

• Black-box Adversarial Attacks with Limited Queries and Information

  Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin

  35th International Conference on Machine Learning (ICML 2018)

  Blog Post, Code

• Identifying Incorrect Annotations in Multi-Label Classification Data

  Aditya Thyagarajan, Elías Snorrason, Curtis Northcutt, and Jonas Mueller

  ICLR Workshop on Trustworthy ML, 2023

  Code (to run method), Code (to reproduce results), Blog Post

• How to Cope with Gradual Data Drift?

  Rasool Fakoor, Jonas Mueller, Zachary Lipton, Pratik Chaudhari, and Alex Smola

  ICML Workshop on Data-centric Machine Learning Research, 2023

• Model-Agnostic Label Quality Scoring to Detect Real-World Label Errors

  Johnson Kuan and Jonas Mueller

  ICML DataPerf Workshop, 2022

  Code