NeurIPS 2019 Outstanding New Directions Paper Award

No Ads!
Continue to support the channel:
NeurIPS 2019 Outstanding New Directions Paper Award: Uniform convergence may be unable to explain generalization in deep learning
Vaishnavh Nagarajan, J. Zico Kolter

Video is reposted for education purpose.
-----------------------------------------------------------------------------------------
Subscribe ⇢
-----------------------------------------------------------------------------------------
Paper:
Slides:

Abstract:
We cast doubt on the power of uniform convergence-based generalization bounds to provide a complete picture of why overparameterized deep networks generalize well. While it is well-known that many existing bounds are numerically large, through a variety of experiments, we first bring to light another crucial and more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the dataset size. Guided by our observations, we then present examples of overparameterized linear classifiers and neural networks trained by stochastic gradient descent (SGD) where uniform convergence provably cannot `explain generalization,’ even if we take into account implicit regularization {\em to the fullest extent possible}. More precisely, even if we consider only the set of classifiers output by SGD that have test errors less than some small ϵ, applying (two-sided) uniform convergence on this set of classifiers yields a generalization guarantee that is larger than 1−ϵ and is therefore nearly vacuous.

machine learning,deep learning,ai,research,conference,NeurIPS2019,neurips 2019,nips,nips2019,artificial intelligence,science,tech,optimization,active learning,python,dimensionality reduction machine learning,cnn,rnn,genetic algorithm in machine learning,perceptron in machine learning,artificial neural network,bayesian machine learning,best paper,outstanding,award,