TY - JOUR A1 - Blanchard, Gilles A1 - Mücke, Nicole T1 - Kernel regression, minimax rates and effective dimensionality BT - beyond the regular case JF - Analysis and applications N2 - We investigate if kernel regularization methods can achieve minimax convergence rates over a source condition regularity assumption for the target function. These questions have been considered in past literature, but only under specific assumptions about the decay, typically polynomial, of the spectrum of the the kernel mapping covariance operator. In the perspective of distribution-free results, we investigate this issue under much weaker assumption on the eigenvalue decay, allowing for more complex behavior that can reflect different structure of the data at different scales. KW - Kernel regression KW - minimax optimality KW - eigenvalue decay Y1 - 2020 U6 - https://doi.org/10.1142/S0219530519500258 SN - 0219-5305 SN - 1793-6861 VL - 18 IS - 4 SP - 683 EP - 696 PB - World Scientific CY - New Jersey ER - TY - JOUR A1 - Blanchard, Gilles A1 - Zadorozhnyi, Oleksandr T1 - Concentration of weakly dependent Banach-valued sums and applications to statistical learning methods JF - Bernoulli : official journal of the Bernoulli Society for Mathematical Statistics and Probability N2 - We obtain a Bernstein-type inequality for sums of Banach-valued random variables satisfying a weak dependence assumption of general type and under certain smoothness assumptions of the underlying Banach norm. We use this inequality in order to investigate in the asymptotical regime the error upper bounds for the broad family of spectral regularization methods for reproducing kernel decision rules, when trained on a sample coming from a tau-mixing process. KW - Banach-valued process KW - Bernstein inequality KW - concentration KW - spectral regularization KW - weak dependence Y1 - 2019 U6 - https://doi.org/10.3150/18-BEJ1095 SN - 1350-7265 SN - 1573-9759 VL - 25 IS - 4B SP - 3421 EP - 3458 PB - International Statistical Institute CY - Voorburg ER - TY - JOUR A1 - Katz-Samuels, Julian A1 - Blanchard, Gilles A1 - Scott, Clayton T1 - Decontamination of Mutual Contamination Models JF - Journal of machine learning research N2 - Many machine learning problems can be characterized by mutual contamination models. In these problems, one observes several random samples from different convex combinations of a set of unknown base distributions and the goal is to infer these base distributions. This paper considers the general setting where the base distributions are defined on arbitrary probability spaces. We examine three popular machine learning problems that arise in this general setting: multiclass classification with label noise, demixing of mixed membership models, and classification with partial labels. In each case, we give sufficient conditions for identifiability and present algorithms for the infinite and finite sample settings, with associated performance guarantees. KW - multiclass classification with label noise KW - classification with partial labels KW - mixed membership models KW - topic modeling KW - mutual contamination models Y1 - 2019 UR - http://arxiv.org/abs/1710.01167 SN - 1532-4435 VL - 20 PB - Microtome Publishing CY - Cambridge, Mass. ER - TY - JOUR A1 - Blanchard, Gilles A1 - Mücke, Nicole T1 - Optimal rates for regularization of statistical inverse learning problems JF - Foundations of Computational Mathematics N2 - We consider a statistical inverse learning (also called inverse regression) problem, where we observe the image of a function f through a linear operator A at i.i.d. random design points X-i , superposed with an additive noise. The distribution of the design points is unknown and can be very general. We analyze simultaneously the direct (estimation of Af) and the inverse (estimation of f) learning problems. In this general framework, we obtain strong and weak minimax optimal rates of convergence (as the number of observations n grows large) for a large class of spectral regularization methods over regularity classes defined through appropriate source conditions. This improves on or completes previous results obtained in related settings. The optimality of the obtained rates is shown not only in the exponent in n but also in the explicit dependency of the constant factor in the variance of the noise and the radius of the source condition set. KW - Reproducing kernel Hilbert space KW - Spectral regularization KW - Inverse problem KW - Statistical learning KW - Minimax convergence rates Y1 - 2018 U6 - https://doi.org/10.1007/s10208-017-9359-7 SN - 1615-3375 SN - 1615-3383 VL - 18 IS - 4 SP - 971 EP - 1013 PB - Springer CY - New York ER - TY - GEN A1 - Blanchard, Gilles A1 - Scott, Clayton T1 - Corrigendum to: Classification with asymmetric label noise BT - Consistency and maximal denoising T2 - Electronic journal of statistics N2 - We point out a flaw in Lemma 15 of [1]. We also indicate how the main results of that section are still valid using a modified argument. Y1 - 2018 U6 - https://doi.org/10.1214/18-EJS1422 SN - 1935-7524 VL - 12 IS - 1 SP - 1779 EP - 1781 PB - Institute of Mathematical Statistics CY - Cleveland ER - TY - JOUR A1 - Mücke, Nicole A1 - Blanchard, Gilles T1 - Parallelizing spectrally regularized kernel algorithms JF - Journal of machine learning research N2 - We consider a distributed learning approach in supervised learning for a large class of spectral regularization methods in an reproducing kernel Hilbert space (RKHS) framework. The data set of size n is partitioned into m = O (n(alpha)), alpha < 1/2, disjoint subsamples. On each subsample, some spectral regularization method (belonging to a large class, including in particular Kernel Ridge Regression, L-2-boosting and spectral cut-off) is applied. The regression function f is then estimated via simple averaging, leading to a substantial reduction in computation time. We show that minimax optimal rates of convergence are preserved if m grows sufficiently slowly (corresponding to an upper bound for alpha) as n -> infinity, depending on the smoothness assumptions on f and the intrinsic dimensionality. In spirit, the analysis relies on a classical bias/stochastic error analysis. KW - Distributed Learning KW - Spectral Regularization KW - Minimax Optimality Y1 - 2018 SN - 1532-4435 VL - 19 PB - Microtome Publishing CY - Cambridge, Mass. ER - TY - JOUR A1 - Blanchard, Gilles A1 - Hoffmann, Marc A1 - Reiss, Markus T1 - Optimal adaptation for early stopping in statistical inverse problems JF - SIAM/ASA Journal on Uncertainty Quantification N2 - For linear inverse problems Y = A mu + zeta, it is classical to recover the unknown signal mu by iterative regularization methods ((mu) over cap,(m) = 0,1, . . .) and halt at a data-dependent iteration tau using some stopping rule, typically based on a discrepancy principle, so that the weak (or prediction) squared-error parallel to A((mu) over cap (()(tau)) - mu)parallel to(2) is controlled. In the context of statistical estimation with stochastic noise zeta, we study oracle adaptation (that is, compared to the best possible stopping iteration) in strong squared- error E[parallel to((mu) over cap (()(tau)) - mu)parallel to(2)]. For a residual-based stopping rule oracle adaptation bounds are established for general spectral regularization methods. The proofs use bias and variance transfer techniques from weak prediction error to strong L-2-error, as well as convexity arguments and concentration bounds for the stochastic part. Adaptive early stopping for the Landweber method is studied in further detail and illustrated numerically. KW - linear inverse problems KW - early stopping KW - discrepancy principle KW - adaptive estimation KW - oracle inequality KW - Landweber iteration Y1 - 2018 U6 - https://doi.org/10.1137/17M1154096 SN - 2166-2525 VL - 6 IS - 3 SP - 1043 EP - 1075 PB - Society for Industrial and Applied Mathematics CY - Philadelphia ER - TY - JOUR A1 - Blanchard, Gilles A1 - Hoffmann, Marc A1 - Reiss, Markus T1 - Early stopping for statistical inverse problems via truncated SVD estimation JF - Electronic journal of statistics N2 - We consider truncated SVD (or spectral cut-off, projection) estimators for a prototypical statistical inverse problem in dimension D. Since calculating the singular value decomposition (SVD) only for the largest singular values is much less costly than the full SVD, our aim is to select a data-driven truncation level (m) over cap is an element of {1, . . . , D} only based on the knowledge of the first (m) over cap singular values and vectors. We analyse in detail whether sequential early stopping rules of this type can preserve statistical optimality. Information-constrained lower bounds and matching upper bounds for a residual based stopping rule are provided, which give a clear picture in which situation optimal sequential adaptation is feasible. Finally, a hybrid two-step approach is proposed which allows for classical oracle inequalities while considerably reducing numerical complexity. KW - Linear inverse problems KW - truncated SVD KW - spectral cut-off KW - early stopping KW - discrepancy principle KW - adaptive estimation KW - oracle inequalities Y1 - 2018 U6 - https://doi.org/10.1214/18-EJS1482 SN - 1935-7524 VL - 12 IS - 2 SP - 3204 EP - 3231 PB - Institute of Mathematical Statistics CY - Cleveland ER - TY - JOUR A1 - Blanchard, Gilles A1 - Carpentier, Alexandra A1 - Gutzeit, Maurilio T1 - Minimax Euclidean separation rates for testing convex hypotheses in R-d JF - Electronic journal of statistics N2 - We consider composite-composite testing problems for the expectation in the Gaussian sequence model where the null hypothesis corresponds to a closed convex subset C of R-d. We adopt a minimax point of view and our primary objective is to describe the smallest Euclidean distance between the null and alternative hypotheses such that there is a test with small total error probability. In particular, we focus on the dependence of this distance on the dimension d and variance 1/n giving rise to the minimax separation rate. In this paper we discuss lower and upper bounds on this rate for different smooth and non-smooth choices for C. KW - Minimax hypothesis testing KW - Gaussian sequence model KW - nonasymptotic minimax separation rate Y1 - 2018 U6 - https://doi.org/10.1214/18-EJS1472 SN - 1935-7524 VL - 12 IS - 2 SP - 3713 EP - 3735 PB - Institute of Mathematical Statistics CY - Cleveland ER - TY - JOUR A1 - Bachoc, Francois A1 - Blanchard, Gilles A1 - Neuvial, Pierre T1 - On the post selection inference constant under restricted isometry properties JF - Electronic journal of statistics N2 - Uniformly valid confidence intervals post model selection in regression can be constructed based on Post-Selection Inference (PoSI) constants. PoSI constants are minimal for orthogonal design matrices, and can be upper bounded in function of the sparsity of the set of models under consideration, for generic design matrices. In order to improve on these generic sparse upper bounds, we consider design matrices satisfying a Restricted Isometry Property (RIP) condition. We provide a new upper bound on the PoSI constant in this setting. This upper bound is an explicit function of the RIP constant of the design matrix, thereby giving an interpolation between the orthogonal setting and the generic sparse setting. We show that this upper bound is asymptotically optimal in many settings by constructing a matching lower bound. KW - Inference post model-selection KW - confidence intervals KW - PoSI constants KW - linear regression KW - high-dimensional inference KW - sparsity KW - restricted isometry property Y1 - 2018 U6 - https://doi.org/10.1214/18-EJS1490 SN - 1935-7524 VL - 12 IS - 2 SP - 3736 EP - 3757 PB - Institute of Mathematical Statistics CY - Cleveland ER -