A Dirty Model for Multiple Sparse Regression | Center for Big Data Analytics

Abstract: The task of sparse linear regression consists of finding an unknown sparse vector from linear measurements. Solving this task even under “high-dimensional” settings, where the number of samples is fewer than the number of variables, is now known to be possible via methods such as the LASSO. We consider the multiple sparse linear regression problem, where the task consists of recovering several related sparse vectors at once. A simple approach to this task would involve solving independent sparse linear regression problems, but a natural question is whether one can reduce the overall number of samples required by leveraging partial sharing of the support sets, or nonzero patterns, of the signal vectors. A line of recent research has studied the use of l1/lq norm block-regularizations with q > 1 for such problems. However, depending on the level of sharing, these could actually perform worse in sample complexity when compared to solving each problem independently. We present a new “adaptive” method for multiple sparse linear regression that can leverage support and parameter overlap when it exists, but not pay a penalty when it does not. We show how to achieve this using a very simple idea: decompose the parameters into two components and regularize these differently. We show, theoretically and empirically, that our method strictly and noticeably outperforms both l1 or l1/lq methods, over the entire range of possible overlaps (except at boundary cases, where we match the best method), even under high-dimensional scaling.

Download: pdf

Citation

A Dirty Model for Multiple Sparse Regression (pdf, software)
A. Jalali, P. Ravikumar, S. Sanghavi.
IEEE Transactions on Information Theory 59(12), pp. 7947-7968, 2013.

Bibtex:
@article{jalali2013adirtymo, author = "Ali Jalali AND Pradeep Ravikumar AND Sujay Sanghavi", title = "A Dirty Model for Multiple Sparse Regression", journal = "IEEE Transactions on Information Theory", page = "7947–7968", volume = "59", issue = "12", number = "0", year = "2013", abstract = "The task of sparse linear regression consists of finding an unknown sparse vector from linear measurements. Solving this task even under “high-dimensional” settings, where the number of samples is fewer than the number of variables, is now known to be possible via methods such as the LASSO. We consider the multiple sparse linear regression problem, where the task consists of recovering several related sparse vectors at once. A simple approach to this task would involve solving independent sparse linear regression problems, but a natural question is whether one can reduce the overall number of samples required by leveraging partial sharing of the support sets, or nonzero patterns, of the signal vectors. A line of recent research has studied the use of l1/lq norm block-regularizations with q > 1 for such problems. However, depending on the level of sharing, these could actually perform worse in sample complexity when compared to solving each problem independently. We present a new “adaptive” method for multiple sparse linear regression that can leverage support and parameter overlap when it exists, but not pay a penalty when it does not. We show how to achieve this using a very simple idea: decompose the parameters into two components and regularize these differently. We show, theoretically and empirically, that our method strictly and noticeably outperforms both l1 or l1/lq methods, over the entire range of possible overlaps (except at boundary cases, where we match the best method), even under high-dimensional scaling." }