Guaranteed Rank Minimization via Singular Value Projection

Minimizing the rank of a matrix subject to affine constraints is a fundamental problem with many important applications in machine learning and statistics. In this paper we propose a simple and fast algorithm SVP (Singular Value Projection) for rank minimization with affine constraints (ARMP) and show that SVP recovers the minimum rank solution for affine constraints that satisfy the "restricted isometry property" and show robustness of our method to noise. Our results improve upon a recent breakthrough by Recht, Fazel and Parillo (RFP07) and Lee and Bresler (LB09) in three significant ways: 1) our method (SVP) is significantly simpler to analyze and easier to implement, 2) we give recovery guarantees under strictly weaker isometry assumptions 3) we give geometric convergence guarantees for SVP even in presense of noise and, as demonstrated empirically, SVP is significantly faster on real-world and synthetic problems. In addition, we address the practically important problem of low-rank matrix completion (MCP), which can be seen as a special case of ARMP. We empirically demonstrate that our algorithm recovers low-rank incoherent matrices from an almost optimal number of uniformly sampled entries. We make partial progress towards proving exact recovery and provide some intuition for the strong performance of SVP applied to matrix completion by showing a more restricted isometry property. Our algorithm outperforms existing methods, such as those of \cite{RFP07,CR08,CT09,CCS08,KOM09,LB09}, for ARMP and the matrix-completion problem by an order of magnitude and is also significantly more robust to noise.Comment: An earlier version of this paper was submitted to NIPS-2009 on June 5, 200

A Divide-and-Conquer Solver for Kernel Support Vector Machines

The kernel support vector machine (SVM) is one of the most widely used classification methods; however, the amount of computation required becomes the bottleneck when facing millions of samples. In this paper, we propose and analyze a novel divide-and-conquer solver for kernel SVMs (DC-SVM). In the division step, we partition the kernel SVM problem into smaller subproblems by clustering the data, so that each subproblem can be solved independently and efficiently. We show theoretically that the support vectors identified by the subproblem solution are likely to be support vectors of the entire kernel SVM problem, provided that the problem is partitioned appropriately by kernel clustering. In the conquer step, the local solutions from the subproblems are used to initialize a global coordinate descent solver, which converges quickly as suggested by our analysis. By extending this idea, we develop a multilevel Divide-and-Conquer SVM algorithm with adaptive clustering and early prediction strategy, which outperforms state-of-the-art methods in terms of training speed, testing accuracy, and memory usage. As an example, on the covtype dataset with half-a-million samples, DC-SVM is 7 times faster than LIBSVM in obtaining the exact SVM solution (to within $10^{-6}$ relative error) which achieves 96.15% prediction accuracy. Moreover, with our proposed early prediction strategy, DC-SVM achieves about 96% accuracy in only 12 minutes, which is more than 100 times faster than LIBSVM

IT and the NHS: Investigating different perspectives of IT using soft systems methodology

The UK NHS National Programme for IT has been criticized for a lack of clinical engagement. This paper uses a soft systems methodology (SSM) analysis of a case study from the use of electronic systems within a National Health Service (NHS) Mental Health Trust in the United Kingdom (UK) to explore the legal and ethical implications of the failure to develop clinical systems which are fit for purpose. Soft systems methodology (SSM) was used as a theoretical model both to derive deeper insights into the survey data and suggest how communication between those producing information and those using it, could be improved. Multiple methods were employed which included a postal survey and participant interviews to triangulate the data The use of SSM reinforced the concept that the national IT programme is based on a 'hard' systems view and does not take local factors (which are related to 'soft systems' thinking) into account. The study found administrative staff to be a crucial link between clinicians and information departments and highlighted the need for a joint-up information strategy and integrated systems. The article concludes with a discussion of the legal and ethical implications of the findings and the lessons for the broader UK national programme. It argues that the failure to deliver systems that are fit for purpose is not value neutral but an ethical issue

Online Embedding Compression for Text Classification using Low Rank Matrix Factorization

Deep learning models have become state of the art for natural language processing (NLP) tasks, however deploying these models in production system poses significant memory constraints. Existing compression methods are either lossy or introduce significant latency. We propose a compression method that leverages low rank matrix factorization during training,to compress the word embedding layer which represents the size bottleneck for most NLP models. Our models are trained, compressed and then further re-trained on the downstream task to recover accuracy while maintaining the reduced size. Empirically, we show that the proposed method can achieve 90% compression with minimal impact in accuracy for sentence classification tasks, and outperforms alternative methods like fixed-point quantization or offline word embedding compression. We also analyze the inference time and storage space for our method through FLOP calculations, showing that we can compress DNN models by a configurable ratio and regain accuracy loss without introducing additional latency compared to fixed point quantization. Finally, we introduce a novel learning rate schedule, the Cyclically Annealed Learning Rate (CALR), which we empirically demonstrate to outperform other popular adaptive learning rate algorithms on a sentence classification benchmark.Comment: Accepted in Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019

Can Communist economies transform incrementally? China's experience

The authors try to answer important questions. How important is the phasing of political and economic liberalization and the active (versus passive) role of the state in reform? What lessons can be learned about comprehensive top-down reform as opposed to experimental bottom-up reforms; fast versus slow liberalization and opening up of the economy; the need to establish full private property rights at the beginning of reform; and reform's implications for welfare and distribution? Can China's excellent performance be linked toparticular reform measure, or does it reflect distinctive initial conditions or social or demographic factors? Is China's performance sustainable without more comprehensive transformation, or does it reflect transient gains that are substantially exhausted? Among the lessons China offers are the following. Partial reform can succeed in raising productivity in agriculture and industry; industrial productivity has grown very rapidly in the nonstate sector but also in state enterprises. A"big bang"is not economically necessary unless justified by the need to address macroeconomic imbalances. There may be virtue in a decentralized"bottom-up"approach to reform. Rapid privatization is not necessary for successful reform, but it is important to diversify ownership and encourage the entry of new firms. Small scale privatization and the liberalization of distribution and service sectors are likely to have the fastest payoff in the reform of property rights. China's rapid growth momentum and macroeconomic stability cannot be sustained without further reforms, including the reform of banking, taxation, and property rights.Environmental Economics&Policies,Economic Theory&Research,Banks&Banking Reform,Health Monitoring&Evaluation,Municipal Financial Management