Learning Active Basis Models by EM-Type Algorithms

EM algorithm is a convenient tool for maximum likelihood model fitting when the data are incomplete or when there are latent variables or hidden states. In this review article we explain that EM algorithm is a natural computational scheme for learning image templates of object categories where the learning is not fully supervised. We represent an image template by an active basis model, which is a linear composition of a selected set of localized, elongated and oriented wavelet elements that are allowed to slightly perturb their locations and orientations to account for the deformations of object shapes. The model can be easily learned when the objects in the training images are of the same pose, and appear at the same location and scale. This is often called supervised learning. In the situation where the objects may appear at different unknown locations, orientations and scales in the training images, we have to incorporate the unknown locations, orientations and scales as latent variables into the image generation process, and learn the template by EM-type algorithms. The E-step imputes the unknown locations, orientations and scales based on the currently learned template. This step can be considered self-supervision, which involves using the current template to recognize the objects in the training images. The M-step then relearns the template based on the imputed locations, orientations and scales, and this is essentially the same as supervised learning. So the EM learning process iterates between recognition and supervised learning. We illustrate this scheme by several experiments.Comment: Published in at http://dx.doi.org/10.1214/09-STS281 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Serverless Federated Learning with flwr-serverless

Federated learning is becoming increasingly relevant and popular as we witness a surge in data collection and storage of personally identifiable information. Alongside these developments there have been many proposals from governments around the world to provide more protections for individuals' data and a heightened interest in data privacy measures. As deep learning continues to become more relevant in new and existing domains, it is vital to develop strategies like federated learning that can effectively train data from different sources, such as edge devices, without compromising security and privacy. Recently, the Flower (\texttt{Flwr}) Python package was introduced to provide a scalable, flexible, and easy-to-use framework for implementing federated learning. However, to date, Flower is only able to run synchronous federated learning which can be costly and time-consuming to run because the process is bottlenecked by client-side training jobs that are slow or fragile. Here, we introduce \texttt{flwr-serverless}, a wrapper around the Flower package that extends its functionality to allow for both synchronous and asynchronous federated learning with minimal modification to Flower's design paradigm. Furthermore, our approach to federated learning allows the process to run without a central server, which increases the domains of application and accessibility of its use. This paper presents the design details and usage of this approach through a series of experiments that were conducted using public datasets. Overall, we believe that our approach decreases the time and cost to run federated training and provides an easier way to implement and experiment with federated learning systems.Comment: Technical report for an open source machine learning python packag

Unsupervised Learning of Stochastic AND-OR Templates for Object Modeling

This paper presents a framework for unsupervised learning of a hierarchical generative image model called AND-OR Template (AOT) for visual objects. The AOT includes: (1) hierarchical composition as “AND ” nodes, (2) deformation of parts as continuous “OR ” nodes, and (3) multiple ways of composition as discrete “OR ” nodes. These AND/OR nodes form the hierarchical visual dictionary. We show that both the structure and parameters of the AOT model can be learned in an unsupervised way from example images using an information projection principle. The learning algorithm consists two steps: i) a recursive Block-Pursuit procedure to learn the hierarchical dictionary of primitives, parts and objects, which form leaf nodes, AND nodes and structural OR nodes and ii) a Graph-Compression operation to minimize model structure for better generalizability, which produce additional OR nodes across the compositional hierarchy. We investigate the conditions under which the learning algorithm can identify, (i.e. recover) an underlying AOT that generates the data, and evaluate the performance of our learning algorithm through both artificial and real examples. 1