HPML 2019

Performance Optimization on Model Synchronization in Parallel Stochastic Gradient Descent Based SVM

Vibhatha Abeykoon, Geoffrey Fox and Minje Kim

Indiana University

Authors' preprint on arXiv

slides

Understanding the bottlenecks in implementing stochastic gradient descent (SGD)-based distributed support vector machines (SVM) algorithm is important in training larger data sets. The communication time to do the model synchronization across the parallel processes is the main bottleneck that causes inefficiency in the training process. The model synchronization is directly affected by the mini-batch size of data processed before the global synchronization. In producing an efficient distributed model, the communication time in training model synchronization has to be as minimum as possible while retaining a high testing accuracy. The effect from model synchronization frequency over the convergence of the algorithm and accuracy of the generated model must be well understood to design an efficient distributed model. In this research, we identify the bottlenecks in model synchronization in parallel stochastic gradient descent (PSGD)-based SVM algorithm with respect to the training model synchronization frequency (MSF). Our research shows that by optimizing the MSF in the data sets that we used, a reduction of 98\% in communication time can be gained (16x - 24x speed up) with respect to high-frequency model synchronization. The training model optimization discussed in this paper guarantees a higher accuracy than the sequential algorithm along with faster convergence.

Distributed MCMC Inference in Dirichlet Process Mixture Models Using Julia

Or Dinari, Angel Yu, Oren Freifeld, John Fisher

Ben-Gurion University of the Negev, MIT

paper

slides

With the advent of large data sets, the need for general-purpose massively-parallel analysis tools become ever greater. In unsupervised learning, Bayesian nonparametric mixture models, exemplified by the Dirichlet-Process Mixture Model (DPMM), provide a principled Bayesian approach while adapting model complexity to the data. Despite their potential, however, DPMM's have yet to become a popular tool. This is partly due to the lack of friendly software tools that can handle large datasets efficiently. Here we show how, using Julia, one can achieve efficient and easily-modifiable implementation of distributed inference in DPMMs. Particularly, we show how a recent parallel MCMC inference algorithm -- originally implemented in C++ for a single multi-core machine -- can be distributed efficiently across multiple multi-core machines using a distributed-memory model. This leads to speedups, alleviates memory and storage limitations, and lets us learn DPMMs from significantly larger datasets and of higher dimensionality. It also turned out that even on a single machine our Julia implementation handles higher dimensions more gracefully (at least for Gaussians) than the original C++ implementation. Finally, we use our implementation to learn a model of image patches and apply the learned model for image denoising.

TensorFlow on state-of-the-art HPC clusters: a machine learning use case

Guillem Ramirez-Gargallo, Marta Garcia-Gasulla, Filippo Mantovani

Barcelona Supercomputing Center

paper

slides

The recent rapid growth of the data-flow programming paradigm enabled the development of specific architectures, e.g., for machine learning. The most known example is the Tensor Processing Unit (TPU) by Google. Standard data-centers, however, still can not foresee large partitions dedicated to machine learning specific architectures. Within data-centers, the High-Performance Computing (HPC) clusters are highly parallel machines targeting a broad class of compute-intensive workflows, as such they can be used for tackling machine learning challenges. On top of this, HPC architectures are rapidly changing, including accelerators and instruction sets other than the classical x86 CPUs. In this blurry scenario, identifying which are the best hardware/software configurations to efficiently support machine learning workloads on HPC clusters is not trivial. In this paper, we considered the workflow of TensorFlow for image recognition. We highlight the strong dependency of the performance in the training phase on the availability of arithmetic libraries optimized for the underlying architecture. Following the example of Intel leveraging the MKL libraries for improving the TensorFlow performance, we plugged the Arm Performance Libraries into TensorFlow and tested on an HPC cluster based on Marvell ThunderX2 CPUs. Also, we performed a scalability study on three state-of-the-art HPC clusters based on different CPU architectures, x86 Intel Skylake, Arm-v8 Marvell ThunderX2, and PowerPC IBM Power9.

Theoretical Scalability Analysis of Distributed Deep Convolutional Neural Networks

Adrián Castelló, Manuel F. Dolz, Enrique S. Quintana-Ortí, Jose Duato

Universitat Politècnica de València, Universitat Jaume I

paper

slides

We analyze the asynthotic performance of the training process of neural network (NN) on clusters in order to determine their scalability. For this purpose, i) we assume a data parallel implementation of the training algorithm, which replicates the model in the cluster nodes and distributes the batches among the cluster nodes; ii) we leverage the roofline model to inspect the performance at the node level, taking into account the floating-point unit (FPU) throughput and memory bandwidth; and iii) we consider distinct collective communication schemes that are optimal depending on the message size and underlying network interconnect topology. We then apply the resulting performance model to analyze the scalability of several well-known deep convolutional neural networks in different scenarios, varying the batch size, node floating-point throughput, node memory bandwidth, cluster dimension, and interconnect bandwidth.

An Evaluation Of Transfer Learning for Classifying Sales Engagement Emails at Large Scale

Yong Liu, Pavel Dmitriev, Yifei Huang, Andrew Brooks, Li Dong

Outreach.io

Authors' preprint on arXiv

slides

This paper conducts an empirical investigation to evaluate transfer learning for classifying sales engagement emails arising from digital sales engagement platforms. Given the complexity of content and context of sales engagement, lack of standardized large corpora and benchmarks, limited labeled examples and heterogenous context of intent, this real-world use case poses both a challenge and an opportunity for adopting a transfer learning approach. We propose an evaluation framework to assess a high performance transfer learning (HPTL) approach in three key areas in addition to commonly used accuracy metrics: 1) effective embeddings and pretrained language model usage, 2) minimum labeled samples requirement and 3) transfer learning implementation strategies. We use in-house sales engagement email samples as the experiment dataset, which includes over 3000 emails labeled as positive, objection, unsubscribe, or not-sure. We discuss our findings on evaluating BERT, ELMo, Flair and GloVe embeddings with both feature-based and fine-tuning approaches and their scalability on a GPU cluster with increasingly larger labeled samples. Our results show that fine-tuning of the BERT model outperforms with as few as 300 labeled samples, but underperforms with fewer than 300 labeled samples, relative to all the feature-based approaches using different embeddings.

Volumetric Segmentation via Neural Networks Improves Neutron Crystallography Data Analysis

Brendan Sullivan, Rick Archibald, Venu Vandavassi, Patricia Langan, Leighton Coates, Vickie Lynch

Oak Ridge National Laboratory

paper

slides

Crystallography is the powerhouse technique for molecular structure determination, with applications in fields ranging from energy storage to drug design. Accurate structure determination, however, relies partly on determining the precise locations and integrated intensities of Bragg peaks in the resulting data. Here, we describe a method for Bragg peak integration that is accomplished using neural networks. The network is based on a U-Net and identifies peaks in three-dimensional reciprocal space through segmentation, allowing prediction of the full 3D peak shape from noisy data that is commonly difficult to process. The procedure for generating appropriate training sets is detailed. Trained networks achieve Dice coefficients of 0.82 and mean IoUs of 0.69. Carrying out integration over entire datasets, it is demonstrated that integrating neural network-predicted peaks results in improved intensity statistics. Furthermore, using a second dataset, the possibility of transfer learning between datasets is shown. Given the ubiquity and growing complexity of crystallography, we anticipate integration by machine learning to play an increasingly important role across the physical sciences. These early results demonstrate the applicability of deep learning techniques for integrating crystallography data and suggest a possible role in the next generation of crystallography experiments.

A Performance Improvement Approach for Second-Order Optimization in Large Mini-batch Training

Hiroki Naganuma, Rio Yokota

Tokyo Institute of Technology

paper

slides

Classical learning theory states that when the number of parameters of the model is too large compared to the data, the model will overfit and the generalization performance deteriorates. However, it has been empirically shown that deep neural networks (DNN) can achieve high generalization capability by training with extremely large amount of data and model parameters, which exceeds the predictions of classical learning theory. One drawback of this is that training of DNN requires enormous calculation time. Therefore, it is necessary to reduce the training time through large scale parallelization. Straightforward data-parallelization of DNN degrades convergence and generalization. In the present work, we investigate the possibility of using second order methods to solve this generalization gap in large-batch training. This is motivated by our observation that each mini-batch becomes more statistically stable, and thus the effect of considering the curvature plays a more important role in large-batch training. We have also found that naively adapting the natural gradient method causes the generalization performance to deteriorate further due to the lack of regularization capability. We propose an improved second order method by smoothing the loss function, which allows second order methods to generalize as well as mini-batch SGD.