Search for: All records

We propose a novel policy gradient method for multi-agent reinforcement learning, which leverages two different variance-reduction techniques and does not require large batches over iterations. Specifically, we propose a momentum-based decentralized policy gradient tracking (MDPGT) where a new momentum-based variance reduction technique is used to approximate the local policy gradient surrogate with importance sampling, and an intermediate parameter is adopted to track two consecutive policy gradient surrogates. MDPGT provably achieves the best available sample complexity of O(N -1 e -3) for converging to an e-stationary point of the global average of N local performance functions (possibly nonconcave). This outperforms the state-of-the-art sample complexity in decentralized model-free reinforcement learning and when initialized with a single trajectory, the sample complexity matches those obtained by the existing decentralized policy gradient methods. We further validate the theoretical claim for the Gaussian policy function. When the required error tolerance e is small enough, MDPGT leads to a linear speed up, which has been previously established in decentralized stochastic optimization, but not for reinforcement learning. Lastly, we provide empirical results on a multi-agent reinforcement learning benchmark environment to support our theoretical findings. 

In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality . In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus -based distributed stochastic gradient descent (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus -based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disagreement (each agent converges to individual model parameters). We analytically establish convergence of the proposed algorithms for strongly convex and nonconvex objective functions; we also analyze the momentum variants of the algorithms for the strongly convex case. We support our algorithms via numerical experiments, and demonstrate significant improvements over existing methods for collaborative deep learning. 

Computer-aided Design for Manufacturing (DFM) systems play an essential role in reducing the time taken for product development by providing manufacturability feedback to the designer before the manufacturing phase. Traditionally, DFM rules are hand-crafted and used to accelerate the engineering product design process by integrating manufacturability analysis during design. Recently, the feasibility of using a machine learning-based DFM tool in intelligently applying the DFM rules have been studied. These tools use a voxelized representation of the design and then use a 3D-Convolutional Neural Network (3D-CNN), to provide manufacturability feedback. Although these frameworks work effectively, there are some limitations to the voxelized representation of the design. In this paper, we introduce a new representation of the computer-aided design (CAD) model using orthogonal distance fields (ODF). We provide a GPU-accelerated algorithm to convert standard boundary representation (B-rep) CAD models into ODF representation. Using the ODF representation, we build a machine learning framework, similar to earlier approaches, to create a machine learning-based DFM system to provide manufacturability feedback. As proof of concept, we apply this framework to assess the manufacturability of drilled holes. The framework has an accuracy of more than 84% correctly classifying the manufacturable and non-manufacturable models using the new representation.