skip to main content


Title: Spark Meets MPI: Towards High-Performance Communication Framework for Spark using MPI
Award ID(s):
1818253 1854828 1931537 2007991 2018627
NSF-PAR ID:
10355057
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
2022 IEEE International Conference on Cluster Computing
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    A spark plug calorimeter is introduced for quantifying the thermal energy delivered to unreactive gas surrounding the spark gap during spark ignition. Unlike other calorimeters, which measure the small pressure rise of the gas above the relatively high gauge pressure or relative to an internal reference, the present calorimeter measured the differential rise in pressure relative to the initial pressure in the calorimeter chamber. By using a large portion of the dynamic range of the chip-based pressure sensor, a high signal to noise ratio is possible; this can be advantageous, particularly for high initial pressures. Using this calorimeter, a parametric study was carried out, measuring the thermal energy deposition in the gas and the electrical-to-thermal energy conversion efficiency over a larger range of initial pressures than has been carried out previously (1–24 bar absolute at 298 K). The spark plug and inductive ignition circuit used gave arc-type rather than glow-type discharges. A standard resistor-type automotive spark plug was tested. The effects of spark gap distance (0.3–1.5 mm) and ignition dwell time (2–6 ms) were studied for an inductive-type ignition system. It was found that energy deposition to the gas (nitrogen) and the electrical-to-thermal energy conversion efficiency increased strongly with increasing gas pressure and spark gap distance. For the same ignition hardware and operating conditions, the thermal energy delivered to the gap varied from less than 1 mJ at 1 atm pressure and a gap distance of 0.3 mm to over 25 mJ at a pressure of 24 bar and a gap distance of 1.5 mm. For gas densities that might be representative of those in an engine at the time of ignition, the electrical-to-thermal energy conversion efficiencies ranged from approximately 3% at low pressures (4 bar) and small gap (0.3 mm) to as much as 40% at the highest pressure of 24 bar and with a gap of 1.5 mm. 
    more » « less
  2. Transparently checkpointing MPI for fault tolerance and load balancing is a long-standing problem in HPC. The problem has been complicated by the need to provide checkpoint-restart services for all combinations of an MPI implementation over all network interconnects. This work presents MANA (MPI-Agnostic Network-Agnostic transparent checkpointing), a single code base which supports all MPI implementation and interconnect combinations. The agnostic properties imply that one can checkpoint an MPI application under one MPI implementation and perhaps over TCP, and then restart under a second MPI implementation over InfiniBand on a cluster with a different number of CPU cores per node. This technique is based on a novel "split-process" approach, which enables two separate programs to co-exist within a single process with a single address space. This work overcomes the limitations of the two most widely adopted transparent checkpointing solutions, BLCR and DMTCP/InfiniBand, which require separate modifications to each MPI implementation and/or underlying network API. The runtime overhead is found to be insignificant both for checkpoint-restart within a single host, and when comparing a local MPI computation that was migrated to a remote cluster against an ordinary MPI computation running natively on that same remote cluster. 
    more » « less
  3. The MPI standard has long included one-sided communication abstractions through the MPI Remote Memory Access (RMA) interface. Unfortunately, the MPI RMA chapter in the 4.0 version of the MPI standard still contains both well-known and lesser known short-comings for both implementations and users, which lead to potentially non-optimal usage patterns. In this paper, we identify a set of issues and propose ways for applications to better express anticipated usage of RMA routines, allowing the MPI implementation to better adapt to the application's needs. In order to increase the flexibility of the RMA interface, we add the capability to duplicate windows, allowing access to the same resources encapsulated by a window using different configurations. In the same vein, we introduce the concept of MPI memory handles, meant to provide life-time guarantees on memory attached to dynamic windows, removing the overhead currently present in using dynamically exposed memory. We will show that our extensions provide improved accumulate latencies, reduced overheads for multi-threaded flushes, and allow for zero overhead dynamic memory window usage. 
    more » « less