More Like this

1. Spark Meets MPI: Towards High-Performance Communication Framework for Spark using MPI

   Al Attar, K.; Shafi, A.; Abduljabbar, M.; H. Subramoni, H.; Panda, DK (January 2022, 2022 IEEE International Conference on Cluster Computing)
2. Optimizing Near-Data Processing for Spark

   Rachuri, Sri Pramodh; Gantasala, Arun; Emanuel, Prajeeth; Gandhi, Anshul; Foley, Robert; Puhov, Peter; Gkountouvas, Theodoros; Lei, Hui (January 2022, Proceedings of the 42nd IEEE International Conference on Distributed Computing Systems (ICDCS '22))

   Resource disaggregation (RD) is an emerging paradigm for data center computing whereby resource-optimized servers are employed to minimize resource fragmentation and improve resource utilization. Apache Spark deployed under the RD paradigm employs a cluster of compute-optimized servers to run executors and a cluster of storage-optimized servers to host the data on HDFS. However, the network transfer from storage to compute cluster becomes a severe bottleneck for big data processing. Near-data processing (NDP) is a concept that aims to alleviate network load in such cases by offloading (or “pushing down”) some of the compute tasks to the storage cluster. Employing NDP for Spark under the RD paradigm is challenging because storage-optimized servers have limited computational resources and cannot host the entire Spark processing stack. Further, even if such a lightweight stack could be developed and deployed on the storage cluster, it is not entirely obvious which Spark queries would benefit from pushdown, and which tasks of a given query should be pushed down to storage. This paper presents the design and implementation of a near-data processing system for Spark, SparkNDP, that aims to address the aforementioned challenges. SparkNDP works by implementing novel NDP Spark capabilities on the storage cluster using a lightweight library of SQL operators and then developing an analytical model to help determine which Spark tasks should be pushed down to storage based on the current network and system state. Simulation and prototype implementation results show that SparkNDP can help reduce Spark query execution times when compared to both the default approach of not pushing down any tasks to storage and the outright NDP approach of pushing all tasks to storage. 

   more » « less
3. Arc-Phase Spark Plug Energy Deposition Characteristics Measured Using a Spark Plug Calorimeter Based on Differential Pressure Measurement

   https://doi.org/10.3390/en13143550  

   Kim, Kyeongmin; Hall, Matthew J.; Wilson, Preston S.; Matthews, Ronald D. (July 2020, Energies)

   null (Ed.)

   A spark plug calorimeter is introduced for quantifying the thermal energy delivered to unreactive gas surrounding the spark gap during spark ignition. Unlike other calorimeters, which measure the small pressure rise of the gas above the relatively high gauge pressure or relative to an internal reference, the present calorimeter measured the differential rise in pressure relative to the initial pressure in the calorimeter chamber. By using a large portion of the dynamic range of the chip-based pressure sensor, a high signal to noise ratio is possible; this can be advantageous, particularly for high initial pressures. Using this calorimeter, a parametric study was carried out, measuring the thermal energy deposition in the gas and the electrical-to-thermal energy conversion efficiency over a larger range of initial pressures than has been carried out previously (1–24 bar absolute at 298 K). The spark plug and inductive ignition circuit used gave arc-type rather than glow-type discharges. A standard resistor-type automotive spark plug was tested. The effects of spark gap distance (0.3–1.5 mm) and ignition dwell time (2–6 ms) were studied for an inductive-type ignition system. It was found that energy deposition to the gas (nitrogen) and the electrical-to-thermal energy conversion efficiency increased strongly with increasing gas pressure and spark gap distance. For the same ignition hardware and operating conditions, the thermal energy delivered to the gap varied from less than 1 mJ at 1 atm pressure and a gap distance of 0.3 mm to over 25 mJ at a pressure of 24 bar and a gap distance of 1.5 mm. For gas densities that might be representative of those in an engine at the time of ignition, the electrical-to-thermal energy conversion efficiencies ranged from approximately 3% at low pressures (4 bar) and small gap (0.3 mm) to as much as 40% at the highest pressure of 24 bar and with a gap of 1.5 mm. 

   more » « less
4. Toric symplectic geometry and full spark frames

   https://doi.org/10.1016/j.acha.2022.07.004  

   Needham, Tom; Shonkwiler, Clayton (November 2022, Applied and Computational Harmonic Analysis)
5. A Heterogeneity-Aware Task Scheduler for Spark

   Xu, Luna; Butt, Ali R.; Lim, Seung-Hwan; Kannan, Ramakrishnan (September 2018, Proceedings of the IEEE International Conference on Cluster Computing (Cluster))