More Like this

1. Shuffle Scheduling for MapReduce Jobs Based on Periodic Network Status

   https://doi.org/10.1109/TNET.2020.2993945  

   Fan, Yuqi; Liu, Wenlong; Guo, Dan; Wu, Weili; Du, Dingzhu (August 2020, IEEE/ACM Transactions on Networking)

   MapReduce jobs need to shuffle a large amount of data over the network between mapper and reducer nodes. The shuffle time accounts for a big part of the total running time of the MapReduce jobs. Therefore, optimizing the makespan of shuffle phase can greatly improve the performance of MapReduce jobs. A large fraction of production jobs in data centers are recurring with predictable characteristics, and the recurring jobs split the network into periodic busy and idle time slots, which allows us to better schedule the shuffle data in order to reduce the makespan of shuffle phase with the future predictable network status available. In this paper, we formulate the shuffle scheduling problem with the aim to minimize the makespan of MapReduce shuffle phase by leveraging the predictable periodic network status. We then propose a simple yet effective network-aware shuffle scheduling algorithm (NAS) to reduce the number of idle time slots required to transfer the shuffle data so as to reduce the shuffle makespan. We also prove that the proposed algorithm NAS is a 32 -approximation algorithm to the shuffle scheduling problem when all the future idle time slots have the same duration. We finally conduct experiments through simulations. Experimental results demonstrate the proposed algorithm can effectively reduce the makespan of MapReduce shuffle phase and increase network utilization. 

   more » « less
2. On the Rényi Differential Privacy of the Shuffle Model

   https://doi.org/10.1145/3460120.3484794  

   Girgis, Antonious M.; Data, Deepesh; Diggavi, Suhas; Suresh, Ananda Theertha; Kairouz, Peter (November 2021, ACM Symposium on Computer and Communication Security (CCS))

   The central question studied in this paper is Rényi Differential Privacy (RDP) guarantees for general discrete local randomizers in the shuffle privacy model. In the shuffle model, each of the 𝑛 clients randomizes its response using a local differentially private (LDP) mechanism and the untrusted server only receives a random permutation (shuffle) of the client responses without association to each client. The principal result in this paper is the first direct RDP bounds for general discrete local randomization in the shuffle pri- vacy model, and we develop new analysis techniques for deriving our results which could be of independent interest. In applications, such an RDP guarantee is most useful when we use it for composing several private interactions. We numerically demonstrate that, for important regimes, with composition our bound yields an improve- ment in privacy guarantee by a factor of 8× over the state-of-the-art approximate Differential Privacy (DP) guarantee (with standard composition) for shuffle models. Moreover, combining with Pois- son subsampling, our result leads to at least 10× improvement over subsampled approximate DP with standard composition. 

   more » « less
3. PSACS: Highly-Parallel Shuffle Accelerator on Computational Storage

   https://doi.org/10.1109/ICCD53106.2021.00080  

   Zou, Chen; Zhang, Hui; Chien, Andrew A.; Seok Ki, Yang (October 2021, 2021 IEEE 39th International Conference on Computer Design (ICCD))

   Shuffle is an indispensable process in distributed online analytical processing systems to enable task-level parallelism exploitation via multiple nodes. As a data-intensive data reorganization process, shuffle implemented on general-purpose CPUs not only incurs data traffic back and forth between the computing and storage resources, but also pollutes the cache hierarchy with almost zero data reuse. As a result, shuffle can easily become the bottleneck of distributed analysis pipelines.Our PSACS approach attacks these bottlenecks with the rising computational storage paradigm. Shuffle is offloaded to the storage-side PSACS accelerator to avoid polluting computing node memory hierarchy and enjoy the latency, bandwidth and energy benefits of near-data computing. Further, the microarchitecture of PSACS exploits data-, subtask-, and task-level parallelism for high performance and a customized scratchpad for fast on-chip random access.PSACS achieves 4.6x—5.7x shuffle throughput at kernel-level and up to 1.3x overall shuffle throughput with only a twentieth of CPU utilization comparing to software baselines. These mount up to 23% end-to-end OLAP query speedup on average. 

   more » « less
4. A Shuffle Theorem for Paths Under Any Line

   https://doi.org/10.1017/fmp.2023.4  

   Blasiak, Jonah; Haiman, Mark; Morse, Jennifer; Pun, Anna; Seelinger, George H. (January 2023, Forum of Mathematics, Pi)

   Abstract We generalize the shuffle theorem and its $(km,kn)$ version, as conjectured by Haglund et al. and Bergeron et al. and proven by Carlsson and Mellit, and Mellit, respectively. In our version the $(km,kn)$ Dyck paths on the combinatorial side are replaced by lattice paths lying under a line segment whose x and y intercepts need not be integers, and the algebraic side is given either by a Schiffmann algebra operator formula or an equivalent explicit raising operator formula. We derive our combinatorial identity as the polynomial truncation of an identity of infinite series of $$\operatorname {\mathrm {GL}}_{l}$$ characters, expressed in terms of infinite series versions of LLT polynomials. The series identity in question follows from a Cauchy identity for nonsymmetric Hall–Littlewood polynomials. 

   more » « less
5. Cyclic Shuffle-Compatibility Via Cyclic Shuffle Algebras

   https://doi.org/10.1007/s00026-023-00669-9  

   Liang, Jinting; Sagan, Bruce_E; Zhuang, Yan (October 2023, Annals of Combinatorics)

   Abstract A permutation statistic$${{\,\textrm{st}\,}}$$ $\text{st}$is said to be shuffle-compatible if the distribution of$${{\,\textrm{st}\,}}$$ $\text{st}$over the set of shuffles of two disjoint permutations$$\pi $$ $\pi$and$$\sigma $$ $\sigma$depends only on$${{\,\textrm{st}\,}}\pi $$ $\text{st}\pi$,$${{\,\textrm{st}\,}}\sigma $$ $\text{st}\sigma$, and the lengths of$$\pi $$ $\pi$and$$\sigma $$ $\sigma$. Shuffle-compatibility is implicit in Stanley’s early work onP-partitions, and was first explicitly studied by Gessel and Zhuang, who developed an algebraic framework for shuffle-compatibility centered around their notion of the shuffle algebra of a shuffle-compatible statistic. For a family of statistics called descent statistics, these shuffle algebras are isomorphic to quotients of the algebra of quasisymmetric functions. Recently, Domagalski, Liang, Minnich, Sagan, Schmidt, and Sietsema defined a version of shuffle-compatibility for statistics on cyclic permutations, and studied cyclic shuffle-compatibility through purely combinatorial means. In this paper, we define the cyclic shuffle algebra of a cyclic shuffle-compatible statistic, and develop an algebraic framework for cyclic shuffle-compatibility in which the role of quasisymmetric functions is replaced by the cyclic quasisymmetric functions recently introduced by Adin, Gessel, Reiner, and Roichman. We use our theory to provide explicit descriptions for the cyclic shuffle algebras of various cyclic permutation statistics, which in turn gives algebraic proofs for their cyclic shuffle-compatibility. 

   more » « less