Search for: All records

Batch systems face issues with workloads comprising millions of tasks with short runtimes—scheduling is most efficient for long-running jobs. In addition, the nature of heterogeneous computing systems makes task bundling impractical. Building on HTCondor, the Event Workflow Management System (EWMS) provides an efficient solution to thrive with both paradigms, while featuring user-friendly and self-healing principles. Here, we describe this method, its implementation, and a real-world application. 

The IceCube Neutrino Observatory is a cubic kilometer neutrino telescope located at the geographic South Pole. To accurately and promptly reconstruct the arrival direction of candidate neutrino events for Multi-Messenger Astrophysics use cases, IceCube employs Skymap Scanner workflows managed by the SkyDriver service. The Skymap Scanner performs maximum-likelihood tests on individual pixels generated from the Hierarchical Equal Area isoLatitude Pixelation (HEALPix) algorithm. Each test is computationally independent, which allows for massive parallelization. This workload is distributed using the Event Workflow Management System (EWMS)—a message-based workflow management system designed to scale to trillions of pixels per day. SkyDriver orchestrates multiple distinct Skymap Scanner workflows behind a REST interface, providing an easy-to-use reconstruction service for real-time candidate, cataloged, and simulated events. Here, we outline the SkyDriver service technique and the initial development of EWMS.