Search for: All records

Ensuring the integrity of petabyte-scale file transfers is essential for the data gathered from scientific instruments. As packet sizes increase, so does the likelihood of errors, resulting in a higher probability of undetected errors in the packet. This paper presents a Multi-Level Error Detection (MLED) framework that leverages in-network resources to reduce undetected error probability (UEP) in file transmission. MLED is based on a configurable recursive architecture that organizes communication in layers at different levels, decoupling network functions such as error detection, routing, addressing, and security. Each layer L_ij at level i implements a policy P_ij that governs its operation, including the error detection mechanism used, specific to the scope of that layer. MLED can be configured to mimic the error detection mechanisms of existing large-scale file transfer protocols. The recursive structure of MLED is analyzed and it shows that adding additional levels of error detection reduces the overall UEP. An adversarial error model is designed to introduce errors into files that evade detection by multiple error detection policies. Through experimentation using the FABRIC testbed the traditional approach, with transport- and data link- layer error detection, results in a corrupt file transfer requiring retransmission of the entire file. Using its recursive structure, an implementation of MLED detects and corrects these adversarial errors at intermediate levels inside the network, avoiding file retransmission under non-zero error rates. MLED therefore achieves a 100% gain in goodput over the traditional approach, reaching a goodput of over 800 Mbps on a single connection with no appreciable increase in delay.