Toward Terabit-per-second Networks: Developing Data Transfer Solutions for Next-Generation Research Networks
Loading...
Authors
Arifuzzaman, Md
Issue Date
2023
Type
Dissertation
Language
Keywords
active probing , high-speed networks , large-scale data transfers , network parallelism , online file transfer optimization
Alternative Title
Abstract
Research networks provide high-speed wide-area network connectivity between research and education institutions to facilitate large-scale data transfers. However, scalability issues of legacy transfer applications (e.g., scp and FTP) and their extensions (e.g., GridFTP and rsync) hinder the effective utilization of these networks. In this dissertation, we propose online optimization algorithms to tune the degree of parallelism for file transfers to maximize transfer throughput while keeping system overhead at a minimum. First, we introduce Falcon that utilizes game theory-inspired novel utility function to evaluate the performance of various parallelism levels such that competing transfers are guaranteed to converge to a fair and stable solution. We assessed the performance of Falcon in isolated and production high-speed networks and found that it can discover optimal transfer parallelism in as little as 20 seconds and outperform the state-of-the-art solutions by more than 2x. On the other hand, Falcon uses the same level of parallelism for network and I/O operations which may result in increased system overhead and unfair resource allocation. To address this issue, we developed modular file transfer architecture, Marlin, that separates I/O and network operations so that parallelism can be independently adjusted for each component. Marlin adopts online gradient descent algorithm to swiftly search the solution space and find the optimal level of parallelism for read, transfer, and write operations. Experimental results collected under various network settings show that Marlin can identify and use a minimum parallelism level for each component, reducing system overhead (e.g., low CPU usage and I/O contention) and improving fairness among competing transfers.