Streaming Transfer Optimization for Distributed Science Workflows

Loading...
Thumbnail Image

Authors

Ucar, Davut

Issue Date

2020

Type

Thesis

Language

Keywords

Distributed workflows , Online profiling , Streaming applications , Throughput optimization

Research Projects

Organizational Units

Journal Issue

Alternative Title

Abstract

Driven by advancements in computing and sensing technology, many scientific projects started to generate a huge volume of data that reaches to petabytes in scale. Distributed and collaborative nature of these projects requires produced data to be streamed to geographically distributed locations in a timely manner to enable real-time (or near-realtime) processing. Thus, robust and predictable network performance is key to streamline end-to-end workflow execution of streaming projects. On the other hand, existing high-performance data transfer applications and services support “best-effort” Quality of Service(QoS) model, thus fail to sustain high transfer performance when transfer conditions (e.g.,dataset characteristics, background traffic, etc.) deviate from initial observations., making them ill-suited for streaming workflows with stringent performance requirements.In this thesis, we propose FStream to offer performance guarantees to time-sensitive streaming applications by continuously monitoring transfer performance and re-adjusting transfer settings to adapt to dynamic transfer conditions and sustain high transfer performance throughout the workflow execution. To achieve this goal in real-time,FStream employs online profiling to rapidly explore solution space of transfer settings and find the one that meets QoS requirements with minimal overhead. It also takes advantage of the long-running nature of streaming workflows and keeps track of past online profiling results to greatly reduce convergence time of future online profiling runs. We evaluated the performance of FStream by transferring several synthetic and real-world workloads using high-performance production networks and showed that it offers up to an order of magnitude performance improvement over state-of-the-art solutions.

Description

Citation

Publisher

License

Journal

Volume

Issue

PubMed ID

DOI

ISSN

EISSN