WASP: Wide-area Adaptive Stream Processing [conference paper]

Three computers with locks in a circle illustration

Conference

Proceedings of the 21st International Middleware Conference - December 7-11, 2020

Authors

Albert Jonathan (Ph.D. student), Abhishek Chandra (professor), Jon Weissman (professor)

Abstract

Adaptability is critical for stream processing systems to ensure stable, low-latency, and high-throughput processing of long-running queries. Such adaptability is particularly challenging for wide-area stream processing due to the highly dynamic nature of the wide-area environment, which includes unpredictable workload patterns, variable network bandwidth, occurrence of stragglers, and failures. Unfortunately, existing adaptation techniques typically achieve these performance goals by compromising the quality/accuracy of the results, and they are often application-dependent. In this work, we rethink the adaptability property of wide-area stream processing systems and propose a resource-aware adaptation framework, called WASP. WASP adapts queries through a combination of multiple techniques: task re-assignment, operator scaling, and query re-planning, and applies them in a WAN-aware manner. It is able to automatically determine which adaptation action to take depending on the type of queries, dynamics, and optimization goals. We have implemented a WASP prototype on Apache Flink. Experimental evaluation with the YSB benchmark and a real Twitter trace shows that WASP can handle various dynamics without compromising the quality of the results.

Link to full paper

WASP: Wide-area Adaptive Stream Processing

Keywords

distributed systems, operating systems