Optimizing Cross-Platform Data Movement
- Data analytics are moving beyond the limits of a single data processing platform. A cross-platform query optimizer is necessary to enable applications to run their tasks over multiple platforms efficiently and in a platform-agnostic manner. For the optimizer to be effective, it must consider data movement costs across different data processing platforms. In this paper, we present the graph-based data movement strategy used by RHEEM, our open-source cross-platform system. In particular, we (i) model the data movement problem as a new graph problem, which we prove to be NP-hard, and (ii) propose a novel graph exploration algorithm, which allows RHEEM to discover multiple hidden opportunities for cross-platform data processing.
Author details: | Sebastian KruseORCiDGND, Zoi Kaoudi, Jorge-Arnulfo Quiane-RuizORCiD, Sanjay ChawlaORCiDGND, Felix NaumannORCiDGND, Bertty Contreras-Rojas |
---|---|
DOI: | https://doi.org/10.1109/ICDE.2019.00162 |
ISBN: | 978-1-5386-7474-1 |
ISBN: | 978-1-5386-7475-8 |
ISSN: | 1084-4627 |
ISSN: | 1063-6382 |
Title of parent work (English): | 2019 IEEE 35th International Conference on Data Engineering (ICDE) |
Publisher: | IEEE |
Place of publishing: | New York |
Publication type: | Other |
Language: | English |
Year of first publication: | 2019 |
Publication year: | 2019 |
Release date: | 2021/05/05 |
Number of pages: | 4 |
First page: | 1642 |
Last Page: | 1645 |
Organizational units: | Digital Engineering Fakultät / Hasso-Plattner-Institut für Digital Engineering GmbH |
DDC classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke |
Peer review: | Referiert |