Optimizing Cross-Platform Data Movement
- Data analytics are moving beyond the limits of a single data processing platform. A cross-platform query optimizer is necessary to enable applications to run their tasks over multiple platforms efficiently and in a platform-agnostic manner. For the optimizer to be effective, it must consider data movement costs across different data processing platforms. In this paper, we present the graph-based data movement strategy used by RHEEM, our open-source cross-platform system. In particular, we (i) model the data movement problem as a new graph problem, which we prove to be NP-hard, and (ii) propose a novel graph exploration algorithm, which allows RHEEM to discover multiple hidden opportunities for cross-platform data processing.
| Author details: | Sebastian KruseORCiDGND, Zoi Kaoudi, Jorge-Arnulfo Quiane-RuizORCiD, Sanjay ChawlaORCiDGND, Felix NaumannORCiDGND, Bertty Contreras-Rojas |
|---|---|
| DOI: | https://doi.org/10.1109/ICDE.2019.00162 |
| ISBN: | 978-1-5386-7474-1 |
| ISBN: | 978-1-5386-7475-8 |
| ISSN: | 1084-4627 |
| ISSN: | 1063-6382 |
| Title of parent work (English): | 2019 IEEE 35th International Conference on Data Engineering (ICDE) |
| Publisher: | IEEE |
| Place of publishing: | New York |
| Publication type: | Other |
| Language: | English |
| Year of first publication: | 2019 |
| Publication year: | 2019 |
| Release date: | 2021/05/05 |
| Number of pages: | 4 |
| First page: | 1642 |
| Last Page: | 1645 |
| Organizational units: | Digital Engineering Fakultät / Hasso-Plattner-Institut für Digital Engineering GmbH |
| DDC classification: | 0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 000 Informatik, Informationswissenschaft, allgemeine Werke |
| Peer review: | Referiert |
