Publikasjoner
-
Volnes, Espen; Plagemann, Thomas Peter & Goebel, Vera Hermine
(2023).
To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing.
IEEE Communications Surveys and Tutorials.
ISSN 1553-877X.
26(1),
s. 670–705.
doi:
10.1109/COMST.2023.3330953.
Fulltekst i vitenarkiv
Vis sammendrag
One of the most important issues in distributed data stream processing systems is using operator migration to handle highly variable workloads cost-efficiently and adapt to the needs at any given time on demand. Operator migration is a complex process involving changes in the state and stream management of a running query, typically without any data loss, and with as little disruption to the execution as possible. This tutorial aims to introduce operator migration, explain the core elements of operator migration, and provide the reader with a good understanding of the design alternatives used in existing solutions. We developed a conceptual model to explain the fundamentals of operator migration and introduce a unified terminology, leading to a taxonomy of existing solutions. The conceptual model separates mechanisms, i.e., how to migrate, and policy, i.e., when to migrate. This separation is further applied to structure the description of existing solutions, offering the reader an algorithmic perspective on various design alternatives. To enhance our understanding of the impact of various design alternatives on migration mechanisms, we also conducted an empirical study that provides quantitative insights. The operator downtime for the naïve migration approach is almost 20 times longer than when applying an incremental checkpoint-based approach.
-
Volnes, Espen; Plagemann, Thomas Peter; Boris, Koldehofe & Goebel, Vera Hermine
(2022).
Travel light: state shedding for efficient operator migration.
I Yonghuan, Zhou (Red.),
DEBS '22: Proceedings of the 16th ACM International Conference on Distributed and Event-Based Systems.
Association for Computing Machinery (ACM).
ISSN 978-1-4503-9308-9.
s. 79–84.
doi:
https:/doi.org/10.1145/3524860.3539638.
Vis sammendrag
Operator migration is a crucial concept to adapt event pro- cessing systems to dynamic changes. When the placement of a stateful operator changes, the operator state must be migrated to the new host. However, operator state size and time constraints can make it impossible to migrate the op- erator without severe Quality of Service (QoS) degradation. As a relief, we propose to perform state shedding in such a situation. The core idea of state shedding is to partition the operator state, assign a utility to each partial state, and use the utility and size of each partial state to identify the most useful partial states that can be migrated in a given time frame. Thus, state shedding can maintain a substantially higher QoS with a lower impact on query results than state- of-the-art solutions targeting consistent state at the old and new host. In this paper, we define this novel approach and in a simulation environment evaluate state shedding in migration scenarios with pattern-matching queries.
-
Volnes, Espen; Plagemann, Thomas Peter; Goebel, Vera Hermine & Kristiansen, Stein
(2021).
EXPOSE: Experimental Performance Evaluation of Stream Processing Engines Made Easy.
I Nambiar, Raghunath (Red.),
Performance Evaluation and Benchmarking
12th TPC Technology Conference.
Springer.
ISSN 978-3-030-84924-5.
s. 18–34.
doi:
https:/doi.org/10.1007/978-3-030-84924-5_2.
Vis sammendrag
Experimental performance evaluation of stream processing engines (SPE) can be a great challenge. Aiming to make fair comparisons of different SPEs raises this bar even higher. One important reason for this challenge is the fact that these systems often use concepts that require expert knowledge for each SPE. To address this issue, we present Expose, a distributed performance evaluation framework for SPEs that enables a user through a declarative approach to specify experiments and conduct them on multiple SPEs in a fair way and with low effort. Experimenters with few technical skills can define and execute distributed experiments that can easily be replicated. We demonstrate Expose by defining a set of experiments based on the existing NEXMark benchmark and conduct a performance evaluation of Flink, Beam with the Flink runner, Siddhi, T-Rex, and Esper, on powerful and resource-constrained hardware.
-
Volnes, Espen; Kristiansen, Stein & Plagemann, Thomas Peter
(2021).
Improving the Accuracy of Timing in Scalable WSN Simulations with Communication Software Execution Models.
Computer Networks.
ISSN 1389-1286.
188.
doi:
10.1016/j.comnet.2021.107855.
-
Volnes, Espen; Kristiansen, Stein; Plagemann, Thomas Peter; Goebel, Vera Hermine & Lindeberg, Morten Gunnar Bjørner
(2019).
Modeling the Software Execution of CEP in DCEP-Sim.
I Koldehove, Boris (Red.),
DEBS '19: Proceedings of the 13th ACM International Conference on Distributed and Event-based Systems
2019.
ACM Publications.
ISSN 978-1-4503-6794-3.
s. 244–247.
doi:
https:/doi.org/10.1145/3328905.3332508.
-
Volnes, Espen; Kristiansen, Stein & Plagemann, Thomas Peter
(2019).
Communication Software Execution Model of a WSN Device for More Accurate Simulation in ns-3.
I Beydoun, Ghassan (Red.),
Proceedings of The 11th International Conference on Computer Modeling and Simulation (ICCMS 2019).
Association for Computing Machinery (ACM).
ISSN 978-1-4503-6619-9.
s. 184–189.
doi:
https:/doi.org/10.1145/3307363.3307371.
-
Dale, Øystein; Kristiansen, Stein; Plagemann, Thomas Peter & Volnes, Espen
(2019).
Many Cores and Still Delays: Simulating Multi-Core Communication Software Execution.
I Beydoun, Ghassan (Red.),
Proceedings of The 11th International Conference on Computer Modeling and Simulation (ICCMS 2019).
Association for Computing Machinery (ACM).
ISSN 978-1-4503-6619-9.
s. 171–175.
doi:
https:/doi.org/10.1145/3307363.3307384.
Vis sammendrag
The effects of software execution are typically ignored in network simulators. This leads to inaccurate simulation results for new and emerging network technologies where packet processing in software has a significant impact on network performance. In this paper we extend our prior work to develop models that capture the temporal aspects of software execution on both uni- and multi-core devices. When executed in parallel with existing network protocol models in network simulators, they significantly improve the accuracy of the simulation results. We demonstrate the applicability of the extended methodology by modeling the execution of communication software on one multi-core device, the Galaxy Nexus smartphone, and evaluate the resulting models in terms of accuracy and scalability. We find that our models are able to reproduce the behavior of the modeled software, and that the models scale well with the number of nodes, cores, and threads included in the simulation; i.e., the simulation overhead increases linearly with the number of nodes.
Se alle arbeider i Cristin
-
Volnes, Espen; Kristiansen, Stein; Plagemann, Thomas Peter; Goebel, Vera Hermine & Lindeberg, Morten Gunnar Bjørner
(2019).
Modeling the Software Execution of CEP in DCEP-Sim.
-
Volnes, Espen; Kristiansen, Stein & Plagemann, Thomas Peter
(2019).
Communication Software Execution Model of a WSN Device for More Accurate Simulation in ns-3.
-
Dale, Øystein; Kristiansen, Stein; Plagemann, Thomas Peter & Volnes, Espen
(2019).
Many Cores and Still Delays: Simulating Multi-Core Communication Software Execution.
Vis sammendrag
The effects of software execution are typically ignored in network simulators. This leads to inaccurate simulation results for new and emerging network technologies where packet processing in software has a significant impact on network performance. In this paper we extend our prior work to develop models that capture the temporal aspects of software execution on both uni- and multi-core devices. When executed in parallel with existing network protocol models in network simulators, they significantly improve the accuracy of the simulation results. We demonstrate the applicability of the extended methodology by modeling the execution of communication software on one multi-core device, the Galaxy Nexus smartphone, and evaluate the resulting models in terms of accuracy and scalability. We find that our models are able to reproduce the behavior of the modeled software, and that the models scale well with the number of nodes, cores, and threads included in the simulation; i.e., the simulation overhead increases linearly with the number of nodes.
-
Volnes, Espen; Plagemann, Thomas Peter & Goebel, Vera Hermine
(2022).
To Migrate or not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing.
arXiv2022.
Vis sammendrag
One of the most important issues in data stream processing systems is to use operator migration to handle highly variable workloads in a cost-efficient manner and adapt to the needs at any given time on demand. Operator migration is a complex process that involves changes in the state and stream management of a running query, typically without any loss of data, and with as little disruption to the execution as possible. This survey provides an overview of solutions for operator migration from a historical perspective as well as the perspective of the goal of migration. It introduces a conceptual model of operator migration to establish a unified terminology and classify existing solutions. Existing work in the area is analyzed to separate the mechanism of migration from the decision to migrate the data. In case of the latter, a cost-benefit analysis is emphasized that is important for operator migration but is often only implicitly addressed, or is neglected altogether. A description of the available solutions provides the reader with a good understanding of the design alternatives from an algorithmic viewpoint. We complement this with an empirical study to provide quantitative insights on the impact of different design alternatives on the mechanisms of migration
Se alle arbeider i Cristin
Publisert
8. apr. 2019 09:59
- Sist endret
6. juli 2023 14:50