Distributed Storm
Distributed Storm is an extension of Apache Storm, developed at the University of Rome "Tor Vergata". We have extended Storm to provide distributed monitoring, scheduling and management capabilities.
Table of content
Short description
Distributed Storm is an extension of Apache Storm that supports the execution of distributed, adaptive, and QoS-aware scheduling algorithms. It introduces some key components on each worker node that enhance the system with adaptation capabilities, relying on a MAPE (Monitor, Analyze, Plan, and Execute) feedback loop periodically executed. Specifically, the newly introduced components are: the AdaptiveSchedulers, the QoSMonitors, and the WorkerMonitors.
- AdaptiveScheduler is located on each Supervisor and represents the distributed scheduler that executes the distributed placement policy;
- QoSMonitor estimates the network latency with respect to the other system nodes and monitors the QoS attributes of the worker node. Currently it monitors the node utilization. The information collected can be used by the AdaptiveScheduler.
- WorkerMonitor is executed for each Storm worker process and computes the data rate exchanged among the application components.The information collected can be used by the AdaptiveScheduler.
Please, refer to the official Storm documentation for better understanding the used terminology.
Further details on our extension of Storm can be found in:
- Valeria Cardellini, Vincenzo Grassi, Francesco Lo Presti, and Matteo Nardelli. 2015. Distributed QoS-aware scheduling in storm. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS '15). ACM, New York, NY, USA, 344-347. ( PDF )
Available Distributed Scheduling Algorithms
The AdaptiveScheduler implements the following scheduling policies:
- Round Robin
- Our version of the Pietzuch's algorithm: PDF
Citation
Dear academics/scientists, please acknowledge the use of Distributed Storm by citing the following publication:
Valeria Cardellini, Vincenzo Grassi, Francesco Lo Presti, and Matteo Nardelli. 2015. Distributed QoS-aware scheduling in storm. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS '15). ACM, New York, NY, USA, 344-347. DOI=http://dx.doi.org/10.1145/2675743.2776766
Status
Instable. Our source code is currently under revision.
Documentation
For more information, you may want to consult our wiki pages.
License
Distributed Storm is under Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0. See the NOTICE file distributed with this work for additional information regarding copyright ownership. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Publications
- V. Cardellini, V. Grassi, F. Lo Presti, M. Nardelli, "Poster: Distributed QoS-aware scheduling in Storm", Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems (DEBS 2015), 4 pages, Oslo, Norway, July 2015. ACM.: http://dl.acm.org/citation.cfm?id=2675743.2776766
- V. Cardellini, V. Grassi, F. Lo Presti, M. Nardelli, "On QoS-aware scheduling of data stream applications over fog computing infrastructures", Proceedings of the 5th International Workshop on Management of Cloud and Smart City Systems 2015 (MOCS 2015), in conjunction with IEEE ISCC 2015, 6 pages, Larnaca, Cyprus, July 2015. IEEE.
Team
Matteo Nardelli (reference)
Gabriele Scolastri
Collaborators
Valeria Cardellini
Francesco Lo Presti
Vincenzo Grassi