SELFMAN Project
From SELFMAN Wiki
| Table of contents |
Self Management for Large-Scale Distributed Systems based on Structured Overlay Networks and Components
European Sixth Framework Programme, Priority 2, Information Society Technologies
Project coordinator: Peter Van Roy, Université catholique de Louvain
June 1, 2006 - Sept. 30, 2009
Breakthrough Results
- PeerTV: The Future of Media Distribution PeerTV is a state-of-the-art application developed by Peerialism that distributes video over Internet with live streaming and progressive download. It uses advanced technology based on peer-to-peer structured overlay networks and optimization algorithms.
- Peerialism home (http://www.peerialism.se/).
- Scalaris: Scalable Transactional Storage for Web 2.0 Services (Scalaris on Google Code (http://code.google.com/p/scalaris/)) Scalaris is an open-source library that provides a replicated transactional key/value store. It scales to hundreds of nodes and provides strong data consistency in the face of concurrent operations, node failures, and network problems. It won first prize in the IEEE International Scalable Computing Challenge 2008.
- Scalaris: A Scalable Transactional Data Store for Web 2.0 Services, Zuse Institute Berlin, 2008.
- Beernet: a relaxed pbeer-to-pbeer network (http://beernet.info.ucl.ac.be/), Université catholique de Louvain, 2008 (related system based on relaxed ring).
Press
- Self-managing internet applications flex their muscles (http://cordis.europa.eu/ictresults/index.cfm?section=news&tpl=article&BrowsingType=Features&ID=90895), ICT Results, Oct. 2, 2009.
- The self-managing, “unbreakable” internet? (http://cordis.europa.eu/ictresults/index.cfm?section=news&tpl=article&BrowsingType=Features&ID=90892), ICT Results, Sept. 30, 2009.
- A self-managing peer-to-peer network, eStrategies Project Magazine, June 2008.
- SELFMAN: Self management of large-scale Internet applications, two-page project summary, Sept. 2006.
(many other articles derived from these have appeared in technical press and blogs)
Partners
- Université catholique de Louvain (http://www.info.ucl.ac.be/~pvr/distribution.html), Belgium
- Royal Institute of Technology (Kungliga Tekniska Högskolan), Sweden
- Institut National de Recherche en Informatique et Automatique (INRIA), France
- France Telecom Research and Development, France
- Konrad-Zuse-Zentrum für Informationstechnik Berlin (http://www.zib.de/CSR/), Germany
- Peerialism AB (http://www.peerialism.se), Stockholm, Sweden
- National University of Singapore, Singapore
Summary
SELFMAN Publishable Final Activity Report, Nov. 2009.
As distributed systems become larger and more complex, the task of managing them becomes overwhelming. "Abnormal" events such as software updates, faults, threats, and performance hotspots become normal and even frequent occurrences. The goal of SELFMAN is to handle these events automatically by making the systems self managing: the systems will reconfigure themselves to handle changes in their environment or requirements without human intervention but according to high-level management policies. We focus on four axes of self management, namely self configuration, self healing, self tuning, and self protection.
A major innovation of SELFMAN is to combine the strengths of structured overlay networks and advanced component models. Structured overlay networks originate in peer-to-peer applications, but have matured to provide strong guarantees and efficient communication and storage operations. They reorganize themselves to maintain these properties in a changing environment, which is already a low-level self-management property. Advanced component models provide introspection, reflection, and dynamic reconfiguration abilities, which give the hooks needed for an application to manage itself. SELFMAN will reformulate the overlay network as part of a self-managing component architecture and use it to support high-level self-managing services.
SELFMAN will build a self-managing replicated storage service and use it as the heart of a self-managing multi-tier application hosting environment. We will build demonstrator applications following both industry standards and advanced ideas from programming language research. We will use industrial trace data to measure the effectiveness of the self-management architecture. We intend to release this work as software under an Open Source license.
Events
Workshop on Architectures and Languages for Self-Managing Distributed Systems. San Francisco, CA. In conjunction with SASO 2009, Sept. 14-18, 2009.
Workshop on Decentralized Self Management for Grids, P2P, and User Communities. Isola di San Servolo (Venice), Italy. In conjunction with SASO 2008 (http://polaris.ing.unimo.it/saso2008/), the Second International Conference on Self-Adaptive and Self-Organizing Systems, Oct. 20-21, 2008.
Kompics: Reactive Component Model for Distributed Computing (talk slides). Invited talk at Grid@Mons 2008 (http://grid.umh.ac.be/), Seif Haridi, May 2008.
Self Management for Large-Scale Distributed Systems (talk slides). Invited talk at Grid@Mons 2008 (http://grid.umh.ac.be/), Peter Van Roy, May 2008.
Reflections on Self Management in Software Development (talk slides). Peter Van Roy. Invited lecture given at Scottish Programming Language Seminar (Sept. 28, 2007, Edinburgh, UK) and Almende Summer School (http://summerschool.almende.com) (Aug. 29, 2007, Rotterdam, NL).
Mini-course on collective intelligence given by Mohammed El-Beltagy at ZIB in Berlin, February 15-16, 2007.
Mini-course on reliable distributed programming given by Seif Haridi at ZIB in Berlin, October 12-13, 2006.
SELFMAN Kickoff Workshop at KTH in Stockholm, June 8-9, 2006.
Key Publications
Enhanced Paxos Commit for Transactions on DHTs, Florian Schintke, Alexander Reinefeld, Seif Haridi, and Thorsten Schütt. ZIB Report 09-28, Sept. 2009.
Kompics: A Message-Passing Component Model for Building Distributed Systems (http://kompics.sics.se), Cosmin Arad, Royal Institute of Technology and Swedish Institute of Computer Science, 2010.
Guidelines for Building Self-Managing Applications, Peter Van Roy. SELFMAN deliverable D5.7, June 2009.
A Practical Approach to Network Size Estimation for Structured Overlays. Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. IWSoS '08 - Third International Workshop on Self-Organizing Systems, Dec. 2008, Austria.
Using Global Information for Load Balancing in DHTs, Mikael Högqvist, Seif Haridi, Nico Kruber, Alexander Reinefeld, Thorsten Schütt. 3rd IEEE SELFMAN workshop, SASO 2008, Oct. 2008.
The Relaxed Ring: A Fault-Tolerant Topology for Structured Overlay Networks, Boris Mejías and Peter Van Roy. Journal of Parallel Processing Letters, 18(3), Sept. 2008.
Overcoming Software Fragility with Interacting Feedback Loops and Reversible Phase Transactions, Peter Van Roy. BCS 08, Sept. 2008.
The Challenges and Opportunities of Multiple Processors: Why Multi-Core Processors are Easy and Internet is Hard, Peter Van Roy. ICMC 2008, Belfast, North Ireland, Aug. 2008.
Self Management for Large-Scale Distributed Systems: An Overview of the SELFMAN Project, Peter Van Roy, Seif Haridi, Alexander Reinefeld, Jean-Bernard Stefani, Roland Yap, and Thierry Coupaye. FMCO 2007, Oct. 2007. Revised postproceedings, Springer LNCS, 2008.
Handling Network Partitions and Mergers in Structured Overlay Networks, Tallat M. Shafaat, Ali Ghodsi, Seif Haridi. Seventh IEEE International Conference on Peer-to-Peer Computing (P2P'07), Galway, Ireland, Sept. 2007.
Atomic Commitment in Transactional DHTs, Monika Moser and Seif Haridi. In proceedings of the CoreGRID Symposium, Rennes, France, Aug. 2007.
Distributed Wikis on Structured Overlays, Stefan Plantikow, Alexander Reinefeld, and Florian Schintke. CoreGRID Workshop on Grid Programming Model, Grid and P2P System Architecture, Grid Systems, Tools and Environments, Heraklion, Crete, June 2007.
Fractal Component-Based Software Engineering - Report on the Fractal workshop at ECOOP'06, Thierry Coupaye and Jean-Bernard Stefani. 20th European Conference on Object-Oriented Programming (ECOOP 2006) Workshop Reader, LNCS 4379, 2007.
Distributed k-ary System: Algorithms for Distributed Hash Tables, Ali Ghodsi. Ph.D. Dissertation, Royal Institute of Technology, Stockholm, Oct. 2006.
Self Management and the Future of Software Design, Peter Van Roy. Formal Aspects of Component Software (FACS '06), Sept. 2006 (talk slides).
The Fractal Component Model and Its Support in Java, Eric Bruneton, Thierry Coupaye, Matthieu Leclercq, Vivien Quema, Jean-Bernard Stefani. Software Practice and Experience, special issue on Experiences with Auto-adaptive and Reconfigurable Systems. 36(11-12), 2006.
Structured Overlay without Consistent Hashing: Empirical Results, Thorsten Schütt, Florian Schintke, and Alexander Reinefeld. Proceedings of the Sixth Workshop on Global and Peer-to-Peer Computing (GP2PC'06), May 2006.
On Adaptability in Grid Systems, Artur Andrzejak, Alexander Reinefeld, Florian Schintke, and Thorsten Schütt. Future Generation Grids, Springer-Verlag CoreGRID Series, Jan. 2006.
Self Management of Large-Scale Distributed Systems by Combining Peer-to-Peer Networks and Components, Peter Van Roy, Ali Ghodsi, Seif Haridi, Jean-Bernard Stefani, Thierry Coupaye, Alexander Reinefeld, Ehrhard Winter, and Roland Yap. CoreGRID Technical Report Number TR-0018, Dec. 14, 2005.
Related Links
Almende Summer School 2008 (http://summerschool.almende.com/): Summer school in coordination techniques for self-organizing human-agent networks, Rotterdam, The Netherlands, August 25-28, 2008.
The Agoric Papers (http://www.agorics.com/Library/agoricpapers.html): Markets and Computation: Agoric Open Systems, Incentive Engineering for Computational Resource Management, Comparative Ecology: A Computational Perspective.
IST Research: Software Technologies (http://cordis.europa.eu/ist/st/index.html)
ObjectWeb Consortium (http://www.objectweb.org/): Open source middleware. Fractal Component Model (http://fractal.objectweb.org) An open, hierarchical, dynamic, reflexive component model with sharing.
Mozart Programming System (http://www.mozart-oz.org): Development platform for intelligent, distributed applications.
CoreGRID Network of Excellence (http://www.coregrid.net): Foundations, software infrastructures and applications for large-scale distributed, GRID, and peer-to-peer technologies.
EVERGROW Integrated Project (http://www.evergrow.org): Ever-growing global scale-free networks, their provisioning, repair and unique functions.
ACCA Project (a former EU FET project) (http://www.autonomic-communication.org/projects/acca/): Autonomic Communication: Coordination Action
CASCADAS project (IP Funded within FET Proactive Initiative IST-2004-2.3.4) (http://www.cascadas-project.org/): Bringing Autonomic Services to Life
MADAM project (IST STREP 004159 Sept. 2004-March 2007) (http://www.intermedia.uio.no/confluence/display/madam/Home): Mobility Adaptation
P2P Computing (http://p2pcomputing.blogspot.com/): Literature Survey
Kompics (http://kompics.sics.se/): Reactive Component Model for Distributed Computing
