Cette page contient quelques informations courantes a propos du Work Package 8 du projet ANR SONGS (ANR-11-INFR-13).

Le lien vers notepad

Work in progress

Work meeting, Paris, Oct 4 2012.

Participants prevus:

• Nice:
  • Olivier Dalle
  • Helene Renard
  • Philippe Mussi
  • Emilio Mancini
• Grenoble
  • Arnaud Legrand
  • Lucas Schnorr
  • Pierre Navarro
  • Cristian Ruiz
  • Augustin Degomme
  • Wagner Kolberg
• Nancy
  • Martin Quinson
  • Lucas Nussbaum
  • Paul Bédaride
  • Maximiliano Geier
  • Sébastien Badia
• Lyon
  • Frederic Suter
• Invités:
  • Pablo Oliveira-Castro
  • Eric Petit

Tentative agenda:

• 9h30 - 10h00 : accueil, presentation WP8/rappels objectifs…
• 10h00 - 10h30 : Tour de table, presentation des participants, presentation succinte des travaux en cours et centre d’interêt sur chaque site(5–10 minutes par site)
• 10h30 - 11h00 Pause cafe
• 11h00 - 13:00 3–4 presentations de travaux de recherche/resultats/papiers (OD: merci de m’envoyer vos propositions pour essayer d’evaluer le temps a prevoir)
  • Sascha: Demo vistrails / skype (5–10 minutes)
  • Arnaud: introduction à R, ggplot2 et à knitr (30 mins ?)
  • Olivier: Resume du papier WSC (voir en bas de la page) (30 mins)
  • Pablo & Eric: Présentation d’ASK (30 mins)
  • Cristian Ruiz: Expo (a confirmer)
• 13:00 - 14h00 dejeuner (plateaux repas)
• 14h00 - 16h00 Discussion/brainstorming/probleme solving
  • Design of Experiment (1h)
  • Open Science et Reproductibilite (1h)
    • Emilio, a propos de terminologie et repositories (15 mins)
• 16h00 - 16h30 Pause cafe
• 16h30 - 17h30 “Business” meeting
  • point sur les livrables
  • budget / point sur les recrutement en cours a venir
  • plan d’action / collaborations possibles
  • Idees/suggestions pour animer le WP8
  • prochaine(s) reunion(s)

Work package description

Since simulation-based studies are much cheaper than real experiments, most studies rely on campaigns comprising thousands of simulation. Usually, the rationale behind a large number of ex- periments is to ensure some statistical confidence in the estimation of the quality of a given approach. Yet, performing such study raises several issues that all need to be addressed to perform a sound study. This is done in the following subtasks.

Subtask 8.1: Design of experiments (Grenoble, Nice)

scheduling (informal): T3 – T45

In general usage, design of experiments (DOE) or experimental design is a methodology that concerns the decision of which parameters to vary and in which range, of which combinations of pa- rameters to test, of how many time each experiment should be performed, etc. Computer scientists too often “design” their simulation campaigns by trying all possible combinations of parameters and testing each configuration “a sufficient number of times”. Yet, to answer a given question, several dif- ferent designs may be used and some of them are much better than others in term of computational effort and confidence. DOE is a set of statistical techniques that have been successfully applied in many fields (agriculture, chemistry, industrial process, …) where experiments are considered ex- pensive. Even though computer simulations are cheap, when used at large scale, they clearly incur a non-negligible cost (e.g., in term of energy consumption, time, . . . ). Further techniques for improving the quality of the whole set of experiments (e.g., variance reduction using blocking, antithetic vari- ables, control variates, . . . ) could also be used. The work started in USS SIMGrid enabled us to realize the importance of such techniques and the need to favor their widespread adoption in our community. Indeed, most of the time, these experiments are very poorly performed and conducted and have thus very little statistical meaning.

Risks and backup solution. Original research in the DOE field requires highly skilled researchers in statistical analysis, which might not be found in the partnership. However, applying existing research results of DOE would already greatly contribute to improve SIMGrid and benefit to all its users.

Subtask 8.2: Campaign management (Grenoble, Nancy)

scheduling (informal): T12 – T36

Once the set of simulations to perform has been determined, they need to be run and returned as fast as possible. The current practice in the field is to set up campaigns comprising dozens of thousands of jobs, then to run them through a batch scheduler for several days, and finally to spend a week or so trying to figure out what can be extracted from this huge amount of data.

Batch schedulers are poorly fitted to handle such workload and the Grid’5000 managers regularly run into this trouble. Surprisingly, projects like BOINC, APST or Condor that address the scheduling are also not really suited to such workload when the computer resources have to be shared by many users. Furthermore, the use of DOE should decrease the size of the campaigns and incur a need for very fast response time to ensure a seamless campaign management. There is thus a real need to propose the right level of abstraction to satisfy both users and platform managers.

‘Risks and backup solution. It might be difficult to allow the SIMGrid campaign management job schedulers to efficiently interact with the platform job schedulers, for example in order to smoothly adjust the resource consumption. In worst case, simple but rather inefficient strategies can always be envisioned.

Subtask 8.3: Open Science (Nice, Nancy)

scheduling (informal): T6 – T48 (informal)

One of the basis of science is to build upon the knowledge of others, but also to question this knowledge. Therefore, reproducing the results of others is a major requirement and, like any other experimental scientist, computer scientists need to reiterate and check previous results. Yet, the lack of sound and efficient experimental workflows prevents such a reproducibility, even for a single experimenter. Common practice on the field is to have a bunch of directories with thousands of log files from which scripts extract data that can then be plotted. Everyone already suffered the pain of not remembering exactly how a particular result was obtained and not managing to reproduce it. The whole process (experiment design, running, data gathering, filtering, analysis) should be monitored so that any step can be reproduced afterwards either by the analyst himself or by an other researcher. Such workflows have started being proposed in other communities and facilitate the detection of blatant flaws in the experiments. We would like to investigate how such techniques apply to simulation for computer science and how they can help us in real case studies.

Risks and backup solution. Simulation may be used with different goals in mind and therefore follow different workflows. Furthermore, no simulation tool to our knowledge does properly identify and support simulation workflows. Therefore the risk is to not be able to exhibit workflows having the proper granularity level. If existing general purpose tools, such as Bonita, can not be integrated in the SIMGrid overall architecture, it is always possible to rely on external workflow specification systems such as myexperiment.org.

Meetings

Kickoff Meeting (Jan 18, 2012)

• Arnaud Legrand. slides (PDF) Δ of Arnaud’s talk.

SUD (SimGrid Users’ Days) (June 2012)

• Martin Quinson. slides Δ
• Lucas Nussbaum. slides Δ

Publications

Part of the project contributions

• Olivier Dalle (2012) On Reproducibility and Traceability of Simulations. Δ In Proceedings of the 2012 Winter Simulation Conference. Berlin, December 2012. (C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose and A. M. Uhrmacher, Eds.). Invited paper. To appear.

Collaborations and external contributions

• P. de Oliveira Castro, E. Petit, JC. Beyler, W. Jalby (2012). ASK: Adaptive Sampling Kit (slides, PDF) Δ. In Proceedings of Europar 2012.