You are here: Home › Workshops › Large Scale Experiments Workshop

Large Scale Experiments on Agile Programming Techniques (LSExp)

For agile researches it is often difficult to find statistically significant evidence for their hypotheses. Large scale coding contests with hundreds of participants provide an opportunity to observe the participants -– how they approach on solving the problem as well as their solution. These contests provide a lot of data points like various participants backgrounds, different experience levels, individuals and teams, etc. This workshop shall bring researchers and practitioners together to brainstorm possible experiments and how to set them up in the context of coding contests. The outcome shall be to increase the amount of usable data for empirical studies by using programming contests as experiments. We encourage agile researchers to present a case study or experience report of an experiment as well as practitioners and researchers without specific experience to participate at the workshop.

Scientific Workshop on Large Scale Experiments on Agile Programming Techniques (LSExp)

Call for paper – PDF Version

Website: http://www.catalysts.cc/lse-12

Themes & Goals

For agile researches it is often difficult to find statistically significant evidence for their hypotheses like “test-driven development leads to shorter development times and to fewer errors”. For their theses they need experiments with at least 40-50 participants. By organizing a programming contest and enforcing specific rules, researches can easily motivate many programmers to participate in the experiment and use the results for their work.
Large scale coding contests with hundreds of participants provide an opportunity to observe the participants – how they approach on solving the problem as well as their solution (i.e. source code).
In 2007, Catalysts, an Austrian company and corporate member of the AgileAlliance, organized a coding contest for the first time with the goal to investigate whether TDD is beneficial and whether teams win over individuals. 180 persons participated at the most recent contest in December 2011. The challenge was specified in six consecutive levels, where only 12 participants solved all levels within three hours.
These contests are now held three times a year – people can participate “on site” or remotely via internet. Information about the progress of the participants has been collected as well as the source codes (level by level) and some profile data of the participants. This information is already publicly available via an Open Data Server.

This workshop shall bring researchers and practitioners together to:
1. exchange experience of coding contests
2. brainstorm possible experiments
3. brainstorm how to perform those experiments in the context of coding contests
The main outcome of this workshop shall be to increase the amount of usable data for agile researchers, so that they can easier make empirical studies. Catalysts organizes three large coding contests per year and they are willing to adapt the context of future contests on the outcome of this workshop.
Other organizations could join in as well and set up appropriate contexts for such experiments. So there are several opportunities to run such large scale experiments.

Workshop’s Relevance

Coding contests with hundreds of participants provide a lot of data points:

  • participants with various backgrounds
  • different experience levels (number of years)
  • pupils, students, practitioners
  • individuals and/or teams
  • various programming languages
  • history of source code level by level
  • etc.

Such data points allow comparative analyses as well as analysis of the evolution of programs.
Admittedly, the data is not collected in a completely clean laboratory environment. However, the context of a coding contest allows to set up the necessary rules to make the experiment “as controlled as necessary” – much more controlled than the unrestricted open field.
Agile researchers can provide valuable contributions on how to evolve coding contests in the future. They can ask and answer concrete questions about the current state of the art and discuss future publications in this topic. Further, they can design contests in other environments and contexts in regard to receive usable data.

Call for papers

We encourage agile researchers to present a case study or an experience report of an experiment, how it was set up, which rules were tried to enforce, what data was collected, how the data was analyzed, etc. We will select three papers to be presented (15-20 minutes) during the workshop, more papers can be shared with the participants in advance to foster a lively and diverse discussion.

The papers have to be in English using the Springer LNBIP, or similar style. Please submit only PDF files, not longer than 10 pages, on https://www.easychair.org/conferences/?conf=lsexp2012 or send it to lsexp2012@easychair.org.

The deadlines are:
Submission due April 7, 2012
Notification of Acceptance April 21, 2012
Final manuscripts due May 11, 2012

The reviewers are:

  • Paul Grünbacher (Johannes Kepler University Linz)
  • Dietmar Winkler (Vienna University of Technology)
  • Christoph Steindl (Catalysts)
  • Martin Kitzler (Vienna University of Technology, Catalysts)

The Agenda

  • Presentation of the coding contest background (slides with pictures from the contest) – 15 min
  • Presentation of the collected data – 15 min
  • Slot 1: Case study or experience report of an experiment – 20 min
  • Slot 2: Case study or experience report of an experiment – 20 min
  • Slot 3: Case study or experience report of an experiment – 20 min
  • Brainstorming ideas for experiments – 20 min
  • Work out the ideas in groups of 3-5 people – 30 min
  • Present the worked out ideas – 30 min
  • Prioritizing ideas – 10 min
  • Brainstorming how to set up the experiments – 20 min
  • Work out the rules in groups of 3-5 people – 30 min
  • Present the rules – 30 min
  • Develop an action plan – 60 min
  • Quick retrospective (how to improve the workshop in the future) – 30 min
  • Breaks & Buffer – 60 min

Short bio of each organizer

Dipl.-Ing. Dietmar Winkler

Institute of Software Technology & Interactive Systems
Vienna University of Technology
Web: http://qse.ifs.tuwien.ac.at

Dietmar Winkler is researcher and lecturer at the Vienna University of Technology. Since 2010 he is working as project assistant in the Christian Doppler Laboratory “Software Engineering Integration for Flexible Automation Systems”. In 2007 he worked as a guest researcher at Czech Technical University, Department of Cybernetics in Prague (CZ). In 2008 he received a scholarship (by Siemens Austria and the Faculty of Informatics, TU Vienna) for a 4 months 2 Large Scale Experiments on Agile Programming Techniques research stay at the Fraunhofer Institute of Experimental Software Engineering in Kaiserslautern (DE).

Martin Kitzler, BSc

Vienna University of Technology and Catalysts GmbH
Web: http://www.catalysts.cc and http://www.tuwien.ac.at

Martin Kitzler studies the master’s program “Software Engineering and Internet Computing” at the Vienna University of Technology and currently writes his master thesis on various agile development hypothesizes by using coding contest data.
In 2011 he joined Catalysts and works as a SW developer and tester.

Dr. Christoph Steindl

Catalysts GmbH
Web: http://www.catalysts.cc

Lecturer at the Vienna University of Technology, at the Johannes Kepler University Linz and at the University of Applied Sciences in Hagenberg. Christoph Steindl has studied Computer Science at the University Linz, worked there as University Assistant with Professor Mössenböck, then joined IBM and became an IT Architect before he founded Catalysts. He has been an agile practitioner for many years, has coached many teams and set up an agile company. He has been organizing coding contests for more than 5 years.
He is a member of the ACM and OCG (Austrian Computer Society).