Empirical methods have grown common in software engineering, but there is no consensus on how to carry it out. Is practical relevance key? Do internally valid studies have any value? Should we replicate more, which is common in other sciences to address the trade off between internal and external validity? We asked the community how empirical research should take place in software engineering, complemented with a literature survey about the status of empirical research.
Literature Review
We analyzed the recent years of ESEC/FSE, ICSE, and EMSE regarding five objectives:
- Was an empirical method applied?
- Were the experimental subjects human or non-human?
- Were the human experimental subjects professionals or students?
- Was an internal or external replication reported?
- How are threats to validity described?
Overview of results:
Download csv file with detailed results
Questionnaire
We designed numerous questions to assess participants' opinion. All questions were optional.
Download questionnaire (pdf)
Responses
- Research Objective 1: Awareness of community for tradeoff between internal and external validity.
- Research Objective 2: Opinion of community regarding how to address this tradeoff.
- Research Objective 3: Opinion of community regarding replication.
Download responses and categorization (csv)
Question 23: Factors for balancing internal vs. external validity (csv)
