case study in empirical research

• Open Access
• Published: 10 November 2020

Case study research for better evaluations of complex interventions: rationale and challenges

• Sara Paparini   ORCID: orcid.org/0000-0002-1909-2481 1 ,
• Judith Green 2 ,
• Chrysanthi Papoutsi 1 ,
• Jamie Murdoch 3 ,
• Mark Petticrew 4 ,
• Trish Greenhalgh 1 ,
• Benjamin Hanckel 5 &
• Sara Shaw 1  

BMC Medicine volume  18 , Article number:  301 ( 2020 ) Cite this article

13k Accesses

29 Citations

38 Altmetric

Metrics details

The need for better methods for evaluation in health research has been widely recognised. The ‘complexity turn’ has drawn attention to the limitations of relying on causal inference from randomised controlled trials alone for understanding whether, and under which conditions, interventions in complex systems improve health services or the public health, and what mechanisms might link interventions and outcomes. We argue that case study research—currently denigrated as poor evidence—is an under-utilised resource for not only providing evidence about context and transferability, but also for helping strengthen causal inferences when pathways between intervention and effects are likely to be non-linear.

Case study research, as an overall approach, is based on in-depth explorations of complex phenomena in their natural, or real-life, settings. Empirical case studies typically enable dynamic understanding of complex challenges and provide evidence about causal mechanisms and the necessary and sufficient conditions (contexts) for intervention implementation and effects. This is essential evidence not just for researchers concerned about internal and external validity, but also research users in policy and practice who need to know what the likely effects of complex programmes or interventions will be in their settings. The health sciences have much to learn from scholarship on case study methodology in the social sciences. However, there are multiple challenges in fully exploiting the potential learning from case study research. First are misconceptions that case study research can only provide exploratory or descriptive evidence. Second, there is little consensus about what a case study is, and considerable diversity in how empirical case studies are conducted and reported. Finally, as case study researchers typically (and appropriately) focus on thick description (that captures contextual detail), it can be challenging to identify the key messages related to intervention evaluation from case study reports.

Whilst the diversity of published case studies in health services and public health research is rich and productive, we recommend further clarity and specific methodological guidance for those reporting case study research for evaluation audiences.

Peer Review reports

The need for methodological development to address the most urgent challenges in health research has been well-documented. Many of the most pressing questions for public health research, where the focus is on system-level determinants [ 1 , 2 ], and for health services research, where provisions typically vary across sites and are provided through interlocking networks of services [ 3 ], require methodological approaches that can attend to complexity. The need for methodological advance has arisen, in part, as a result of the diminishing returns from randomised controlled trials (RCTs) where they have been used to answer questions about the effects of interventions in complex systems [ 4 , 5 , 6 ]. In conditions of complexity, there is limited value in maintaining the current orientation to experimental trial designs in the health sciences as providing ‘gold standard’ evidence of effect.

There are increasing calls for methodological pluralism [ 7 , 8 ], with the recognition that complex intervention and context are not easily or usefully separated (as is often the situation when using trial design), and that system interruptions may have effects that are not reducible to linear causal pathways between intervention and outcome. These calls are reflected in a shifting and contested discourse of trial design, seen with the emergence of realist [ 9 ], adaptive and hybrid (types 1, 2 and 3) [ 10 , 11 ] trials that blend studies of effectiveness with a close consideration of the contexts of implementation. Similarly, process evaluation has now become a core component of complex healthcare intervention trials, reflected in MRC guidance on how to explore implementation, causal mechanisms and context [ 12 ].

Evidence about the context of an intervention is crucial for questions of external validity. As Woolcock [ 4 ] notes, even if RCT designs are accepted as robust for maximising internal validity, questions of transferability (how well the intervention works in different contexts) and generalisability (how well the intervention can be scaled up) remain unanswered [ 5 , 13 ]. For research evidence to have impact on policy and systems organisation, and thus to improve population and patient health, there is an urgent need for better methods for strengthening external validity, including a better understanding of the relationship between intervention and context [ 14 ].

Policymakers, healthcare commissioners and other research users require credible evidence of relevance to their settings and populations [ 15 ], to perform what Rosengarten and Savransky [ 16 ] call ‘careful abstraction’ to the locales that matter for them. They also require robust evidence for understanding complex causal pathways. Case study research, currently under-utilised in public health and health services evaluation, can offer considerable potential for strengthening faith in both external and internal validity. For example, in an empirical case study of how the policy of free bus travel had specific health effects in London, UK, a quasi-experimental evaluation (led by JG) identified how important aspects of context (a good public transport system) and intervention (that it was universal) were necessary conditions for the observed effects, thus providing useful, actionable evidence for decision-makers in other contexts [ 17 ].

The overall approach of case study research is based on the in-depth exploration of complex phenomena in their natural, or ‘real-life’, settings. Empirical case studies typically enable dynamic understanding of complex challenges rather than restricting the focus on narrow problem delineations and simple fixes. Case study research is a diverse and somewhat contested field, with multiple definitions and perspectives grounded in different ways of viewing the world, and involving different combinations of methods. In this paper, we raise awareness of such plurality and highlight the contribution that case study research can make to the evaluation of complex system-level interventions. We review some of the challenges in exploiting the current evidence base from empirical case studies and conclude by recommending that further guidance and minimum reporting criteria for evaluation using case studies, appropriate for audiences in the health sciences, can enhance the take-up of evidence from case study research.

Case study research offers evidence about context, causal inference in complex systems and implementation

Well-conducted and described empirical case studies provide evidence on context, complexity and mechanisms for understanding how, where and why interventions have their observed effects. Recognition of the importance of context for understanding the relationships between interventions and outcomes is hardly new. In 1943, Canguilhem berated an over-reliance on experimental designs for determining universal physiological laws: ‘As if one could determine a phenomenon’s essence apart from its conditions! As if conditions were a mask or frame which changed neither the face nor the picture!’ ([ 18 ] p126). More recently, a concern with context has been expressed in health systems and public health research as part of what has been called the ‘complexity turn’ [ 1 ]: a recognition that many of the most enduring challenges for developing an evidence base require a consideration of system-level effects [ 1 ] and the conceptualisation of interventions as interruptions in systems [ 19 ].

The case study approach is widely recognised as offering an invaluable resource for understanding the dynamic and evolving influence of context on complex, system-level interventions [ 20 , 21 , 22 , 23 ]. Empirically, case studies can directly inform assessments of where, when, how and for whom interventions might be successfully implemented, by helping to specify the necessary and sufficient conditions under which interventions might have effects and to consolidate learning on how interdependencies, emergence and unpredictability can be managed to achieve and sustain desired effects. Case study research has the potential to address four objectives for improving research and reporting of context recently set out by guidance on taking account of context in population health research [ 24 ], that is to (1) improve the appropriateness of intervention development for specific contexts, (2) improve understanding of ‘how’ interventions work, (3) better understand how and why impacts vary across contexts and (4) ensure reports of intervention studies are most useful for decision-makers and researchers.

However, evaluations of complex healthcare interventions have arguably not exploited the full potential of case study research and can learn much from other disciplines. For evaluative research, exploratory case studies have had a traditional role of providing data on ‘process’, or initial ‘hypothesis-generating’ scoping, but might also have an increasing salience for explanatory aims. Across the social and political sciences, different kinds of case studies are undertaken to meet diverse aims (description, exploration or explanation) and across different scales (from small N qualitative studies that aim to elucidate processes, or provide thick description, to more systematic techniques designed for medium-to-large N cases).

Case studies with explanatory aims vary in terms of their positioning within mixed-methods projects, with designs including (but not restricted to) (1) single N of 1 studies of interventions in specific contexts, where the overall design is a case study that may incorporate one or more (randomised or not) comparisons over time and between variables within the case; (2) a series of cases conducted or synthesised to provide explanation from variations between cases; and (3) case studies of particular settings within RCT or quasi-experimental designs to explore variation in effects or implementation.

Detailed qualitative research (typically done as ‘case studies’ within process evaluations) provides evidence for the plausibility of mechanisms [ 25 ], offering theoretical generalisations for how interventions may function under different conditions. Although RCT designs reduce many threats to internal validity, the mechanisms of effect remain opaque, particularly when the causal pathways between ‘intervention’ and ‘effect’ are long and potentially non-linear: case study research has a more fundamental role here, in providing detailed observational evidence for causal claims [ 26 ] as well as producing a rich, nuanced picture of tensions and multiple perspectives [ 8 ].

Longitudinal or cross-case analysis may be best suited for evidence generation in system-level evaluative research. Turner [ 27 ], for instance, reflecting on the complex processes in major system change, has argued for the need for methods that integrate learning across cases, to develop theoretical knowledge that would enable inferences beyond the single case, and to develop generalisable theory about organisational and structural change in health systems. Qualitative Comparative Analysis (QCA) [ 28 ] is one such formal method for deriving causal claims, using set theory mathematics to integrate data from empirical case studies to answer questions about the configurations of causal pathways linking conditions to outcomes [ 29 , 30 ].

Nonetheless, the single N case study, too, provides opportunities for theoretical development [ 31 ], and theoretical generalisation or analytical refinement [ 32 ]. How ‘the case’ and ‘context’ are conceptualised is crucial here. Findings from the single case may seem to be confined to its intrinsic particularities in a specific and distinct context [ 33 ]. However, if such context is viewed as exemplifying wider social and political forces, the single case can be ‘telling’, rather than ‘typical’, and offer insight into a wider issue [ 34 ]. Internal comparisons within the case can offer rich possibilities for logical inferences about causation [ 17 ]. Further, case studies of any size can be used for theory testing through refutation [ 22 ]. The potential lies, then, in utilising the strengths and plurality of case study to support theory-driven research within different methodological paradigms.

Evaluation research in health has much to learn from a range of social sciences where case study methodology has been used to develop various kinds of causal inference. For instance, Gerring [ 35 ] expands on the within-case variations utilised to make causal claims. For Gerring [ 35 ], case studies come into their own with regard to invariant or strong causal claims (such as X is a necessary and/or sufficient condition for Y) rather than for probabilistic causal claims. For the latter (where experimental methods might have an advantage in estimating effect sizes), case studies offer evidence on mechanisms: from observations of X affecting Y, from process tracing or from pattern matching. Case studies also support the study of emergent causation, that is, the multiple interacting properties that account for particular and unexpected outcomes in complex systems, such as in healthcare [ 8 ].

Finally, efficacy (or beliefs about efficacy) is not the only contributor to intervention uptake, with a range of organisational and policy contingencies affecting whether an intervention is likely to be rolled out in practice. Case study research is, therefore, invaluable for learning about contextual contingencies and identifying the conditions necessary for interventions to become normalised (i.e. implemented routinely) in practice [ 36 ].

The challenges in exploiting evidence from case study research

At present, there are significant challenges in exploiting the benefits of case study research in evaluative health research, which relate to status, definition and reporting. Case study research has been marginalised at the bottom of an evidence hierarchy, seen to offer little by way of explanatory power, if nonetheless useful for adding descriptive data on process or providing useful illustrations for policymakers [ 37 ]. This is an opportune moment to revisit this low status. As health researchers are increasingly charged with evaluating ‘natural experiments’—the use of face masks in the response to the COVID-19 pandemic being a recent example [ 38 ]—rather than interventions that take place in settings that can be controlled, research approaches using methods to strengthen causal inference that does not require randomisation become more relevant.

A second challenge for improving the use of case study evidence in evaluative health research is that, as we have seen, what is meant by ‘case study’ varies widely, not only across but also within disciplines. There is indeed little consensus amongst methodologists as to how to define ‘a case study’. Definitions focus, variously, on small sample size or lack of control over the intervention (e.g. [ 39 ] p194), on in-depth study and context [ 40 , 41 ], on the logic of inference used [ 35 ] or on distinct research strategies which incorporate a number of methods to address questions of ‘how’ and ‘why’ [ 42 ]. Moreover, definitions developed for specific disciplines do not capture the range of ways in which case study research is carried out across disciplines. Multiple definitions of case study reflect the richness and diversity of the approach. However, evidence suggests that a lack of consensus across methodologists results in some of the limitations of published reports of empirical case studies [ 43 , 44 ]. Hyett and colleagues [ 43 ], for instance, reviewing reports in qualitative journals, found little match between methodological definitions of case study research and how authors used the term.

This raises the third challenge we identify that case study reports are typically not written in ways that are accessible or useful for the evaluation research community and policymakers. Case studies may not appear in journals widely read by those in the health sciences, either because space constraints preclude the reporting of rich, thick descriptions, or because of the reported lack of willingness of some biomedical journals to publish research that uses qualitative methods [ 45 ], signalling the persistence of the aforementioned evidence hierarchy. Where they do, however, the term ‘case study’ is used to indicate, interchangeably, a qualitative study, an N of 1 sample, or a multi-method, in-depth analysis of one example from a population of phenomena. Definitions of what constitutes the ‘case’ are frequently lacking and appear to be used as a synonym for the settings in which the research is conducted. Despite offering insights for evaluation, the primary aims may not have been evaluative, so the implications may not be explicitly drawn out. Indeed, some case study reports might properly be aiming for thick description without necessarily seeking to inform about context or causality.

Acknowledging plurality and developing guidance

We recognise that definitional and methodological plurality is not only inevitable, but also a necessary and creative reflection of the very different epistemological and disciplinary origins of health researchers, and the aims they have in doing and reporting case study research. Indeed, to provide some clarity, Thomas [ 46 ] has suggested a typology of subject/purpose/approach/process for classifying aims (e.g. evaluative or exploratory), sample rationale and selection and methods for data generation of case studies. We also recognise that the diversity of methods used in case study research, and the necessary focus on narrative reporting, does not lend itself to straightforward development of formal quality or reporting criteria.

Existing checklists for reporting case study research from the social sciences—for example Lincoln and Guba’s [ 47 ] and Stake’s [ 33 ]—are primarily orientated to the quality of narrative produced, and the extent to which they encapsulate thick description, rather than the more pragmatic issues of implications for intervention effects. Those designed for clinical settings, such as the CARE (CAse REports) guidelines, provide specific reporting guidelines for medical case reports about single, or small groups of patients [ 48 ], not for case study research.

The Design of Case Study Research in Health Care (DESCARTE) model [ 44 ] suggests a series of questions to be asked of a case study researcher (including clarity about the philosophy underpinning their research), study design (with a focus on case definition) and analysis (to improve process). The model resembles toolkits for enhancing the quality and robustness of qualitative and mixed-methods research reporting, and it is usefully open-ended and non-prescriptive. However, even if it does include some reflections on context, the model does not fully address aspects of context, logic and causal inference that are perhaps most relevant for evaluative research in health.

Hence, for evaluative research where the aim is to report empirical findings in ways that are intended to be pragmatically useful for health policy and practice, this may be an opportune time to consider how to best navigate plurality around what is (minimally) important to report when publishing empirical case studies, especially with regards to the complex relationships between context and interventions, information that case study research is well placed to provide.

The conventional scientific quest for certainty, predictability and linear causality (maximised in RCT designs) has to be augmented by the study of uncertainty, unpredictability and emergent causality [ 8 ] in complex systems. This will require methodological pluralism, and openness to broadening the evidence base to better understand both causality in and the transferability of system change intervention [ 14 , 20 , 23 , 25 ]. Case study research evidence is essential, yet is currently under exploited in the health sciences. If evaluative health research is to move beyond the current impasse on methods for understanding interventions as interruptions in complex systems, we need to consider in more detail how researchers can conduct and report empirical case studies which do aim to elucidate the contextual factors which interact with interventions to produce particular effects. To this end, supported by the UK’s Medical Research Council, we are embracing the challenge to develop guidance for case study researchers studying complex interventions. Following a meta-narrative review of the literature, we are planning a Delphi study to inform guidance that will, at minimum, cover the value of case study research for evaluating the interrelationship between context and complex system-level interventions; for situating and defining ‘the case’, and generalising from case studies; as well as provide specific guidance on conducting, analysing and reporting case study research. Our hope is that such guidance can support researchers evaluating interventions in complex systems to better exploit the diversity and richness of case study research.

Availability of data and materials

Not applicable (article based on existing available academic publications)

Abbreviations

Qualitative comparative analysis

Quasi-experimental design

Randomised controlled trial

Diez Roux AV. Complex systems thinking and current impasses in health disparities research. Am J Public Health. 2011;101(9):1627–34.

Article   Google Scholar  

Ogilvie D, Mitchell R, Mutrie N, M P, Platt S. Evaluating health effects of transport interventions: methodologic case study. Am J Prev Med 2006;31:118–126.

Walshe C. The evaluation of complex interventions in palliative care: an exploration of the potential of case study research strategies. Palliat Med. 2011;25(8):774–81.

Woolcock M. Using case studies to explore the external validity of ‘complex’ development interventions. Evaluation. 2013;19:229–48.

Cartwright N. Are RCTs the gold standard? BioSocieties. 2007;2(1):11–20.

Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Soc Sci Med. 2018;210:2–21.

Salway S, Green J. Towards a critical complex systems approach to public health. Crit Public Health. 2017;27(5):523–4.

Greenhalgh T, Papoutsi C. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018;16(1):95.

Bonell C, Warren E, Fletcher A. Realist trials and the testing of context-mechanism-outcome configurations: a response to Van Belle et al. Trials. 2016;17:478.

Pallmann P, Bedding AW, Choodari-Oskooei B. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Med. 2018;16:29.

Curran G, Bauer M, Mittman B, Pyne J, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812 .

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015 [cited 2020 Jun 27];350. Available from: https://www.bmj.com/content/350/bmj.h1258 .

Evans RE, Craig P, Hoddinott P, Littlecott H, Moore L, Murphy S, et al. When and how do ‘effective’ interventions need to be adapted and/or re-evaluated in new contexts? The need for guidance. J Epidemiol Community Health. 2019;73(6):481–2.

Shoveller J. A critical examination of representations of context within research on population health interventions. Crit Public Health. 2016;26(5):487–500.

Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10(1):37.

Rosengarten M, Savransky M. A careful biomedicine? Generalization and abstraction in RCTs. Crit Public Health. 2019;29(2):181–91.

Green J, Roberts H, Petticrew M, Steinbach R, Goodman A, Jones A, et al. Integrating quasi-experimental and inductive designs in evaluation: a case study of the impact of free bus travel on public health. Evaluation. 2015;21(4):391–406.

Canguilhem G. The normal and the pathological. New York: Zone Books; 1991. (1949).

Google Scholar  

Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43:267–76.

King G, Keohane RO, Verba S. Designing social inquiry: scientific inference in qualitative research: Princeton University Press; 1994.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.

Yin R. Enhancing the quality of case studies in health services research. Health Serv Res. 1999;34(5 Pt 2):1209.

CAS   PubMed   PubMed Central   Google Scholar  

Raine R, Fitzpatrick R, Barratt H, Bevan G, Black N, Boaden R, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res. 2016 [cited 2020 Jun 30];4(16). Available from: https://www.journalslibrary.nihr.ac.uk/hsdr/hsdr04160#/abstract .

Craig P, Di Ruggiero E, Frohlich KL, E M, White M, Group CCGA. Taking account of context in population health intervention research: guidance for producers, users and funders of research. NIHR Evaluation, Trials and Studies Coordinating Centre; 2018.

Grant RL, Hood R. Complex systems, explanation and policy: implications of the crisis of replication for public health research. Crit Public Health. 2017;27(5):525–32.

Mahoney J. Strategies of causal inference in small-N analysis. Sociol Methods Res. 2000;4:387–424.

Turner S. Major system change: a management and organisational research perspective. In: Rosalind Raine, Ray Fitzpatrick, Helen Barratt, Gywn Bevan, Nick Black, Ruth Boaden, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res. 2016;4(16) 2016. https://doi.org/10.3310/hsdr04160.

Ragin CC. Using qualitative comparative analysis to study causal complexity. Health Serv Res. 1999;34(5 Pt 2):1225.

Hanckel B, Petticrew M, Thomas J, Green J. Protocol for a systematic review of the use of qualitative comparative analysis for evaluative questions in public health research. Syst Rev. 2019;8(1):252.

Schneider CQ, Wagemann C. Set-theoretic methods for the social sciences: a guide to qualitative comparative analysis: Cambridge University Press; 2012. 369 p.

Flyvbjerg B. Five misunderstandings about case-study research. Qual Inq. 2006;12:219–45.

Tsoukas H. Craving for generality and small-N studies: a Wittgensteinian approach towards the epistemology of the particular in organization and management studies. Sage Handb Organ Res Methods. 2009:285–301.

Stake RE. The art of case study research. London: Sage Publications Ltd; 1995.

Mitchell JC. Typicality and the case study. Ethnographic research: A guide to general conduct. Vol. 238241. 1984.

Gerring J. What is a case study and what is it good for? Am Polit Sci Rev. 2004;98(2):341–54.

May C, Mort M, Williams T, F M, Gask L. Health technology assessment in its local contexts: studies of telehealthcare. Soc Sci Med 2003;57:697–710.

McGill E. Trading quality for relevance: non-health decision-makers’ use of evidence on the social determinants of health. BMJ Open. 2015;5(4):007053.

Greenhalgh T. We can’t be 100% sure face masks work – but that shouldn’t stop us wearing them | Trish Greenhalgh. The Guardian. 2020 [cited 2020 Jun 27]; Available from: https://www.theguardian.com/commentisfree/2020/jun/05/face-masks-coronavirus .

Hammersley M. So, what are case studies? In: What’s wrong with ethnography? New York: Routledge; 1992.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11(1):100.

Luck L, Jackson D, Usher K. Case study: a bridge across the paradigms. Nurs Inq. 2006;13(2):103–9.

Yin RK. Case study research and applications: design and methods: Sage; 2017.

Hyett N, A K, Dickson-Swift V. Methodology or method? A critical review of qualitative case study reports. Int J Qual Stud Health Well-Being. 2014;9:23606.

Carolan CM, Forbat L, Smith A. Developing the DESCARTE model: the design of case study research in health care. Qual Health Res. 2016;26(5):626–39.

Greenhalgh T, Annandale E, Ashcroft R, Barlow J, Black N, Bleakley A, et al. An open letter to the BMJ editors on qualitative research. Bmj. 2016;352.

Thomas G. A typology for the case study in social science following a review of definition, discourse, and structure. Qual Inq. 2011;17(6):511–21.

Lincoln YS, Guba EG. Judging the quality of case study reports. Int J Qual Stud Educ. 1990;3(1):53–9.

Riley DS, Barber MS, Kienle GS, Aronson JK, Schoen-Angerer T, Tugwell P, et al. CARE guidelines for case reports: explanation and elaboration document. J Clin Epidemiol. 2017;89:218–35.

Download references

Acknowledgements

Not applicable

This work was funded by the Medical Research Council - MRC Award MR/S014632/1 HCS: Case study, Context and Complex interventions (TRIPLE C). SP was additionally funded by the University of Oxford's Higher Education Innovation Fund (HEIF).

Author information

Authors and affiliations.

Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK

Sara Paparini, Chrysanthi Papoutsi, Trish Greenhalgh & Sara Shaw

Wellcome Centre for Cultures & Environments of Health, University of Exeter, Exeter, UK

Judith Green

School of Health Sciences, University of East Anglia, Norwich, UK

Jamie Murdoch

Public Health, Environments and Society, London School of Hygiene & Tropical Medicin, London, UK

Mark Petticrew

Institute for Culture and Society, Western Sydney University, Penrith, Australia

Benjamin Hanckel

You can also search for this author in PubMed   Google Scholar

Contributions

JG, MP, SP, JM, TG, CP and SS drafted the initial paper; all authors contributed to the drafting of the final version, and read and approved the final manuscript.

Corresponding author

Correspondence to Sara Paparini .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Paparini, S., Green, J., Papoutsi, C. et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med 18 , 301 (2020). https://doi.org/10.1186/s12916-020-01777-6

Download citation

Received : 03 July 2020

Accepted : 07 September 2020

Published : 10 November 2020

DOI : https://doi.org/10.1186/s12916-020-01777-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative
• Case studies
• Mixed-method
• Public health
• Health services research
• Interventions

BMC Medicine

ISSN: 1741-7015

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Empirical Research: Definition, Methods, Types and Examples

Content Index

Empirical research: Definition

Empirical research: origin, quantitative research methods, qualitative research methods, steps for conducting empirical research, empirical research methodology cycle, advantages of empirical research, disadvantages of empirical research, why is there a need for empirical research.

Empirical research is defined as any research where conclusions of the study is strictly drawn from concretely empirical evidence, and therefore “verifiable” evidence.

This empirical evidence can be gathered using quantitative market research and  qualitative market research  methods.

For example: A research is being conducted to find out if listening to happy music while working may promote creativity? An experiment is conducted by using a music website survey on a set of audience who are exposed to happy music and another set who are not listening to music at all, and the subjects are then observed. The results derived from such a research will give empirical evidence if it does promote creativity or not.

LEARN ABOUT: Behavioral Research

You must have heard the quote” I will not believe it unless I see it”. This came from the ancient empiricists, a fundamental understanding that powered the emergence of medieval science during the renaissance period and laid the foundation of modern science, as we know it today. The word itself has its roots in greek. It is derived from the greek word empeirikos which means “experienced”.

In today’s world, the word empirical refers to collection of data using evidence that is collected through observation or experience or by using calibrated scientific instruments. All of the above origins have one thing in common which is dependence of observation and experiments to collect data and test them to come up with conclusions.

LEARN ABOUT: Causal Research

Types and methodologies of empirical research

Empirical research can be conducted and analysed using qualitative or quantitative methods.

• Quantitative research : Quantitative research methods are used to gather information through numerical data. It is used to quantify opinions, behaviors or other defined variables . These are predetermined and are in a more structured format. Some of the commonly used methods are survey, longitudinal studies, polls, etc
• Qualitative research:   Qualitative research methods are used to gather non numerical data.  It is used to find meanings, opinions, or the underlying reasons from its subjects. These methods are unstructured or semi structured. The sample size for such a research is usually small and it is a conversational type of method to provide more insight or in-depth information about the problem Some of the most popular forms of methods are focus groups, experiments, interviews, etc.

Data collected from these will need to be analysed. Empirical evidence can also be analysed either quantitatively and qualitatively. Using this, the researcher can answer empirical questions which have to be clearly defined and answerable with the findings he has got. The type of research design used will vary depending on the field in which it is going to be used. Many of them might choose to do a collective research involving quantitative and qualitative method to better answer questions which cannot be studied in a laboratory setting.

LEARN ABOUT: Qualitative Research Questions and Questionnaires

Quantitative research methods aid in analyzing the empirical evidence gathered. By using these a researcher can find out if his hypothesis is supported or not.

• Survey research: Survey research generally involves a large audience to collect a large amount of data. This is a quantitative method having a predetermined set of closed questions which are pretty easy to answer. Because of the simplicity of such a method, high responses are achieved. It is one of the most commonly used methods for all kinds of research in today’s world.

Previously, surveys were taken face to face only with maybe a recorder. However, with advancement in technology and for ease, new mediums such as emails , or social media have emerged.

For example: Depletion of energy resources is a growing concern and hence there is a need for awareness about renewable energy. According to recent studies, fossil fuels still account for around 80% of energy consumption in the United States. Even though there is a rise in the use of green energy every year, there are certain parameters because of which the general population is still not opting for green energy. In order to understand why, a survey can be conducted to gather opinions of the general population about green energy and the factors that influence their choice of switching to renewable energy. Such a survey can help institutions or governing bodies to promote appropriate awareness and incentive schemes to push the use of greener energy.

Learn more: Renewable Energy Survey Template Descriptive Research vs Correlational Research

• Experimental research: In experimental research , an experiment is set up and a hypothesis is tested by creating a situation in which one of the variable is manipulated. This is also used to check cause and effect. It is tested to see what happens to the independent variable if the other one is removed or altered. The process for such a method is usually proposing a hypothesis, experimenting on it, analyzing the findings and reporting the findings to understand if it supports the theory or not.

For example: A particular product company is trying to find what is the reason for them to not be able to capture the market. So the organisation makes changes in each one of the processes like manufacturing, marketing, sales and operations. Through the experiment they understand that sales training directly impacts the market coverage for their product. If the person is trained well, then the product will have better coverage.

• Correlational research: Correlational research is used to find relation between two set of variables . Regression analysis is generally used to predict outcomes of such a method. It can be positive, negative or neutral correlation.

LEARN ABOUT: Level of Analysis

For example: Higher educated individuals will get higher paying jobs. This means higher education enables the individual to high paying job and less education will lead to lower paying jobs.

• Longitudinal study: Longitudinal study is used to understand the traits or behavior of a subject under observation after repeatedly testing the subject over a period of time. Data collected from such a method can be qualitative or quantitative in nature.

For example: A research to find out benefits of exercise. The target is asked to exercise everyday for a particular period of time and the results show higher endurance, stamina, and muscle growth. This supports the fact that exercise benefits an individual body.

• Cross sectional: Cross sectional study is an observational type of method, in which a set of audience is observed at a given point in time. In this type, the set of people are chosen in a fashion which depicts similarity in all the variables except the one which is being researched. This type does not enable the researcher to establish a cause and effect relationship as it is not observed for a continuous time period. It is majorly used by healthcare sector or the retail industry.

For example: A medical study to find the prevalence of under-nutrition disorders in kids of a given population. This will involve looking at a wide range of parameters like age, ethnicity, location, incomes  and social backgrounds. If a significant number of kids coming from poor families show under-nutrition disorders, the researcher can further investigate into it. Usually a cross sectional study is followed by a longitudinal study to find out the exact reason.

• Causal-Comparative research : This method is based on comparison. It is mainly used to find out cause-effect relationship between two variables or even multiple variables.

For example: A researcher measured the productivity of employees in a company which gave breaks to the employees during work and compared that to the employees of the company which did not give breaks at all.

LEARN ABOUT: Action Research

Some research questions need to be analysed qualitatively, as quantitative methods are not applicable there. In many cases, in-depth information is needed or a researcher may need to observe a target audience behavior, hence the results needed are in a descriptive analysis form. Qualitative research results will be descriptive rather than predictive. It enables the researcher to build or support theories for future potential quantitative research. In such a situation qualitative research methods are used to derive a conclusion to support the theory or hypothesis being studied.

LEARN ABOUT: Qualitative Interview

• Case study: Case study method is used to find more information through carefully analyzing existing cases. It is very often used for business research or to gather empirical evidence for investigation purpose. It is a method to investigate a problem within its real life context through existing cases. The researcher has to carefully analyse making sure the parameter and variables in the existing case are the same as to the case that is being investigated. Using the findings from the case study, conclusions can be drawn regarding the topic that is being studied.

For example: A report mentioning the solution provided by a company to its client. The challenges they faced during initiation and deployment, the findings of the case and solutions they offered for the problems. Such case studies are used by most companies as it forms an empirical evidence for the company to promote in order to get more business.

• Observational method:   Observational method is a process to observe and gather data from its target. Since it is a qualitative method it is time consuming and very personal. It can be said that observational research method is a part of ethnographic research which is also used to gather empirical evidence. This is usually a qualitative form of research, however in some cases it can be quantitative as well depending on what is being studied.

For example: setting up a research to observe a particular animal in the rain-forests of amazon. Such a research usually take a lot of time as observation has to be done for a set amount of time to study patterns or behavior of the subject. Another example used widely nowadays is to observe people shopping in a mall to figure out buying behavior of consumers.

• One-on-one interview: Such a method is purely qualitative and one of the most widely used. The reason being it enables a researcher get precise meaningful data if the right questions are asked. It is a conversational method where in-depth data can be gathered depending on where the conversation leads.

For example: A one-on-one interview with the finance minister to gather data on financial policies of the country and its implications on the public.

• Focus groups: Focus groups are used when a researcher wants to find answers to why, what and how questions. A small group is generally chosen for such a method and it is not necessary to interact with the group in person. A moderator is generally needed in case the group is being addressed in person. This is widely used by product companies to collect data about their brands and the product.

For example: A mobile phone manufacturer wanting to have a feedback on the dimensions of one of their models which is yet to be launched. Such studies help the company meet the demand of the customer and position their model appropriately in the market.

• Text analysis: Text analysis method is a little new compared to the other types. Such a method is used to analyse social life by going through images or words used by the individual. In today’s world, with social media playing a major part of everyone’s life, such a method enables the research to follow the pattern that relates to his study.

For example: A lot of companies ask for feedback from the customer in detail mentioning how satisfied are they with their customer support team. Such data enables the researcher to take appropriate decisions to make their support team better.

Sometimes a combination of the methods is also needed for some questions that cannot be answered using only one type of method especially when a researcher needs to gain a complete understanding of complex subject matter.

We recently published a blog that talks about examples of qualitative data in education ; why don’t you check it out for more ideas?

Since empirical research is based on observation and capturing experiences, it is important to plan the steps to conduct the experiment and how to analyse it. This will enable the researcher to resolve problems or obstacles which can occur during the experiment.

Step #1: Define the purpose of the research

This is the step where the researcher has to answer questions like what exactly do I want to find out? What is the problem statement? Are there any issues in terms of the availability of knowledge, data, time or resources. Will this research be more beneficial than what it will cost.

Before going ahead, a researcher has to clearly define his purpose for the research and set up a plan to carry out further tasks.

Step #2 : Supporting theories and relevant literature

The researcher needs to find out if there are theories which can be linked to his research problem . He has to figure out if any theory can help him support his findings. All kind of relevant literature will help the researcher to find if there are others who have researched this before, or what are the problems faced during this research. The researcher will also have to set up assumptions and also find out if there is any history regarding his research problem

Step #3: Creation of Hypothesis and measurement

Before beginning the actual research he needs to provide himself a working hypothesis or guess what will be the probable result. Researcher has to set up variables, decide the environment for the research and find out how can he relate between the variables.

Researcher will also need to define the units of measurements, tolerable degree for errors, and find out if the measurement chosen will be acceptable by others.

Step #4: Methodology, research design and data collection

In this step, the researcher has to define a strategy for conducting his research. He has to set up experiments to collect data which will enable him to propose the hypothesis. The researcher will decide whether he will need experimental or non experimental method for conducting the research. The type of research design will vary depending on the field in which the research is being conducted. Last but not the least, the researcher will have to find out parameters that will affect the validity of the research design. Data collection will need to be done by choosing appropriate samples depending on the research question. To carry out the research, he can use one of the many sampling techniques. Once data collection is complete, researcher will have empirical data which needs to be analysed.

LEARN ABOUT: Best Data Collection Tools

Step #5: Data Analysis and result

Data analysis can be done in two ways, qualitatively and quantitatively. Researcher will need to find out what qualitative method or quantitative method will be needed or will he need a combination of both. Depending on the unit of analysis of his data, he will know if his hypothesis is supported or rejected. Analyzing this data is the most important part to support his hypothesis.

Step #6: Conclusion

A report will need to be made with the findings of the research. The researcher can give the theories and literature that support his research. He can make suggestions or recommendations for further research on his topic.

A.D. de Groot, a famous dutch psychologist and a chess expert conducted some of the most notable experiments using chess in the 1940’s. During his study, he came up with a cycle which is consistent and now widely used to conduct empirical research. It consists of 5 phases with each phase being as important as the next one. The empirical cycle captures the process of coming up with hypothesis about how certain subjects work or behave and then testing these hypothesis against empirical data in a systematic and rigorous approach. It can be said that it characterizes the deductive approach to science. Following is the empirical cycle.

• Observation: At this phase an idea is sparked for proposing a hypothesis. During this phase empirical data is gathered using observation. For example: a particular species of flower bloom in a different color only during a specific season.
• Induction: Inductive reasoning is then carried out to form a general conclusion from the data gathered through observation. For example: As stated above it is observed that the species of flower blooms in a different color during a specific season. A researcher may ask a question “does the temperature in the season cause the color change in the flower?” He can assume that is the case, however it is a mere conjecture and hence an experiment needs to be set up to support this hypothesis. So he tags a few set of flowers kept at a different temperature and observes if they still change the color?
• Deduction: This phase helps the researcher to deduce a conclusion out of his experiment. This has to be based on logic and rationality to come up with specific unbiased results.For example: In the experiment, if the tagged flowers in a different temperature environment do not change the color then it can be concluded that temperature plays a role in changing the color of the bloom.
• Testing: This phase involves the researcher to return to empirical methods to put his hypothesis to the test. The researcher now needs to make sense of his data and hence needs to use statistical analysis plans to determine the temperature and bloom color relationship. If the researcher finds out that most flowers bloom a different color when exposed to the certain temperature and the others do not when the temperature is different, he has found support to his hypothesis. Please note this not proof but just a support to his hypothesis.
• Evaluation: This phase is generally forgotten by most but is an important one to keep gaining knowledge. During this phase the researcher puts forth the data he has collected, the support argument and his conclusion. The researcher also states the limitations for the experiment and his hypothesis and suggests tips for others to pick it up and continue a more in-depth research for others in the future. LEARN MORE: Population vs Sample

LEARN MORE: Population vs Sample

There is a reason why empirical research is one of the most widely used method. There are a few advantages associated with it. Following are a few of them.

• It is used to authenticate traditional research through various experiments and observations.
• This research methodology makes the research being conducted more competent and authentic.
• It enables a researcher understand the dynamic changes that can happen and change his strategy accordingly.
• The level of control in such a research is high so the researcher can control multiple variables.
• It plays a vital role in increasing internal validity .

Even though empirical research makes the research more competent and authentic, it does have a few disadvantages. Following are a few of them.

• Such a research needs patience as it can be very time consuming. The researcher has to collect data from multiple sources and the parameters involved are quite a few, which will lead to a time consuming research.
• Most of the time, a researcher will need to conduct research at different locations or in different environments, this can lead to an expensive affair.
• There are a few rules in which experiments can be performed and hence permissions are needed. Many a times, it is very difficult to get certain permissions to carry out different methods of this research.
• Collection of data can be a problem sometimes, as it has to be collected from a variety of sources through different methods.

LEARN ABOUT:  Social Communication Questionnaire

Empirical research is important in today’s world because most people believe in something only that they can see, hear or experience. It is used to validate multiple hypothesis and increase human knowledge and continue doing it to keep advancing in various fields.

For example: Pharmaceutical companies use empirical research to try out a specific drug on controlled groups or random groups to study the effect and cause. This way, they prove certain theories they had proposed for the specific drug. Such research is very important as sometimes it can lead to finding a cure for a disease that has existed for many years. It is useful in science and many other fields like history, social sciences, business, etc.

LEARN ABOUT: 12 Best Tools for Researchers

With the advancement in today’s world, empirical research has become critical and a norm in many fields to support their hypothesis and gain more knowledge. The methods mentioned above are very useful for carrying out such research. However, a number of new methods will keep coming up as the nature of new investigative questions keeps getting unique or changing.

Create a single source of real data with a built-for-insights platform. Store past data, add nuggets of insights, and import research data from various sources into a CRM for insights. Build on ever-growing research with a real-time dashboard in a unified research management platform to turn insights into knowledge.

LEARN MORE         FREE TRIAL

MORE LIKE THIS

Stripe’s Role in Achieving Exceptional Customer Experience

Sep 1, 2023

Navigating Blind Spots in Leadership with 360 Feedback Surveys

Aug 31, 2023

Qualitative Data Analysis: What is it, Methods + Examples

Square Feedback: Importance in A Business’s Customer Journey

Other categories.

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• [email protected]
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Uncategorized
• Video Learning Series
• What’s Coming Up
• Workforce Intelligence

• What is Empirical Research Study? [Examples & Method]

The bulk of human decisions relies on evidence, that is, what can be measured or proven as valid. In choosing between plausible alternatives, individuals are more likely to tilt towards the option that is proven to work, and this is the same approach adopted in empirical research. 

In empirical research, the researcher arrives at outcomes by testing his or her empirical evidence using qualitative or quantitative methods of observation, as determined by the nature of the research. An empirical research study is set apart from other research approaches by its methodology and features hence; it is important for every researcher to know what constitutes this investigation method. 

What is Empirical Research? 

Empirical research is a type of research methodology that makes use of verifiable evidence in order to arrive at research outcomes. In other words, this  type of research relies solely on evidence obtained through observation or scientific data collection methods. 

Empirical research can be carried out using qualitative or quantitative observation methods , depending on the data sample, that is, quantifiable data or non-numerical data . Unlike theoretical research that depends on preconceived notions about the research variables, empirical research carries a scientific investigation to measure the experimental probability of the research variables 

Characteristics of Empirical Research

• Research Questions

An empirical research begins with a set of research questions that guide the investigation. In many cases, these research questions constitute the research hypothesis which is tested using qualitative and quantitative methods as dictated by the nature of the research.

In an empirical research study, the research questions are built around the core of the research, that is, the central issue which the research seeks to resolve. They also determine the course of the research by highlighting the specific objectives and aims of the systematic investigation. 

• Definition of the Research Variables

The research variables are clearly defined in terms of their population, types, characteristics, and behaviors. In other words, the data sample is clearly delimited and placed within the context of the research. 

• Description of the Research Methodology

 An empirical research also clearly outlines the methods adopted in the systematic investigation. Here, the research process is described in detail including the selection criteria for the data sample, qualitative or quantitative research methods plus testing instruments. 

An empirical research is usually divided into 4 parts which are the introduction, methodology, findings, and discussions. The introduction provides a background of the empirical study while the methodology describes the research design, processes, and tools for the systematic investigation. 

The findings refer to the research outcomes and they can be outlined as statistical data or in the form of information obtained through the qualitative observation of research variables. The discussions highlight the significance of the study and its contributions to knowledge. 

Uses of Empirical Research

Without any doubt, empirical research is one of the most useful methods of systematic investigation. It can be used for validating multiple research hypotheses in different fields including Law, Medicine, and Anthropology. 

• Empirical Research in Law : In Law, empirical research is used to study institutions, rules, procedures, and personnel of the law, with a view to understanding how they operate and what effects they have. It makes use of direct methods rather than secondary sources, and this helps you to arrive at more valid conclusions. 
• Empirical Research in Medicine : In medicine, empirical research is used to test and validate multiple hypotheses and increase human knowledge. 
• Empirical Research in Anthropology : In anthropology, empirical research is used as an evidence-based systematic method of inquiry into patterns of human behaviors and cultures. This helps to validate and advance human knowledge. 

The Empirical Research Cycle

The empirical research cycle is a 5-phase cycle that outlines the systematic processes for conducting and empirical research. It was developed by Dutch psychologist, A.D. de Groot in the 1940s and it aligns 5 important stages that can be viewed as deductive approaches to empirical research. 

In the empirical research methodological cycle, all processes are interconnected and none of the processes is more important than the other. This cycle clearly outlines the different phases involved in generating the research hypotheses and testing these hypotheses systematically using the empirical data. 

• Observation:This is the process of gathering empirical data for the research. At this stage, the researcher gathers relevant empirical data using qualitative or quantitative observation methods, and this goes ahead to inform the research hypotheses. 
• Induction: At this stage, the researcher makes use of inductive reasoning in order to arrive at a general probable research conclusion based on his or her observation. The researcher generates a general assumption that attempts to explain the empirical data and s/he goes on to observe the empirical data in line with this assumption.  
• Deduction: This is the deductive reasoning stage. This is where the researcher generates hypotheses by applying logic and rationality to his or her observation. 
• Testing: Here, the researcher puts the hypotheses to test using qualitative or quantitative research methods. In the testing stage, the researcher combines relevant instruments of systematic investigation with empirical methods in order to arrive at objective results that support or negate the research hypotheses. 
• Evaluation: The evaluation research is the final stage in an empirical research study. Here, the research outlines the empirical data, the research findings and the supporting arguments plus any challenges encountered during the research process. 

This information is useful for further research. 

Examples of Empirical Research 

• An empirical research study can be carried out to determine if listening to happy music improves the mood of individuals. The researcher may need to conduct an experiment that involves exposing individuals to happy music to see if this improves their moods.

The findings from such an experiment will provide empirical evidence that confirms or refutes the hypotheses. 

• An empirical research study can also be carried out to determine the effects of a new drug on specific groups of people. The researcher may expose the research subjects to controlled quantities of the drug and observe research subjects to controlled quantities of the drug and observe the effects over a specific period of time in order to gather empirical data. 
• Another example of empirical research is measuring the levels of noise pollution found in an urban area to determine the average levels of sound exposure experienced by its inhabitants. Here, the researcher may have to administer questionnaires or carry out a survey in order to gather relevant data based on the experiences of the research subjects. 
• Empirical research can also be carried out to determine the relationship between seasonal migration and the body mass of flying birds. A researcher may need to observe the birds and carry out necessary observation and experimentation in order to arrive at objective outcomes that answer the research question. 

Empirical Research Data Collection Methods

Empirical data can be gathered using qualitative and quantitative data collection methods. Quantitative data collection methods are used for numerical data gathering while qualitative data collection processes are used to gather empirical data that cannot be quantified, that is, non-numerical data. 

The following are common methods of gathering data in empirical research

• Survey/ Questionnaire

A survey is a method of data gathering that is typically employed by researchers to gather large sets of data from a specific number of respondents with regards to a research subject. This method of data gathering is often used for quantitative data collection , although it can also be deployed during quantitative research.  

A survey contains a set of questions that can range from close-ended to open-ended questions together with other question types that revolve around the research subject. A survey can be administered physically or with the use of online data-gathering platforms like Formplus. 

Empirical data can also be collected by carrying out an experiment. An experiment is a controlled simulation in which one or more of the research variables is manipulated using a set of interconnected processes in order to confirm or refute the research hypotheses. 

An experiment is a useful method of measuring causality; that is cause and effect between dependent and independent variables in a research environment. It is an integral data gathering method in an empirical research study because it involves testing calculated assumptions in order to arrive at the most valid data and research outcomes. 

T he case study method is another common data gathering method in an empirical research study. It involves sifting through and analyzing relevant cases and real-life experiences about the research subject or research variables in order to discover in-depth information that can serve as empirical data. 

• Observation

The observational method is a method of qualitative data gathering that requires the researcher to study the behaviors of research variables in their natural environments in order to gather relevant information that can serve as empirical data. 

How to collect Empirical Research Data with Questionnaire

With Formplus, you can create a survey or questionnaire for collecting empirical data from your research subjects. Formplus also offers multiple form sharing options so that you can share your empirical research survey to research subjects via a variety of methods.

Here is a step-by-step guide of how to collect empirical data using Formplus:

Sign in to Formplus

In the Formplus builder, you can easily create your empirical research survey by dragging and dropping preferred fields into your form. To access the Formplus builder, you will need to create an account on Formplus. 

Once you do this, sign in to your account and click on “Create Form ” to begin. 

Edit Form Title

Click on the field provided to input your form title, for example, “Empirical Research Survey”.

Edit Form  

• Click on the edit button to edit the form.
• Add Fields: Drag and drop preferred form fields into your form in the Formplus builder inputs column. There are several field input options for survey forms in the Formplus builder. 
• Edit fields
• Click on “Save”
• Preview form. 

Customize Form

Formplus allows you to add unique features to your empirical research survey form. You can personalize your survey using various customization options. Here, you can add background images, your organization’s logo, and use other styling options. You can also change the display theme of your form. 

• Share your Form Link with Respondents

Formplus offers multiple form sharing options which enables you to easily share your empirical research survey form with respondents. You can use the direct social media sharing buttons to share your form link to your organization’s social media pages. 

You can send out your survey form as email invitations to your research subjects too. If you wish, you can share your form’s QR code or embed it on your organization’s website for easy access. 

Empirical vs Non-Empirical Research

Empirical and non-empirical research are common methods of systematic investigation employed by researchers. Unlike empirical research that tests hypotheses in order to arrive at valid research outcomes, non-empirical research theorizes the logical assumptions of research variables. 

Definition: Empirical research is a research approach that makes use of evidence-based data while non-empirical research is a research approach that makes use of theoretical data. 

Method: In empirical research, the researcher arrives at valid outcomes by mainly observing research variables, creating a hypothesis and experimenting on research variables to confirm or refute the hypothesis. In non-empirical research, the researcher relies on inductive and deductive reasoning to theorize logical assumptions about the research subjects.

The major difference between the research methodology of empirical and non-empirical research is while the assumptions are tested in empirical research, they are entirely theorized in non-empirical research. 

Data Sample: Empirical research makes use of empirical data while non-empirical research does not make use of empirical data. Empirical data refers to information that is gathered through experience or observation. 

Unlike empirical research, theoretical or non-empirical research does not rely on data gathered through evidence. Rather, it works with logical assumptions and beliefs about the research subject. 

Data Collection Methods : Empirical research makes use of quantitative and qualitative data gathering methods which may include surveys, experiments, and methods of observation. This helps the researcher to gather empirical data, that is, data backed by evidence.  

Non-empirical research, on the other hand, does not make use of qualitative or quantitative methods of data collection . Instead, the researcher gathers relevant data through critical studies, systematic review and meta-analysis. 

Advantages of Empirical Research 

• Empirical research is flexible. In this type of systematic investigation, the researcher can adjust the research methodology including the data sample size, data gathering methods plus the data analysis methods as necessitated by the research process. 
• It helps the research to understand how the research outcomes can be influenced by different research environments.
• Empirical research study helps the researcher to develop relevant analytical and observation skills that can be useful in dynamic research contexts. 
• This type of research approach allows the researcher to control multiple research variables in order to arrive at the most relevant research outcomes. 
• Empirical research is widely considered as one of the most authentic and competent research designs. 
• It improves the internal validity of traditional research using a variety of experiments and research observation methods. 

Disadvantages of Empirical Research 

• An empirical research study is time-consuming because the researcher needs to gather the empirical data from multiple resources which typically takes a lot of time.
• It is not a cost-effective research approach. Usually, this method of research incurs a lot of cost because of the monetary demands of the field research. 
• It may be difficult to gather the needed empirical data sample because of the multiple data gathering methods employed in an empirical research study. 
• It may be difficult to gain access to some communities and firms during the data gathering process and this can affect the validity of the research.
• The report from an empirical research study is intensive and can be very lengthy in nature. 

Conclusion 

Empirical research is an important method of systematic investigation because it gives the researcher the opportunity to test the validity of different assumptions, in the form of hypotheses, before arriving at any findings. Hence, it is a more research approach. 

There are different quantitative and qualitative methods of data gathering employed during an empirical research study based on the purpose of the research which include surveys, experiments, and various observatory methods. Surveys are one of the most common methods or empirical data collection and they can be administered online or physically. 

You can use Formplus to create and administer your online empirical research survey. Formplus allows you to create survey forms that you can share with target respondents in order to obtain valuable feedback about your research context, question or subject. 

In the form builder, you can add different fields to your survey form and you can also modify these form fields to suit your research process. Sign up to Formplus to access the form builder and start creating powerful online empirical research survey forms. 

Collect and analyze empirical research data with Formplus

• advantage of empirical research
• disadvantages of empirical resarch
• empirical research characteristics
• empirical research cycle
• empirical research method
• example of empirical research
• uses of empirical research
• Temitope Ayanyemi

You may also like:

Extrapolation in Statistical Research: Definition, Examples, Types, Applications

In this article we’ll look at the different types and characteristics of extrapolation, plus how it contrasts to interpolation.

Research Questions: Definitions, Types + [Examples]

A comprehensive guide on the definition of research questions, types, importance, good and bad research question examples

Recall Bias: Definition, Types, Examples & Mitigation

This article will discuss the impact of recall bias in studies and the best ways to avoid them during research.

What is Pure or Basic Research? + [Examples & Method]

Simple guide on pure or basic research, its methods, characteristics, advantages, and examples in science, medicine, education and psychology

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

The encyclopedia for writers, researchers, and knowledge workers

• APA – 7th Edition
• Collaboration
• Information Literacy
• MLA – 9th Edition
• Organization
• Writing Studies
• The Writing Process – Research on Composing
• Fake News: an Undergraduate Composition Course
• Professional Writing Course Schedule
• How to Cite Sources in Academic and Professional Writing
• How to Navigate Writing Commons
• How to Shape Information for Maximum Impact
• How to Write for the Academic Community
• How to Write for the Professional World
• How To Revise Your Work
• Teacher’s Guide
• Writer’s Guide
• Aaron Swartz Award
• Contribute – Write for Us
• Dear Past Authors
• Project History & Acknowledgements
• Student Voices
• Terms of Service & Privacy
•   Joseph M. Moxley

What is a Case Study?

Case Studies are in-depth investigations of an individual or event. Clinicians use interviews, participant observations, and archival information (e.g., medical information) to develop robust portraits of others and the circumstances that led individuals to act.

Some Case Study researchers assume the method produces positivistic knowledge whereas others argue it produces postpositivistic knowledge.

The design of your case study is determined by your purpose .

Writers conduct case studies for many reasons. Researchers may conduct interviews to achieve multiple purposes:

• interview people who can tell stories about life in the past.
• interview experts, such as famous inventors, entrepreneurs, political leaders, or trend-setters
• interview “man/woman on the street,” profiling the life of “ordinary people”
• perhaps someone said something in a clever way that supports your work

Some researchers argue that their interviews of individuals can be used to generalize to broader populations. For example, an urban sociologist might interview gang members and then try to generalize to other gangs, other cities. In contrast, some researchers argue that interviews can only generate knowledge about individuals, that researchers who use interviews are simply telling stories

Suggest an edit to this page?

Please share your feedback, both favorable and unfavorable.

• Please select the purpose of your message. * - Corrections, Typos, or Edits Technical Support/Problems using the site Advertising with Writing Commons Copyright Issues I am contacting you about something else
• Your full name
• Your email address *
• Page URL needing edits *
• Comments This field is for validation purposes and should be left unchanged.

Empirical Tools of Research: Case Studies, Interviewing and Questionnaire Survey

In socio-legal research, collecting data through observation is extensively used for empirical analysis. Empirical legal research has gained contemporary prominence due to the increasing value and importance of the study of ‘law m action’ instead of ‘law in books.

Empirical legal scholarship involves applying social science research methods-whether qualitative, quantitative, or mixed methods to gather and describe data and test hypotheses related to legal issues.

Empirical legal scholarship can help to test conventional wisdom, dispel myths, and provide new findings.

It enhances a lawyer’s ability to understand the implications and effects of the law on society. There is an increasing recognition that law students should know a wide range of research methodologies to understand the law’s function in society and the law’s nature and foundational principles.

As socio-legal research involves the analysis of social behavior , observation is the most appropriate technique. Case studies, questionnaire surveys, and interviews are considered important tools for such observation.

However, case studies and interviews are qualitative, while questionnaire surveys are quantitative. These tools involve collecting data about people, institutions, and their social context.

Empirical research requires linking data to concepts and connecting a concept to its empirical findings. In contrast to the traditional methods, the empirical approach tests legal propositions in their proper setting, that is, the socio-political conditions.

These are frequently used as a data collection method for substantiating the hypothesis of the research project .

Thus, empirical tools are important in testing the validity of the hypothesis framed.

The empirical approach stresses the link between research questions and data. The choice of data collection method may depend on personal preference, cost, time constraints, potential response rate, or many other factors important to a particular research project.

What is the significance of empirical legal research?

Empirical legal research is significant because it involves applying social science research methods to gather and describe data and test hypotheses related to legal issues. It helps test conventional wisdom, dispel myths, and provide new findings, enhancing a lawyer’s understanding of the law’s implications and effects on society.

What are the primary tools used in socio-legal research for observation?

The primary tools used in socio-legal research for observation are case studies, questionnaire surveys, and interviews. These tools are essential for collecting data about people, institutions, and their social context.

Case Studies

The case study is significant for empirical research and a corresponding sense of relevance to contemporary social problems.

The need for a case study arises from the desire to understand complex social phenomena, allowing an investigation to retain the holistic and meaningful characteristics of real-life events.

Case studies emphasize detailed contextual analysis of a limited number of events or conditions and their relationships.

The case study is defined as an empirical inquiry that investigates a contemporary phenomenon within its real-life context or impact when the boundaries between phenomenon and context are not clearly evident. It can be an exploration of something unique, special, or interesting.

A case study also seeks evidence which supports the findings of the research.

The case study method explores and analyzes the individual life, single situation, institution, particular group, or community, which are too complex for the survey or experimental strategies. The main feature of a case study is that it examines individual instances, or cases, of some phenomenon.

The case study method generally calls for the researcher to choose from some possible events, people, organizations, etc. Then, the researcher needs to pick out one example from a wider range of examples of the class of things being investigated.

The principal advantage of a case study is the opportunity to research the real-life context of practice, technique, system, or phenomena.

Case studies are focused on circumstantial uniqueness. Case studies may require a description of relevant provisions and contexts. Case studies may be used to test the hypothesis.

As a result, conclusions are not based on the author’s judgment but on the results revealed by the case studies. The case study method depends upon the narrative type of description of life situations. It takes into consideration comparatively fewer cases and aims at more intensive study.

An in-depth case study can provide an understanding of the important aspects of a new or persistently problematic research area.

Discovering the important features, developing an understanding of them, and conceptualizing them for further study, are often best achieved through the case study strategy.

However, the main concerns regarding the case study have been the subjective bias of the researcher in choosing it and lack of generalization.

• First, the researcher may be biased and subjective in selecting a case study to support or refute his argument.
• The second criticism is that case studies provide very little evidence of inference or generalization.

Case studies may be single or multiple. A single case study is applied to testing a well-founded theory to determine whether its propositions are correct.

It is also applied when a case or situation is a unique one. The multiple or collective case study covers several cases to learn more about the phenomenon, population, or general condition.

A study may contain more than one case study to produce more compelling empirical evidence.

However, each case must be carefully selected to produce results that either support or contradict the initial set of propositions. If results are contradictory, the initial propositions must be revised and retested with another set of cases.

Evidence of case study may come from documents, archival records, interviews, and direct observation by the researcher (field visit).

However, one should always keep in mind how to place the findings of a case study in research work or contribute to the thesis. A case study must collect pervasive data to understand the topic being studied to make a meaningful contribution.

Before undertaking a case study, the following things should be considered: availability of data, confidentiality, the cost involved in collecting data, and the cultural context of the collection; data collected should be consistent and scientific.

Careful attention should also be paid to the quality of data.

For example, analysis of data derived from the case study can be based on theoretical propositions. The theoretical propositions can help the researcher to focus on certain data and to ignore other data.

How is a case study defined, and what is its significance in empirical research?

A case study is defined as an empirical inquiry that investigates a contemporary phenomenon within its real-life context or impact, especially when the boundaries between the phenomenon and context are unclear. It is significant for empirical research because it allows for an in-depth understanding of complex social phenomena, retaining the holistic and meaningful characteristics of real-life events.

Interviewing

The interview is the most frequently used technique for obtaining information and qualitative data. Interviewing is a particular type of conversation where one person seeks responses for a particular purpose from the other person: the interviewee.

It is seen as an effective, informal verbal and nonverbal conversation initiated for specific purposes and focused on certain issues.

The personal interview is a face-to-face interpersonal role situation designed to elicit answers pertinent to the research hypotheses.

The interview is purposive communication between two persons and a psychological process of social interaction. Thus, the interview is also an inter-actional process.

Therefore, the use of interviews can help in gathering valid and reliable data that are relevant to research questions.

The interview’s objectives also include exchanging ideas and experiences and eliciting information about a wide range of data in which the interviewee is well conversant.

An interview based on a well-thought-out questionnaire is a useful tool for research. An interview is a flexible research tool used at any stage of the research work.

However, the interviewer must keep in sight the hypothesis of the research. This purpose determines the form and style of an interview. Interviewing as a data collection method can serve three purposes.

• Firstly, it can be used as an exploratory device to help identify variables and relations.
• Secondly, it can be the main instrument of research.
• Thirdly, it can supplement other data-collection methods to investigate the situation at hand.

Interviews are conducted face to face with the obvious benefit of observing the interviewee’s verbal and non-verbal behavior.

An interview can also be conducted over the phone or by email. Phone or email interviews offer the opportunity to conduct more interviews within the same time frame and draw interviewees from a wider geographical area.

The first step in preparing for interviews is identifying the information you want to gather.

Using the interviewing method to collect data has many advantages.

First, this method helps the researcher explore and understand complex issues that could not be stated m the questionnaire survey.

Interviews are beneficial for producing data that deals with topics in depth and in detail.

Thus, the researcher can gain valuable insights based on the depth of the information gathered.

Another major advantage of the interview is that it allows the interviewer to determine the wording of questions, clarify unclear terms, and control the order in which the questions are presented.

Also, carrying out a personal interview enables the researcher to probe for additional and detailed data.

Finally, an interviewer can collect supplementary information about respondents in an interview situation to aid the researcher in interpreting the results.

However, there are undeniable shortcomings in the interview method.

The cost is significantly higher than that of the self-administrated questionnaire regarding the skills needed, training, analysis, and especially when interviews are spread over a large geographical area.

Furthermore, the interviewer’s personal influence and bias can affect the interview.

Finally, the interview lacks the anonymity of the self-administrated questionnaire . Thus, the interviewer’s presence may jeopardize anonymity, especially if a respondent is sensitive to the topic or some of the questions.

Thus, the main disadvantages of interviewing are: it is open to bias, poor reliability of the information, time-consuming, and getting access to suitable respondents.

One should keep in mind who are potential interviewees. Are they individuals, corporations, policymakers, including government officials, lawmakers, NGO personalities?

One must be careful in selecting interviewees to get comprehensive and reliable information. The researcher should obtain as much background information as possible on the interviewees and the research issues.

The researcher should establish trust and resolve confidentiality issues. The questionnaire should be linked to research questions and relevant to the conceptual framework.

The interview’s success depends on the interviewer’s capacity to build rapport with the respondent, the right type of questions should be asked in the right manner, and the recording of the responses properly and accurately at the time of the interview.

In addition, rapport building with the respondent requires a thorough understanding of the respondent and his social environment.

The interviewer should be friendly, courteous, conversational, and unbiased. A good interviewer should also be attentive and sensitive to the interviewee’s feelings.

He should explain the study’s purpose and create confidence that the information will be kept in confidence. The researcher should try to know in advance, if possible, what kind of information he is looking for.

The interviewer needs to be flexible, objective, emphatic, persuasive, and a good listener. In addition, the interviewer must have a thorough knowledge of the nature of the problem, its various aspects, and the importance of the study.

You can use two methods to record the interview responses:

• Note-taking: Interviewers should plan to take notes during the interview and directly after.
• Tape recording: interviewers can also use a tape recorder to document what key informants say. This approach allows the interviewer to engage freely in the conversation without worrying about note-taking.

The interviewer may take brief notes during the interview, write down and organize notes at the end, and use the tape recording to fill in information gaps or details.

Getting informed consent from the key informant is necessary to audiotape the interview. The interview will be recorded so that none of their important insights and discussions are missed.

Types of interview

These are the main types of interviews;

• Structured Interview,
• Unstructured Interview ,

Focused Interview

Focused group discussion (fgd).

Check out our article types of interviews which explains 10 types of interviews .

Structured Interview

It refers to formal, controlled, guided interviews. The interview is carried out based on pre-determined questions similar in format to a questionnaire survey.

Thus, a structured interview means all the questions are decided precisely in advance. The structured interview involves a prescribed set of questions that the researcher asks in a fixed order and generally requires the interviewee to respond in a standardized way.

It means that some questions are presented to all the respondents in the same order.

The questions are set out in a close-ended way, giving alternative responses for the respondents’ options. The main advantage of a structured interview is that it provides uniformity in generalizations.

However, it tends to be rigid and mechanical sometimes. Because the questions limit the respondent’s answer, he is asked.

Consequently, the structured interview may not get all the information even though the respondent may be willing to provide it.

Unstructured Interview

The unstructured interview is based on flexible and open-ended questions. The interviewer bases his interview on purpose rather than the form.

The interviewer is given more freedom to choose the form of interview depending on specific situations. It is generally held in the form of free discussion or a story-type narrative. In fact, it relies on developing a dialogue between interviewer and interviewee.

The interviewer is informed of the topic and invited to comment. The wording and the sequence of questions are changed, keeping in view the response pattern. This open-ended approach of unstructured interviews requires creativity from the interviewer.

It allows the researcher to address any topic that may interest the researcher. But interviewee needs sufficient knowledge and skills to maintain focus and link with research questions.

The main advantage of open-ended interviews is that responses are flexible, and in-depth answers may be provided.

The main disadvantages of unstructured interviews are that the responses’ analysis is much more difficult and time-consuming. It also demands deep knowledge and skill on the part of the interviewer .

In a focused interview, the objective is to focus attention on the given experience of the respondent who is known to have been involved in a particular situation.

Thus, the interviewer tries to focus on the particular aspect of the problem and to know his experiences, attitudes, and emotional responses regarding the concrete situation under study.

The focused interview is differentiated from other types of interviews by the following characteristics:

• firstly, it takes place with persons known to have been involved in a particular concrete situation;
• secondly, it refers to situations that have been analyzed before the interview;
• thirdly, it is focused on the subjective experiences- attitudes and emotional responses regarding the particular concrete situations under study.

It is a systematic questioning or interview of several individuals simultaneously in formal or informal settings. FGD is gaining popularity among legal researchers as it can provide a perspective on the research problem not available through individual interviews.

It is a process where group members talk freely and spontaneously about a topic guided by the interviewer. Its purpose is to obtain in-depth information on a group’s concepts, perceptions, and ideas.

Thus, FGD aims to be more than a question-answer interaction. It helps develop relevant research hypotheses by exploring the problem to be investigated and its possible causes in greater depth.

FGD has been described as some form of collective activity, and it provides multiple lines of communication and an environment for people to share ideas and experiences. In addition, FGD provides a situation where people can leam and educate others.

FGD has the advantages of becoming inexpensive, generating rich data, flexibility, and stimulating respondents. However, this type of interview is not without problems.

One person may dominate the group. The researcher has less control over a group than a one-on-one interview. It also requires a highly skilled and trained interviewer or facilitator to conduct the discussion.

Interview of Key Informants

The interview of key informants is also known as an in-depth interview.

In-depth interviewing is a qualitative research technique involving intensive individual interviews with a small number of respondents to explore their perspectives on a particular idea, program, or situation.

In-depth interviews are useful when one wants detailed information about a person’s thoughts and behaviors or wants to explore new issues in depth.

A key informant is someone who has first-hand knowledge of the information you need or issues you seek to address or who can offer specific, specialized knowledge on a particular issue that the researcher wants to understand better.

A key informant can frequently offer a particular perspective of an issue or problem and give more candid or in-depth answers.

The purpose of key informant interviews is to collect information from a wide range of people- including community leaders, experts, professionals, or government officials – who have first-hand knowledge about the community or a pressing issue or problem.

With their particular knowledge and understanding, these experts can provide insight into the nature of problems and give solutions.

Questionnaire Survey

A questionnaire survey is a structured data-collection technique whereby each respondent is asked a pre-formulated written set of questions to which he gives his answers.

The term questionnaire is defined as “any structured research instrument used to collect social research data in a face-to-face interview, self-completion survey, telephone interview or web survey.”

The main objective of a questionnaire survey is to investigate social problems, conditions, and structures within a definite geographical limit to collect scientific and well-ordered information.

Generally, the findings of a questionnaire survey provide researchers a valuable tool to assess a problem’s qualitative and quantitative aspects.

The questionnaire survey involves gathering data from a sample of a large, diverse, varied, and scattered population from different places.

The researcher collects data directly from people about their feelings, motivation, plans, beliefs, and personal, educational, and financial background in the questionnaire survey method.

This questionnaire survey design aims to generalize from a sample to a population so that inferences can be made about some characteristic, behavior, or attitude of the total population.

There are three criteria for evaluating a questionnaire survey: an assessment of the likelihood that the questionnaire will provide full information on the particular research topic.

The value of the questionnaire will depend significantly on the extent to which it includes coverage of all vital information in the area of research.

The second criterion concerns the likelihood that the questionnaire will provide accurate information. Again, it depends considerably on how honest and full the responses to the questionnaire.

Thirdly, the questionnaire can be evaluated according to its likelihood of achieving a decent response rate.

Before undertaking a questionnaire survey, it is necessary to select a problem, define the aim and purpose of the survey, and determine time limits for its completion.

But the identification of an appropriate sample of the population is vital as the survey may cover different people, situations, and institutions that are not absolutely similar to each other.

Therefore, the basic assumption is that the sample should represent the group as a whole, and the sample selection should be unbiased.

Usually, the questionnaire is mailed to the respondents, who are to give answers in the manner specified.

The sender does not meet and help the respondent in filling out the questionnaire. It should be noted that a questionnaire can contain both questions and evaluative statements. Statements are useful for discerning and measuring the attitudes of the respondents.

In the case of statements, the respondent may be asked to indicate his agreement or disagreement or make his own evaluation of the issue raised.

The questionnaire may contain either closed-ended or open-ended, or it can be a mixture of both. A list of pre-determined fixed alternative answers is provided in a closed-ended question, and the respondents are asked to choose an answer closest to their own opinion from the list.

Thus, closed-ended questions provide greater precision and uniformity of responses. Closed-ended questions also enhance the comparability of answers. But one obvious disadvantage of it is the lack of spontaneity in respondents’ answers.

On the other hand, in the open-ended questionnaire, the respondents have the flexibility to provide their own opinions. Therefore, the open-ended questionnaire can provide a wide variety of responses, which may be helpful for the researcher for an in-depth understanding of the whole issue.

But open-ended questions also present problems for the researcher. First, they are time-consuming for researchers to administer. In general, a good questionnaire should contain both categories of questions in varying proportions.

There is no hard and fast rule about the number of questions in a questionnaire. It depends on factors like the topic, how complex the questions are, and the nature of the targeted respondents.

However, a questionnaire should contain only those questions crucial to the research.

Sources of the questionnaire can be drawn from the literature, a similar questionnaire employed in other countries areas of research, an earlier questionnaire used in the same country, the issue to be tackled in the research, and the individuals to be interviewed.

Guidelines for Designing Questionnaire

• Questions should be relevant to most respondents. The wording of questions should be specific so that the respondents know exactly what the researcher want.
• Questions should avoid technical jargon because the respondents are usually drawn from people of varied backgrounds.
• A mix of open-ended and closed questions should be used to generate better findings of the study.
• The length of the questionnaire should be kept to a minimum. A questionnaire should cover only those issues necessary for pursuing the research objectives .
• The distribution of the questions should be carefully planned.
• In some cases, the identity of the respondents should be anonymous.
• Logical sequences should be maintained in framing questions.
• Leading questions, personal questions, and complex questions should be avoided.
• Simple and uncontroversial questions should be given first, and more complicated questions should follow.
• Avoid double-barreled questions. It means a question that has multiple parts.

Before setting a questionnaire, the researcher should decide what he is testing: hypothesis or extant literature. There should be explanatory notes with interpretation to answer difficult questions wherever necessary.

The questionnaire survey is a method of collecting data and, like any other method, has many advantages and disadvantages.

Advantages of Questionnaire Survey

The main advantages of a questionnaire survey are that it can ascertain the views of a large sample of individuals, obtain both quantitative and qualitative views, generalize from the sample responses, and the research can be performed relatively quickly.

The following are some of the advantages of a questionnaire survey:

• respondents have more flexibility in answering. They can take more time to collect detailed information required for the questionnaire and/or consult other sources. Furthermore, the respondents can answer the survey at their convenience.
• Data can be gathered from a sample questionnaire that was widely dispersed geographically.
• The absence of an interviewer provides greater anonymity for the respondent. The assurance of anonymity that a questionnaire provides is beneficial when the survey deals with sensitive issues, and non-disclosure of the respondents’ identity facilitates better questionnaire findings.
• It allows for large numbers of respondents to be surveyed in a relatively short period.
• Questionnaire surveys provide greater uniformity than interviews. Everyone responds to the same questions, which ensures a degree of consistency in the findings.
• It produces data that can easily be expressed in statistical form. This enables comparisons to be made between different groups and populations.
• If the survey is properly conducted, the results are reliable and representative of a much wider population than directly investigated.

Disadvantages of Questionnaire Survey

The main disadvantages of a questionnaire survey are:

• Low response rates have always been a problem with a questionnaire survey. The respondent may not be able or willing to answer the question;
• The researcher may get a biased set of responses.
• It may be costly when the respondents are spread all over the country.
• The answers must be accepted as final; the researcher cannot correct a misunderstanding, help the respondent, or probe for further information.

Despite the above disadvantages, the questionnaire survey is probably the best method for collecting from a population too large to observe directly. However, data collected from the questionnaire can be useful when properly interpreted and analyzed.

As described above, each empirical study method has its advantages and disadvantages. To overcome this problem, one method may be supplemented by another.

A combination of a questionnaire survey, case study, and interviewing often yields the best results, but the balance of emphasis on a particular method shifts with the frame of reference and the study’s objectives.

What are the different types of interviews used in research?

The main types of interviews include Structured Interviews, Unstructured Interviews, Focused Interviews, Focused Group Discussions (FGD), and Interviews of Key Informants.

What is the purpose of a questionnaire survey in research?

A questionnaire survey aims to investigate social problems, conditions, and structures within a specific geographical limit to collect scientific and well-ordered information. It involves gathering data from a sample of a large, diverse, and scattered population about their feelings, motivations, beliefs, and backgrounds.

What factors should be considered when designing a questionnaire for a survey?

When designing a questionnaire, one should ensure questions are relevant specific, and avoid technical jargon. A mix of open-ended and closed questions should be used, and the length should be kept minimal. Logical sequences should be maintained, and leading or complex questions should be avoided. The researcher should also decide what they are testing, be it a hypothesis or extant literature.

As you now covered empirical tools of research; check out explore complete guideline on legal research and research and research methodology concepts .

• Ethics in Research: Plagiarism and Academic Dishonesty in Research
• Legal Research Design and Structure [11 Elements]
• Legal Writing Meaning [Legal Research and Legal Writing]
• Legal Research Methodology: Types And Approaches of Legal Research
• Referencing: Meaning, Types of Reference Styles and Systems
• 9 Steps in Legal Research
• Academic Legal Writing: Techniques, Rules, Tips
• Legal Research Statutes: Principles and Canons of Interpretation of Statutes
• Legal Research: Meaning, Definitions, and Example
• Legal Reasoning: Criteria and Forms of Legal Reasoning
• Research Proposal: Components, Format, Structure, Sample, Example
• 9 Sources of Legal Research
• Data Analysis and Interpretation
• Data Collection: Meaning, Types
• Sampling Methods: Techniques, Types, Examples
• Experimental Research Design: Types, Examples, Methods
• Kuder-Richardson Formula
• Engineering Research: Definition, Examples
• Stratified Random Sampling: Procedure, Types, Examples
• Education Research: Definition, Examples
• Observation Method of Data Collection: Advantages, Disadvantages, Techniques, Types
• Evaluative Research: Definition, Examples
• Thurstone Scale: Definition, Example
• Semantic Differential: Definition, Example
• Validity in Experimentation
• Research Process: 8 Steps in Research Process
• Design Effect: Definition, Examples
• Population Research: Definition, Examples
• Writing a Research Report
• Level of Measurement: 4 Scales of Measurement
• Exploratory Research: Definition, Types, Examples
• Monitoring and Evaluation: Process, Design, Methods

Systematic empirical research versus clinical case studies: a valid antagonism?

Affiliation.

• 1 Derner Institute for Advanced Psychological Studies, Adelphi University, Los Angeles, USA.
• PMID: 21836152
• DOI: 10.1177/0003065111416652

This paper considers the issue of systematic empirical research versus clinical case studies raised by Hoffman (2009). A rebuttal of Hoffman's arguments is offered, followed by an argument that each method addresses itself to different questions and that posing them in opposition is not fruitful. Finally, criteria and requirements of the case study method are proposed that, if met, would enhance its evidential value.

• Empirical Research*
• Psychoanalysis
• Research Design*

• Write my thesis
• Thesis writers
• Buy thesis papers
• Bachelor thesis
• Master's thesis
• Thesis editing services
• Thesis proofreading services
• Buy a thesis online
• Write my dissertation
• Dissertation proposal help
• Pay for dissertation
• Custom dissertation
• Dissertation help online
• Buy dissertation online
• Cheap dissertation
• Dissertation editing services
• Write my research paper
• Buy research paper online
• Pay for research paper
• Research paper help
• Order research paper
• Custom research paper
• Cheap research paper
• Research papers for sale
• Thesis subjects
• How It Works

What Is Empirical Research? Definition And Tips For Students

So what is an empirical research article? What does empirical research entail? What are good examples of empirical research topics? These are common questions most students find themselves asking whenever a research paper instructs them to read and analyze empirical articles or evidence to answer a particular question. Such questions also arise when it comes to graduation papers and many students just ask writers  to help. If this points out your current dilemma, then you are in good company because below is a well-detailed guide on all you need to know about empirical research.

Definition of Empirical Research

Empirical research is a type of research whose findings or conclusions are mainly drawn from empirical or verifiable evidence rather than rationality.

Simply put, empirical research is any research whose findings are based on observable or experimentation evidence rather than through reasoning or logic alone. For instance, if a scientist wants to find out whether soft music promotes sleep, he or she will find a group of people and divide them into two groups.

The first group will be in a space with soft music while the second one will be in an area with no music at all for observation. Once the experiment is over, the results or conclusion drawn from it will be termed as empirical evidence whether or not soft music promotes sleep.

The Objectives of Empirical Research

There is more to empirical research than just observations. In fact, these observations will be useless if the scientist does not turn them into testable questions. So, now that you know what empirical research entails, what are the objectives? Well, the main reason why you’ll be asked to conduct such research is so you can use the findings drawn from your observations to answer well-defined questions that go with or against a particular hypothesis.

In the earlier mentioned example, for instance, the hypothesis would be “soft music promotes sleep.” Thus, the aim of carrying out empirical research, in this case, would be to come up with conclusions that either accept or reject the hypotheses.

Now to achieve this, you can either use quantitative or qualitative methodologies. Here is a breakdown of what each of the mentioned entails

Quantitative Methods

Quantitative empirical research methods are often used to collect information and draw conclusions through numerical data. Quantitative methods are usually predetermined and are set in a more structured format. Some good examples include

• Longitudinal studies
• Cross-sectional studies
• Experimental research
• Causal comparative research

The above techniques are often more effective for physics or medicine.

Qualitative Methods

Qualitative empirical research methods, on the other hand, are used to gather non-numerical data. They are mainly unstructured or semi-structured and are used to find meaning or underlying reasons for a particular phenomenon. In other words, they are used to provide more insight into the problem being researched. So, is qualitative research empirical? The short answer to that is yes. It is indeed empirical but more appropriate in finding answers to social science-related questions.

Types of Empirical Research

Note, there is a difference between methodologies and types of empirical research. Methodologies are the earlier mentioned quantitative and qualitative, and they refer to the methods used to conduct or analyze data. But when it comes to types of empirical research, it can be either experimental or non-experimental.

In experimental research, a particular intervention is often used to drive a hypothesized changed in the variables of interest. In non-experimental empirical research, however, the subjects of the study are simply observed without any form of intervention. This is why it’s also referred to as informal research.

How to Identify Empirical Research Articles

Now that you know the types of empirical research as well as methodologies, how do you distinguish an empirical research paper from a regular research paper? Well, as with any other research question, empirical research questions usually have features that can help you identify them as shown below

Empirical Research Article Format

The easiest way to identify an empirical research article is through its unique format. It’s usually divided into these sections

• The Title. It offers an overview of the research and also includes the author(s) who conducted it
• An abstract. It provides a very brief yet comprehensive summary of the empirical research study. It’s usually a paragraph long.
• The Introduction. Here the author provides background information on the research problem. For instance, they talk about similar studies and explain why the research was conducted in the first place.
• The MethodsIn this section, the author offers a detailed description of all methods used to conduct the empirical research study. In some articles, it may be titled methodology.
• The Results. Here the author provides the answer to the research question.
• DiscussionThe author will then go on to give a detailed discussion of the data obtained or the results found above. They may also compare the results of their empirical research study to the results obtained by other empirical studies on similar topics.
• ReferencesHere as you may have guessed, the author lists citations of any journal articles, studies, or books mentioned or used in coming up with the results.

Keywords Used in an Empirical Research Paper

Other than the format, you can also identify an empirical research paper based on the phrases used within it. Some of the must-have keywords include:

• Measure or measurements
• Qualitative and quantitative research
• Sample size
• Methodology
• Original study or research study

Empirical Research Examples

Some of the empirical questions you could come across in empirical research include:

• According to Maslow’s hierarchy of needs, at which level are, leaders need level?
• The lop-sided sales of illegal guns among licensed handgun retailers.
• Freeway truck travel-time prediction for seamless freight planning using GPS data.
• Use empirical research to explain the entrepreneurial mindset concerning cognition and motivation.
• Are meta-analysis reviews theoretical or empirical research?

Theoretical vs. Empirical Research

Unlike empirical research, which is based on valid or observational evidence, theoretical research is more logical than observational. It is a logical exploration of a system of assumptions and involves defining how a particular system and its environment behave without the analysis of concrete data.

Therefore, theoretical research can be classified as non-empirical data as it is conducted without data. It is essential as it offers anyone conducting research a place to start. For instance, saying soft music promotes sleep gives the researcher a hypothesis to base their research proposal on and an easy way to start. However, theoretical data is only useful if empirical research is conducted.

Leave a Reply Cancel reply

In order to continue enjoying our site, we ask that you confirm your identity as a human. Thank you very much for your cooperation.

• Open Access
• Published: 01 September 2023

Is gambling like a virus?: A conceptual framework and proposals based on empirical data for the prevention of gambling addiction

• Mariano Chóliz 1  

BMC Public Health volume  23 , Article number:  1686 ( 2023 ) Cite this article

145 Accesses

7 Altmetric

Metrics details

The objective of this study is to present a conceptual framework for the prevention of gambling disorder and try to corroborate some of its postulates. The assumption of gambling as if it were acting like a virus may have important considerations in terms of preventing gambling disorder in society and, therefore, it could be a relevant public health issue.

Like COVID-19, gambling disorder is a disease which is caused by the action of an external agent. The external agent was already in existence, but certain environmental conditions (absence of regulatory measures based on the prevention of gambling disorder) favored its propagation. Regarding immunization, for SARS-CoV-2, it is obtained through vaccination and prevention of exposure. However, it is unlikely that immunization can be developed for any gambling addiction prevention program to immunize everyone who is exposed to the “gambling virus”. So, in the case of gambling disorder, preventive strategies should rather prevent gambling from affecting most people by limiting availability (supply) and accessibility (ease of access) to gambling.

Study design

This research is a quasi-experimental investigation aimed to evaluate the effects of anti-COVID measures on the frequency of gambling and evolution of gambling disorder. The present study analyzed gambling patterns and the problems caused by gambling in 2,903 people, including those who were at-risk gamblers or had a gambling disorder.

In general terms, restrictive measures to combat COVID-19 worked to prevent the consolidation of gambling habits and the appearance of gambling disorder, but they did not seem to be sufficient for people who already had this disorder. The most affected games were electronic games machines (EGMs) that took place in public places (gambling halls, bars and restaurants, etc.).

Conclusions

The findings of this work support the hypothesis that, just as the SAR-CoV-2 virus is responsible for the global pandemic of COVID-19, which can only be stopped with vaccines and social distancing, in the case of gambling, the absence of an effective vaccine for "gambling virus" can lead to an epidemic of gambling disorders in societies, if the environmental conditions that are favorable for the spread of such virus are not modified. Some preventive strategies that can be useful from a public health frame of reference are suggested.

Peer Review reports

Introduction

The assumption of gambling as if it were acting like a virus may have important considerations in terms of preventing gambling disorder in society and, therefore, it is a relevant public health issue. So, the comparison between gambling and SARS-CoV-2 seems appropriate to guide health policies that aim to prevent gambling disorder, just as they have been taken worldwide for the prevention of COVID-19.

Gambling disorder shares some characteristics of infectious viral spread, such as that of the COVID-19 pandemic. The consideration of gambling as a virus is metaphorical, but it seems adequate for describing the increasing prevalence of gambling disorder in countries where gambling has been legalized and promoted; just as for COVID-19, preventive measures must be developed for this disorder.

First, gambling disorder shares some of the characteristics of infectious spread caused by viral transmission with COVID-19. Some of the most relevant include the following:

Like COVID-19, gambling disorder is a disease. Not all psychological problems are considered illnesses. Only the psychological problems listed in DSM-5-TR or ICD-11 are considered mental disorders [ 1 , 2 ].

The disease is caused by the action of an external agent. The agent of COVID-19 is SARS-CoV-2, whereas the activity of betting itself is ultimately responsible for the genesis of gambling disease. This assertion is based on the guidelines in the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition Text Revision) of the American Psychiatric Association (APA), which states that “ gambling behaviors activate reward systems similar to those activated by drugs of abuse and produce some behavioral symptoms that appear comparable to those produced by substance use disorders” (DSM-5-TR, p. 543) [ 1 ].

On the contrary, many other mental illnesses (i.e., schizophrenia and psychotic, bipolar, obsessive-compulsive, neurocognitive, and personality disorders, etc.) are not typically caused by an external agent.

The external agent was already in existence, but certain environmental conditions favored its propagation, which then occurred to a greater extent with a greater speed. SARS-CoV-2 jumped from other animals to humans and spread extremely quickly. In the case of gambling, it has always been present, but when economic interests and favorable regulations generate a “breeding ground,” its expansion in society is favored. Global commercial gambling has grown to be an industry of extraordinary size and power [ 3 ]. The effects of the expansion of gambling not only harm the most vulnerable people [ 4 ], but also condition government policies, affecting society in general [ 5 ].

This turns gambling disorder from a mental health problem into a public health problem [ 6 ], since they are the environmental conditions that favor the appearance, development and spread of gambling disorder. Not all mental disorders caused by an external agent are a public health problem (i.e., trauma and stressor-related disorders, feeding and eating disorders, etc.). For that reason, gambling addiction requires policy action to prevent harm [ 7 , 8 ], mainly reduce availability, make access difficult and restrict (or forbid) the commercial promotions [ 9 ].

Second, if gambling is a disease that is transmitted due to favorable environmental conditions, which is why it has become a public health problem, it is worth asking whether the principles upon which measures to prevent the spread of COVID-19 are based would be useful in preventing gambling addiction in society.

Prevention of COVID-19 is based on two principles: immunization against the virus and prevention of the contagion.

Regarding immunization, for SARS-CoV-2, it is obtained through vaccination. The effect of virus inoculation in provoking the body's autoimmune response is well known in Medicine. However, there is nothing quite like it in Psychology when it comes to gambling, since gambling a bit (even "responsibly") does not prevent the onset of gambling disorder [ 10 ]. Rather, on the contrary, it favors the spread of the disease because, with responsible gambling actions, governments and gambling companies make the gambling look better [ 11 , 12 ]. So, it is unlikely that immunization can be developed for any gambling addiction prevention program to immunize everyone who is exposed to the “gambling virus”. Actually, the psychological resources that could immunize anyone involved in gambling are unknown. But even if those resources were discovered, what would not be possible is to train all citizens in such skills, contrary to what has happened with the vaccination of SARS-CoV-2.

Thus, in the case of gambling disorder, preventive strategies should rather reflect the second tier of action against COVID-19, that is, prevent gambling from affecting most people by limiting availability (supply) and accessibility (ease of access) to gambling [ 9 ]. This is especially important for the more dangerous variants of gambling, such as electronic gaming machines (EGMs) and online gambling [ 13 , 14 ]. Unlike the variants of SARS-CoV-2, in the case of gambling we can identify previously where the different variants of "gambling virus" are, which would allow us to implement appropriate preventive measures for specific games. Likewise, just as there are less contagious and lethal variants of SARS-CoV-2, there are also games, such as lotteries, that are less addictive and harmful than EGMs and various types of online gambling. In the case of COVID-19, the danger posed depends on the DNA structure whereas, for gambling, the structural characteristics of the games are the most important factors [ 15 , 16 , 17 ]. Therefore, measures to prevent gambling addiction must be adapted to each type of game.

However, once a person has been exposed to the effects of gambling, the next phase of prevention (selective prevention) would be to control the effect that gambling has on people who risk their money; i.e., recognizing the appearance of symptoms and acting effectively in response. As with the vaccine, it is not possible to train all gamblers to carry out gambling behaviors that prevent the development of gambling disorder. It is not possible for players to develop responsible gambling behaviors if the conditions in which gambling is offered in society do not drastically change [ 18 ]. In the case of selective prevention, again it must be public health policies that must be implemented.

Probably the most effectives preventive strategies in selective prevention are to limit losses and prevent affected people’s access to gambling [ 18 ]. In these cases, governmental regulation of gambling seems essential, because those affected are not able to reduce their exposure to gambling, nor are companies interested in reducing their income, which mainly comes from people who suffer from gambling addiction [ 19 ].

Finally, once a person has been infected and suffers from gambling disorder, it is necessary to use other measures beyond access control or limit losses. Gambling disorder is a clinical phenomenon [ 20 ] characterized by a loss of control over behavior that results not only in spending excessive amounts of money but also in alterations in emotional adjustment and interpersonal relationships. Psychological treatments for gambling disorder should not only reduce or eliminate gambling behavior, but also promote other alternatives that favor a new lifestyle without gambling [ 21 ]. Behavior modification techniques have been shown to be effective in reducing or eliminating excessive behaviors, training in coping techniques and in promoting alternative adaptive behaviors [ 21 , 22 , 23 ]. This is the only way to immunize against the effects of the “gambling virus”, but it is not a universal prevention procedure, since it is not possible to "immunize" the entire population in this way, but only patients undergoing psychological treatment. Effective preventive measures for the entire population must be carried out through gambling policies, that is, through gambling regulation [ 8 ].

In this sense, the effect on the pattern of gambling and gambling problems of the measures carried out for the prevention of COVID-19 can guide legislators and governments on the specific measures that must be taken to prevent gambling disorder from a public health perspective [ 15 ].

In a recent systematic review of 34 studies from 12 countries [ 24 ], it was concluded an overall reduction in gambling amongst the general population during the COVID-19 pandemic at the level of the general population. However, marked increases in gambling amongst young adults (18–30 year olds) and people with pre-existing at-risk gambling. There was conflicting evidence among the different studies regarding educational, employment status or socioeconomic level.

The main objective of the research is to describe the changes in gambling patterns and addiction that have occurred in Spain one year after the lockdown was implemented to counteract the COVID-19 pandemic. The results of this study analyzed from the conceptual framework that we have just described, will serve to guide gambling policies based on public health.

The first research hypothesis is that the frequency of gambling will decrease because the measures to prevent COVID-19 also restrict access to gambling. However, such measures will not affect all types of gambling equally, only those types that take place in public spaces (e.g., gambling halls, casinos, etc.). Online gambling via electronic devices (e.g., mobile phones, computers, and tablets) will not be affected.

The second hypothesis is that the type of game is relevant when it comes to causing addiction, due to the structural characteristics of the different games. Therefore, people who play landscape gambling and online gambling (e.g., casinos, bingo, and slots online) are more likely to suffer from gambling disorder than those who play lotteries.

Participants

In total, 2,903 people (55.6% women and 44.4% men) between the ages of 15 and 85 (Mean = 36.5; SD = 14.6) years participated in this study by responding to an Internet survey during the period from May–November, 2021. The survey was distributed over the Internet by 251 professionals and attendees of gambling addiction prevention training courses from several regions of Spain. The participants knew the objective of the research and freely agreed to participate.

Instruments

Gambling participation.

A survey on gambling behavior was administered. In this survey, participation in gambling before and after the measures taken to combat the COVID-19 pandemic was evaluated by self-report. The results were categorized into three groups based on the restriction conditions applied by government authorities aiming to prevent COVID-19, as follows:

No restrictions: online gambling .

Moderate restrictions: lotteries . There were 2 months without lottery draws at the beginning of the restriction period. After the restriction period, the lotteries returned to pre-pandemic conditions.

Severe restrictions: landscape gambling . For several months access to some game types was prevented and subsequently the capacity of gaming halls was limited.

Gambling problems

Gambling participation and gambling problems before and after the measures taken to minimize SARS-CoV-2 virus transmission were evaluated in the same survey. To avoid response bias, two different diagnostic questionnaires were used, both of which met the necessary methodological requirements:

Brief Problem Gambling Screen [ 25 ]. A five-item questionnaire to identify people who suffer from gambling disorder and at-risk gambling. The psychometric analysis of the scale performed with the data from this study showed adequate internal consistency ( Cronbach α = .76).

NORC DSM-IV Screen for Gambling Problems, NODS [ 26 ]. A 17-item yes/no scale that aims to diagnose pathological gambling according to the diagnostic criteria of the DSM-IV-TR. It was adapted to the current DSM-5 criteria. The range of the scale scores is 0–9. The psychometric analysis of the scale using the data from this study showed high internal consistency ( Cronbach α = .94).

People who regularly (≥ 1–2 times per month) played different types of games based on the above categories were selected for analysis. Responses pertaining to gambling participation and the incidence of problem gambling were compared between two time points: before the implementation of preventive measures against the pandemic (March 20, 2020) and approximately 1 year later (May–November 2021), once the restrictive measures had been eliminated and it was possible to play again with relative normality.

To avoid bias in the response to the gambling addiction evaluation questionnaires, two different diagnostic questionnaires (BPGS and NODS) were used. The diagnosis of pathological gambling before the pandemic was made with the BPGS scale, while the evaluation of this disorder after the measures taken to minimize SARS-CoV-2 virus transmission was done using NODS.

Table 1 gives the percentages of people in this study who regularly played some game (> 1–2 times per month) before and after COVID-19 preventive measures were in place.

There was a reduction in frequent participation in all types of gambling, with the greatest reductions for landscape games.

A complementary way to understand the changes that occurred is to study whether current regular gamblers were also regular players before the pandemic. Table 2 shows the percentage of regular gamblers after implementation of the COVID-19 preventive measures who already were frequent gamblers, considering the different game types.

The type of gambling with a lower percentage of new gamblers was lotteries (5.33%). No differences were found in the percentage of new gamblers between landscape and online gambling.

Differences according to sex

The percentage of women and men affected by gambling problems (risk gambling and gambling disorder) in this study are indicated in Table 3 .

Women who participated in this study reported fewer gambling problems than men, both in terms of gambling disorder ( χ 2  = 20.65; p  < 0.001; φ  = 0.09) and risk gambling ( χ 2  = 45.77; p  < 0.001; φ  = 0.13).

Changes in gambling disorder incidence

Regarding gambling disorder, Table 4 lists the percentages of participants who exhibited gambling disorder before and after the implementation of pandemic-related preventive measures.

More survey participants exhibited pathological gambling after the pandemic than before the restrictive measures were taken (231 vs. 67). Most people who exhibited gambling disorder before the pandemic also manifested it later (74.6%), whereas only 6.4% of those who did not engage in pathological gambling before the pandemic developed gambling disorder after the measures were implemented. Of the people with gambling disorder after the pandemic, 21.6% had a gambling disorder before, while only 0.6% of those without current gambling disorder showed pathological gambling before the restrictive measures were taken. The difference in these percentages was significant ( χ 2  = 416.21; p  < 0.001; φ  = 0.38).

Gambling disorder with regard to the different types of gambling

Regarding gambling disorder among those who frequently engaged in different types of gambling, we summarize the main results in Table 5 . Our results also consider whether gamblers regularly partake in a single type of gambling (lotteries, landscape, or online gambling) or several types.

Conclusions and discussion

The objective of the research was to analyze the effect on gambling behavior and gambling disorder that the measures to restrict access to public places that were taken to avoid COVID. Some preventive strategies based on the the conceptual framework and the results of this research are suggested.

The results were partially consistent with the hypotheses, because the main reduction in gambling frequency occurred in landscape gambling, which is the type of gambling that suffered the most from restrictive access measures. There was also a reduction in the frequency of lottery gambling, although the measures were temporary. These results are congruent with other research showing a reduction in gambling frequency during lockdown measures [ 27 , 28 , 29 , 30 ]. Unexpectedly, there was also a decrease in the frequency of online gambling, even though it was widely promoted and advertised and there was a very noticeable increase in spending on online gambling during this period of time [ 31 ]. This result may be due to the fact that this research is not an epidemiological study, in which it is intended to evaluate the prevalence of gambling behavior before and after the measures adopted to prevent COVID-19, but the changes produced in the gambling behavior after the implementation of such measures. For that reason, only the results for people who gambled regularly were analyzed.

The percentage of people who participated in different betting games regularly decreased markedly after preventive measures were taken, especially in games that take place in gambling venues or public places with slot machines, as is the case for bars and restaurants in Spain. Therefore, it seems that the measures taken globally to prevent the spread of SARS-CoV-2 could have had an effect in reducing the frequency of gambling, because at one year after implementation of the most restrictive measures, the percentage of people who frequently participate in gambling seemed to be lower. This is a positive outcome in terms of preventing gambling addiction, because frequent gambling is one of the main factors favoring the development of gambling disorder.

This reduction occurred especially in games that take place in venues (gambling halls, bars, etc.) where gamblers have to be physically present and can spend several hours playing at a time. Many of the people who gambled frequently stopped doing so, especially those who previously went to gambling venues or gambled in public places. It is likely that the increase in new frequent players will be at a rate similar to that found in this study, in the range of 1.5–3%. With the restrictive measures taken against the expansion of COVID-19, many of the frequent gamblers who had not yet consolidated the habit of gambling or developed gambling disorder before the pandemic may not return to a frequent pattern of gambling when conditions return to normal, at least for now. If this has helped people to modify their lifestyles, it would have served as a positive preventive measure against gambling addiction.

When it comes to gambling disorder, the majority of those who currently suffer from pathological gambling had already suffered from it before the implementation of COVID-19 measures, whereas only a small percentage of people who did not currently suffer from gambling disorder exhibited symptoms before the pandemic. This may be related to the restrictive measures implemented to prevent the spread of SARS-CoV-2 being useful in also preventing the promotion of new cases of pathological gambling. However, the measures were not sufficient to solve the problem for those who were already suffering from gambling disorder. That is to say: pathological gamblers need specialized treatment. These results are consistent with other investigations that have found no significant reduction in gambling frequency for those who were most engaged in gambling pre-lockdown, especially pathological gamblers [ 32 ].

Not all types of gambling were equally affected by the restrictive measures. Hence, when analyzing the changes associated with pathological gambling after the implementation of preventive measures, differences in the addictive potential of the different types of gambling (landscape, lotteries, and online gambling) must be considered. The addictive potential of the different types of gambling is evidenced when comparing the percentage of pathological gamblers in the groups that regularly gamble in only one type of game. Regular lottery players had five to six times lower rates of gambling disorder and risky gambling behavior compared to those who frequently played landscape gambling or online gambling. Approximately 80% of the people who regularly played these games were found to suffer from pathological or at-risk gambling, which is a very high figure in our opinion. This is due to the structural characteristics of electronic games [ 15 , 16 , 17 ] (landscape gambling) and online gambling [ 33 ].

For this reason, we consider it necessary to take measures restricting access to these specific games (EGMs and online gambling) to prevent the development of pathological gambling in society, i.e., to avoid the spread of the “gambling virus.” However, once a person has been infected and suffers from gambling disorder, it is probably necessary to use other therapeutic measures beyond access control itself.

This study had some limitations. Like most studies that have analyzed the effect of COVID-19 on gambling behavior [ 24 ], it is a cross-sectional study, rather than longitudinal, and may have been affected by recall bias. Another limitation is that we focused on self-report data. Although this study was carried out with general population (that is, it was not a study with clinical population), the sample is not random and it was selected by addiction prevention professionals. Therefore, it is not an epidemiological study and, accordingly, data on the prevalence of gambling disorder in general population cannot be concluded. The fact that there were more participants in the survey with problem gambling after the pandemic than before does not necessarily mean that there was an increase in the incidence of pathological gambling, but rather that people with a current problem with pathological gambling were more interested in responding to the survey. However, most of the analyses conducted on problem gambling have been conducted, not with the general population, but with people who frequently participate in gambling. This allows us to assume that the conclusions deduced here about differences in the risk of addiction for the different types of gambling and the differential impacts of preventive measures are somewhat valid. However, the conclusions must be treated with caution because it is a correlational study and, although the number of respondents is high, it lacks an experimental design.

The main conclusions of this study are the following:

Conceiving of gambling as a virus has important implications for the prevention of gambling disorder. Although it is not possible to implement universal vaccination for consequences of gambling, such that people are immune to it, some measures taken to prevent the spread of SARS-CoV-2 based on lockdown and social distancing may be also useful to prevent gambling disorder. Some examples could be regulating long distances between bookies and schools or among gambling rooms, authorizing EGMs only in gambling rooms and casinos (not in bars or restaurants), etc [ 18 ].

Just as there are less contagious and lethal variants of SARS-CoV-2, there are also gambling games, such as lotteries, that are less addictive and harmful than EGMs and various types of online gambling. In the case of COVID-19, the danger posed depends on the DNA structure, whereas for gambling the structural characteristics of the games are the most important factors. Some preventive measures could include the modification, by law, of some parameters of the games to make the game virus less addictive. For example: restrictions on gambling speed; delaying the time between the bet and the outcome; reduction of maximum bet size; diminishing the percentage of win; posting the payoff probabilities; reducing the frequency of “near-miss” outcomes on EGMs; or prevent, through the use of gambling smart cards, gamblers from losing large amounts of money [ 18 ].

As in the case of virus infections, measures to prevent the spread of disease must also be adapted to social and environmental conditions, placing special emphasis on the most socially and economically vulnerable groups. Therefore, gambling advertising and commercial promotions must be limited. Even in capitalist societies, public health must take precedence over the economic benefits of companies. Paraphrasing the philosopher Michel Sandel, moral limits must be applied to the market [ 34 ]; in the case of gambling virus, such moral limits should enforce to gambling companies.

Availability of data and materials

For security reasons, the datasets generated and/or analysed during the current study are not publicly available due none Gambling and Technological Addictions Research Unit databases can be found on the Internet (Management agreement), but are available from the corresponding author on reasonable request.

American Psychiatric Association. Diagnostic and statistical manual of mental disorder, fifth edition text revision. Washington: American Psychiatric Association Pub; 2022.

Book   Google Scholar  

World Health Organization. Manual of the International Statistical Classification of Diseases, Injuries, and Causes of Death, 11th Revision . Geneve; 2018. https://icd.who.int/browse11/l-m/en#/http%3a%2f%2fid.who.int%2ficd%2fentity%2f499894965 .

Markham F, Young M. “Big gambling”: the rise of the global industry-state gambling complex. Addict Res Theory. 2015;23(1):1–4. https://doi.org/10.3109/16066359.2014.929118.

Article   Google Scholar  

Chóliz M, Marcos M. “That’s No Country for ‘Young’ Men”: a critical perspective on responsible online gambling policies for gambling disorder prevention in Spanish minors. J Gambl Issues. 2022;49:158–73. https://doi.org/10.4309/jgi.2022.49.7 .

Adams PJ. Gambling, freedom and democracy. New York: Routledge; 2008.

Google Scholar  

Korn DA, Shaffer HJ. Gambling and the health of the public: Adopting a public health perspective. J Gambl Stud. 1999;15(4):289–365.

Article   PubMed   Google Scholar  

Marshall D. Gambling as a public health issue: the critical role of the local environment. J Gambl Issues. 2009;23:66–80. https://doi.org/10.4309/jgi.2009.23.4 .

Wardle H, Reith G, Langham E, Rogers RD. Gambling and public health: we need policy action to prevent harm. BMJ. 2019;365. https://doi.org/10.1136/bmj.l1807 .

Chóliz M, Sáiz-Ruiz J. Regulating gambling to prevent addiction: more necessary now than ever. Adicciones. 2016;28(3):174–181. https://doi.org/10.20882/adicciones.820 .

Livingstone C, Rintoul A, Francis L. What is the evidence for harm minimization measures in gambling venues? Evidence Base. 2014;2:1–24.

Cassidy R. Vicious games: capitalism and gambling. London: Pluto Press; 2020.

Hancock L, Smith G. Critiquing the Reno Model I-IV international influence on regulators and governments (2004–2015)—the distorted reality of “responsible gambling. Int J Mental Health Addict. 2017;15(6):1151–76. https://doi.org/10.1007/s11469-017-9746-y .

Binde P, Romild U, Volberg RA. Forms of gambling, gambling involvement and problem gambling: evidence from a Swedish population survey. Int Gambl Stud. 2017;17(3):490–507. https://doi.org/10.1080/14459795.2017.1360928 .

MacLaren VV. Video lottery is the most harmful form of gambling in Canada. J Gambl Stud. 2016;32:459–85. https://doi.org/10.1007/s10899-015-9560-z .

Dowling N, Smith D, Thomass T. Electronic gaming machines: are they the ‘crack-cocaine’of gambling? Addiction. 2005;100(1):33–45. https://doi.org/10.1111/j.1360-0443.2005.00962.x .

Griffiths M. Fruit machine gambling: The importance of structural characteristics. J Gamb Stud. 1993;9(2):101–20. https://doi.org/10.1007/BF01014863 .

Schüll ND. Addiction by design. Princeton: Princeton University Press; 2012.

Chóliz M. Ethical gambling: a necessary new point of view of gambling in public health policies. Front Public Health. 2018;6:12. https://doi.org/10.3389/fpubh.2018.00012 .

Article   PubMed   PubMed Central   Google Scholar  

Williams RJ, Wood RT. The proportion of Ontario gambling revenue derived from problem gamblers. Can Public Pol. 2007;33(3):367–87. https://doi.org/10.3138/cpp.33.3.367 .

Hunt ChJ, Blasczczynski A. Gambling disorder as a clinical phenomenon. In: Heinz A, Romanczuck-Seiferth N, Potenza MN, editors. Gambling disorder. Charm: Springer; 2019. p. 15–28.

Chapter   Google Scholar  

Chóliz M, Marcos M. Psychological treatment of online gambling addiction [Tratamiento psicológico de la adicción al juego online]. Madrid: Alianza Editorial; 2020.

Dowling N, Jackson AC, Thomas SA. Behavioral interventions in the treatment of pathological gambling: a review of activity scheduling and desensitization. Int J Behav Consult Ther. 2008;4(2):172–87. https://doi.org/10.1037/h0100841 .

Echeburúa E, Fernández-Montalvo J. Psychological treatment of slot-machine pathological gambling: new perspectives. J Gambl Stud. 2005;21:21–6. https://doi.org/10.1007/s10899-004-1918-6 .

Quinn A, Grant JE, Chamberlain SR. COVID-19 and resultant restrictions on gambling behavior. Neurosc Biobehav Rev. 2022;143:104932. https://doi.org/10.1016/j.neubiorev.2022.104932 .

Volberg RA, Williams RJ. Developing a brief problem gambling screen using clinically validated samples of at-risk, problem and pathological gamblers Health Sciences 2011. Retrieved from: http://opus.uleth.ca/bitstream/handle/10133/2561/2011-Brief%20Screen-AGRI.pdf?sequence=1 .

Gerstein D, Volberg RA, Toce MT, Harwood H, Johnson RA, Buie T, et al. Gambling impact and behavior study: report to the National Gambling Impact Study Commission. Chicago: National Opinion Research Center; 1999.

Auer M, Malischnig D, Griffiths MD. Gambling before and during the COVID-19 pandemic among European regular sports bettors: an empirical study using behavioral tracking data. Int J Mental Health Addict. 2020:1–8. https://doi.org/10.1007/s11469-020-00327-8 .

Gainsbury SM, Swanton TB, Burgess MT, Blaszczynski A. Impacts of the COVID-19 shutdown on gambling patterns in Australia: Consideration of problem gambling and psychological distress. J Addict Med. 2021;15(6):468–76. https://doi.org/10.1097/ADM.0000000000000793 .

Article   CAS   PubMed   Google Scholar  

Håkansson A. Changes in gambling behavior during the COVID-19 pandemic—a web survey study in Sweden. Int J Environ Res Public Health. 2020;17(11):4013. https://doi.org/10.3390/ijerph17114013 .

Price A. Online gambling in the midst of COVID-19: a nexus of mental health concerns, substance use and financial stress. Int J Mental Health Addict. 2022;20:362–79. https://doi.org/10.1007/s11469-020-00366-1 .

Chóliz M. Crisis, What Crisis? The effect of economic crises on spending on online and offline gambling in Spain: implications for preventing gambling disorder. Int J Environ Res Public Health. 2023;20(4):2909.  https://doi.org/10.3390/ijerph20042909 .

Sharman S, Roberts A, Bowden-Jones H, Strang J. Gambling and COVID-19: initial findings from a UK sample. Int J Mental Health Addict. 2021:1–12. https://doi.org/10.1007/s11469-021-00545-8 .

Chóliz M, Marcos M, Lázaro-Mateo J. The risk of online gambling: a study of gambling disorder prevalence rates in Spain. Int J Mental Health Addict. 2021;19(2):404–17. https://doi.org/10.1007/s11469-019-00067-4 .

Sandel MJ. What Money Can’t Buy: The Moral Limits of Markets. New York: Macmillan; 2012.

Download references

Acknowledgements

• Addictions Service of the Government of the Valencia City Council (PMD).

• Addictions Service of the Government of the Balearic Islands (PADIB).

• Mapfre Foundation.

This research has not been funded.

Author information

Authors and affiliations.

Gambling and Technological Addictions Research Unit, Psychology School, University of Valencia, Avda. Blasco Ibáñez, 21, 46010, Valencia, Spain

Mariano Chóliz

You can also search for this author in PubMed   Google Scholar

Contributions

M.Ch. wrote the main manuscript.

Authors’ information

Dr. Mariano Chóliz is Professor at the University of Valencia (Spain) and director of the Gambling and Technological Addictions Research Unit.

Corresponding author

Correspondence to Mariano Chóliz .

Ethics declarations

Ethics approval and consent to participate.

• This study is in accordance with the ethical standards of the Spanish government and with the 1964 Helsinki Declaration and its later amendments. All data are anonymous and are in accordance with Law 3/2018, on the protection of personal data and guarantee of digital rights.

• The contents of the survey were approved by the ethics commission of the University of Valencia (procedure number: H1482079199937).

• The participants knew the objective of the research and freely agreed to participate. Informed written consent was obtained to participate in the survey.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Chóliz, M. Is gambling like a virus?: A conceptual framework and proposals based on empirical data for the prevention of gambling addiction. BMC Public Health 23 , 1686 (2023). https://doi.org/10.1186/s12889-023-16610-x

Download citation

Received : 23 December 2022

Accepted : 24 August 2023

Published : 01 September 2023

DOI : https://doi.org/10.1186/s12889-023-16610-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Gambling disorder prevention
• Public health
• Gambling policies
• Gambling addiction
• Responsible gambling

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An empirically based model of software prototyping: a mapping study and a multi-case study

• Open Access
• Published: 30 August 2023
• volume  28 , Article number:  115 ( 2023 )

You have full access to this open access article

• Elizabeth Bjarnason   ORCID: orcid.org/0000-0001-9070-0008 1 ,
• Franz Lang 1 &
• Alexander Mjöberg 1  

80 Accesses

Cite this article

Prototyping is an established practice within product and user interface design that is also used as a requirements engineering (RE) practice within agile development. Even so, there is a lack of theory on prototyping.

Our main research objective is to support practitioners in improving on their prototyping practices.

We have designed a model that describes key aspects of the practice of prototyping. The model is based on a systematic mapping study consisting of thirty-three primary studies and on empirical data from twelve case companies. We validate and demonstrate the applicability of our model through a focus group at one company and through semi-structured interviews at eleven (other) startup companies.

Our prototyping aspects model (PAM) consists of five aspects of prototyping, namely purpose, prototype scope, prototype media, prototype use, and exploration strategy. This model has enabled practitioners to discuss their prototyping practices in terms of the concepts provided by our model.

Conclusions

The model can be used to categorise prototyping instances and can thereby support practitioners in reflecting and improving on their prototyping practices.

Working on a manuscript?

1 introduction.

Prototyping is a creative practice commonly applied within product design (Tronvoll et al. 2017 ), usability and user-interface design (Hakim and Spitzer 2000 ; Lauesen 2005 ; Hartson 2019 ), and software development (Acosta et al. 1994 , 1993 ; Goldman and Narayanaswamy 1992 ) to explore the problem and/or the solution domain through use of a prototype, i.e. an early sample, model, or release of a product that simulates one or many dimensions of the (future) product. A prototype can range from a simple paper sketch through a computer-generated mock-up to an (incomplete) version of the production software, e.g. a minimum viable product (MVP), and can be used to learn about user problems and to evaluate solution ideas (Sergio 2015 ). Thus, prototyping enables validating a solution proposal before developing the full product through cost effective testing with real users (Nielsen 1993 ). In our research, we focus on the practice of prototyping, namely the use of prototypes to obtain learning, rather than on the construction of prototypes, meaning that any representation that aids in exploring and learning about a feature or dimension of a new product or business model can be used for prototyping. This broad definition of a prototype, also includes entities such as PowerPoint sketches and videos (Karras et al. 2017 ), e.g. used as marketing material, and interview questions (Batova et al. 2016 ), e.g. used to explore the customer and user domain.

Within human–computer interaction (HCI) and design, the practice of prototyping is well established, and used to explore and test ideas and solutions for user interface designs. However, researchers within HCI pointed out a lack of knowledge about the fundamental nature and characteristics of prototyping, already in 2008 (Lim et al. 2008 ). While there is more recent research on prototyping, we find no comprehensive theory of the practice or its methodology, e.g. overarching principles or procedures for achieving a certain outcome, and only one related literature review of prototyping concerning the definition of MVP (Lenarduzzi et al. 2016 ). For these reasons, we were interested in exploring prototyping methodology with the aim of supporting practitioners in describing and discussing their prototyping practices, and thereby enable them to improve on these through reflection-based learning (Bjarnason et al. 2014 ).

The overall objective of our study was to explore how to categorise prototyping practices from the perspective of prototyping methodology. The initial part of our study was performed in collaboration with our case company Telavox, who were interested in further improving their use of prototyping. We based our research on the current body of knowledge and defined the following four research questions to guide our research:

RQ1: What are the main aspects of prototyping methodology? RQ2: What fields of research have previously investigated the main aspects of prototyping methodology (RQ1)? RQ3: What types of research have investigated the main aspects of prototyping methodology (RQ1)? RQ4: How can the initial version of our prototyping aspects model be improved to better support practitioners?

In this paper, we extend on previously published results for RQ1 (Bjarnason et al. 2021a ) by presenting a revised version of our prototyping aspects model (PAM) including improvements to the initial version of the model (RQ4). These improvements are based on additional empirical data from a multi-case study of eleven startup companies. The initial version of PAM was based on a systematic mapping study (RQ2 and RQ3) and was validated through a focus group at one case company. Herein, we also present results for RQ4 based on semi-structured interviews with twelve practitioners where PAM was used to discuss and categorise forty-three prototyping instances applied within the case companies. This additional empirical material and analysis enables us to improve on PAM and provide a revised answer to RQ1. Initial results on the prototyping practices of startups are published in Bjarnason ( 2021 ) based on four of the twelve interviews. This article is based on the full set of interviews and focuses on validating and improving PAM rather than on describing the overall prototyping practices of the companies, as in Bjarnason ( 2021 ).

Our prototyping aspects model (PAM) may be used to characterise and compare prototyping instances, e.g. regarding the scope and media of a prototype, how it is reviewed with users, and how this affects the cost–benefit balance of using the practice of prototyping for exploring the problem domain and developing software products. We believe that the additional refinements of the model presented in this paper, further improve the applicability of PAM and the model’s usefulness for practitioners.

The rest of this article is organised as follows. We describe related work on prototyping in Section  2 . In Section  3 , we outline our research method, and in Section  4 , we describe the case companies. The model of prototyping (PAM), which is our main contribution, is presented in Section  5 , in response to RQ1, based on previous work on prototyping and on our findings from the multi-case study. Section  6 contains results on RQ2-RQ4 that are discussed in Section  7 , before concluding in Sect.  8 .

2 Related work

While our point of departure and main area of competence is requirements engineering (RE), the research presented in this paper is based on previous work on prototyping within software engineering, RE, as well as, within human factors, usability and user-interaction design. Furthermore, prototyping is commonly used within software startups and in the early stages of software development to explore and validate user needs and requirements. In this section, we provide a brief introduction to related work on prototyping within human–computer interaction and design, agile RE, and for software startups. More detailed references are provided in Section  5 , as part of describing our prototyping aspects model.

2.1 Prototyping within human–computer interaction and design

Within product- and human–computer interaction design, prototyping is commonly used to design and to evaluate the user interface by “ producing or building a model or mockup of a design that can be manipulated and used … to simulate a user experience ” and thereby test this experience without having “to build the real thing” (Hartson 2019 ). While prototyping is mainly used to support design and evaluation within usability and user interaction design, several other advantages of the practice are described including its role in supporting communication and creativity. Prototypes can facilitate communication of design ideas by providing “a concrete basis for discussions” between developers, designers, and users (Budde and Zullighoven 1990 ). As such, prototypes “ serve as a vehicle that enables users and designers to develop a common language ” (Hakim and Spitzer 2000 ). Thus, the use of prototypes can stimulate user involvement, and support marketing and selling new product ideas to customers and to (internal) management. Many of these benefits and uses of prototyping are covered in our model by the aspect of Purpose (see Section  5.1 ) .

Prototyping can be performed throughout the design process starting with simple sketches that evolve into wireframes and into gradually more complete representations of the design. The variation in prototype richness and scope was characterized by Nielsen using the terms breadth and depth (or detailing) of functionality represented by the prototype, where a horizontal prototype provides a broad set of features but with a low degree of refinement (depth), while a vertical prototype can provide detailed (deep) representation of one feature only, i.e. a narrow breadth (Nielsen 1993 ). Other variants of prototype scope include local prototypes that focus on specific issues and are thus both shallow and narrow, and T prototypes that represent a broad set of features but only details (depth) for a few parts (Hartson 2019 ). We base one of the aspects of our model ( Scope ) on this terminology and include the dimension of breadth, while expanding the dimension of depth to distinguish between the refinement of one or more facets, namely functional, visual, interactive and data, see Section  5.2 .

The term fidelity is used to indicate how close to the final product the prototype is with regards to appearance and interaction (Tullis 1990 ). Low- versus high-fidelity prototypes are often used for different purposes within the design community. Low-fidelity prototypes can convey a general look of an interface and are often used to communicate, educate and inform, while high-fidelity prototypes are more expensive to build they can be used to test and evaluate further details of the design, and even serve as a basis for development of the product under design (Rudd et al. 1996 ). In our model, fidelity is represented by a combination of the two aspects Scope (see Section  5.2 ) and Prototype Media (see Section  5.3 ).

Research on the use of prototyping in generating new user interaction designs found that prototyping multiple design in parallel and sharing these led to more creative and better final designs (Dow et al. 2011 ). In our previous research on the use of prototyping in startups, we found that very few startups work with parallel exploration and prototyping, and that those who did work with parallel prototypes tended to have a background in usability and human-interaction design (Bjarnason 2021 ). This dimension of the practice of prototyping is covered in our model by the aspect of Exploration Strategy .

Finally, prototypes can be used as a means to present/demonstrate a product idea, or to evaluate a design through allowing users to interact with the prototype. Such interactions often occur in a meeting or lab settings, e.g. during usability testing, either through free testing or by providing scenarios or other protocols for the user to follow (Tronvoll et al. 2017 ). Evaluating prototypes in “the wild”, i.e. in the environment of the final product, provides a more realistic setting for evaluating a prototype both in terms of the actual physical environment and w.r.t. the people available (Hendry et al. 2005 ). These dimensions are covered in our model by the aspect of Prototype Use, see Section  5.4 .

2.2 Prototyping in agile requirements engineering

Prototyping has been identified as a practice commonly applied within agile software development to manage rapidly evolving requirements by the practices’ ability to support customer communication, and for validating and refining requirements (Ramesh et al. 2010 ). Käpayaho and Kauppinen report from a case study of using prototyping for user interface development at a large retail company applying an agile approach. Their findings indicate that prototyping addresses several challenges within agile but also identify a need to complement prototyping with additional practices. They observed that prototyping helped with challenges related to managing with very little requirements documentation, such as intangible and unaligned views and plans on what to develop. Furthermore, the quality of stakeholder communication at the case company was increased through prototyping and mutual understanding between the development team and the customers was reached faster. Prototyping also increased the motivation for requirements work since updating a prototype was considered more motivating than writing requirements documents (Käpyaho and Kauppinen 2015 ).

The benefits of using prototypes in the customer communication have also been observed by several other researchers including Ramesh et al. and Zink et al. Illustrating and communicating product requirements through a prototype reduces risks related to low requirements specification quality (Ramesh et al. 2010 ). As the prototype representation of the (future) product becomes more concrete with each iteration, the product requirements thereby gradually become more specific (Zink et al. 2017 ). In particular, an executable prototype provides a rich means of communicating requirements details, and reduces the risk of ambiguity, incompleteness and inconsistency in the requirements communication (Acosta et al. 1994 ).

While prototyping provides benefits to agile development, the practice also imposes risks. In particular, research shows that the use of production software in prototyping (rather than throw-away prototypes) incurs risks related to product quality such as scalability, security, and robustness (Ramesh et al. 2010 ). In this case, demonstrating a fully functioning (but early) version of production software may convey an overly positive view of development status to customers and other stakeholders, that in turn may lead to a push to release software prematurely, before sufficient quality has been achieved. Agile projects need to be aware of these risks and apply other practices to mitigate these. Käpayaho and Kauppinen suggest complementary practices such as clearly stating customer responsibilities, management of quality requirements, and consideration of the bigger picture (Käpyaho and Kauppinen 2015 ).

2.3 Prototyping in software startups

Software startups are new companies that develop novel software-based products or services, and that commonly apply prototyping (Bjarnason 2021 ) for exploring and evaluating new ideas and technology in a quick and relatively cheap way (Lauesen 2005 ), and thus, enables cost effective testing with users (Shepherd and Gruber 2020 ). This cost–benefit aspects is especially important to software startups who operate under very uncertain and resource-constraint conditions with the aim of exploring new business opportunities and develop innovative products (Giardino et al. 2014 ; Paternoster et al. 2014 ). While availability of open-source software and pay-as-you-go services provide software-based business opportunities, startups struggle to define solutions for which there is product-market fit and risk wasting time and resources on developing unsuccessful features (Giardino et al. 2015 ). One important success factor is to test the business idea early on to validate its viability in the market (Block and MacMillan 1985 ).

Software startups typically perform light-weight, informal and ad-hoc requirements engineering, in particular during the early stages of the startup venture due to limited resources (Giardino et al. 2014 ; Klotins et al. 2019 ; Nguyen-Duc et al. 2017 ; Terho et al. 2016 ). Requirements are initially elicited and prioritised mainly based on internal sources and on problems experienced by founders. The source of requirements gradually shifts as potential customers are identified, and prototypes are produced that can be used to elicit new ideas and priorities also from customers and other external parties (Nguyen-Duc et al. 2017 ; Terho et al. 2016 ). Tripathi et al. provide similar findings albeit with some more details and based on a broad coverage of literature and cases (Klotins et al. 2019 ).

Within startups, an early version of the final product is often used to validate product ideas with users (Alves et al. 2020 ; Tripathi et al. 2018 ), e.g. by demonstrating mock-ups to customers, as a cost-effective way to obtain market feedback. Thus, prototyping is used as a means to learn about the user and the market (Giardino et al. 2015 ; Paternoster et al. 2014 ), and to increase the chances of business success. However, the use of a live product version as a prototype poses conflicts between the need for quick feedback and a focus on product quality (Ciriello et al. 2017 ). One case study of prototyping in twenty Norwegian startups, by Nguyen-Duc et al., identified factors that affect the speed of prototyping, including the choice of prototype tools and components, varying competences, and the communication and involvement of customers and external stakeholders in the prototyping. They observed that the purpose of prototyping and how the prototype is used regarding customer involvement are factors that need to be considered when selecting prototyping practices (Nguyen-Duc et al. 2017 ).

In an earlier publication, we reported on the prototyping practices of four early-stage startups (Companies A-D, of this article) to understand how they use prototyping to elicit, prioritise, validate, and communicate requirements. In that study (Bjarnason 2021 ), we found that prototyping is commonly used within early-stage startups as a natural part of developing and validating a product, but also that prototyping is implicitly required to ensure funding of the startup venture. Internally, prototyping is used to explore and communicate detailed product requirements , while prototypes are used externally to communicate and validate product-market fit . Thus, for startups, prototyping plays a vital role in market validation and in obtaining paying customers. This validation is required by most investors and prototyping thus plays a critical part in securing the funding needed for startup ventures. Prototyping instances tend to be gradually refined from sketches and interactive mock-ups to fully functioning software versions, often MVPs. The more refined prototype versions (realistic mock-ups and early versions of production software) are more costly to produce and require software engineering expertise. Despite this, our case study found that many startups prefer using these more refined prototypes instead of prototyping using simpler media such as paper sketches or mock-ups. This preference, though more costly, may be due to that a more refined prototype appears inductive to a higher degree of customer trust. Thus, software startups face the challenge of balancing the cost of prototype scope and media against the gains and value that can be obtained from more refined prototypes.

3 Research method

We have addressed our four research questions (RQ1-RQ4) by designing a model of prototyping (PAM) using a combination of a systematic mapping study, a focus group at one case company, and a multi-case study of eleven additional case companies, see Fig.  1 . The mapping study provided a broad base of scientific knowledge that supported our design of the model. The focus group with practitioners provided an initial validation of our model. The model and its practical applicability and usefulness were further validated through semi-structured interviews at the other eleven case companies by using the model with practitioners to discuss and describe their prototyping practices.

Overview of the research method used to design and validate the Prototyping Aspects Model (PAM)

The model was designed iteratively in multiple steps. The initial design was performed by the last two authors resulting in a draft of the model. This draft was then revised by the first author after re-analysis of the literature and after using the model with the initial case companies resulting in the first version of the model (published in Bjarnason et al. ( 2021a )). The first version was then further validated by the first author through semi-structured interviews with practitioners from eleven startup companies and revised based on analysis of the prototyping instances described by the interviewees. The changes to the model were discussed and agreed among all three authors resulting in the second version of our model, which is presented in this paper.

3.1 Systematic mapping study

We performed a systematic mapping study based on guidelines by Petersen et al. (Petersen et al. 2015 ) to explore and draw from the current body of knowledge on prototyping to compile and provide an overarching view of prototyping methodology based on current knowledge. The literature review was guided by our research questions RQ1-RQ3 and consisted of searches , study selection , data extraction , and data analysis . The list of scanned articles and our categorisation of the ones included are provided on-line (Bjarnason et al. 2021b ) to enable other researchers to validate and to facilitate replicating the analysis of our systematic map.

Searches were defined iteratively in-line with RQ1 through test searches and by consulting with two experts in prototyping. The initial searches were combinations of “software prototype”, “software prototyping”, “prototype”, and “prototyping” that yielded large amounts of hits (almost 80.000). In a second iteration the test searches were limited to the terms “prototyping agile”, “requirements engineering prototyping”, “agile requirement engineering prototyping”. To further guide our study and help focus our searches, we consulted two experts in user experience and design, i.e. one senior manager at the initial case company (Telavox) and one senior researcher in user experience at Lund University. These experts provided insights into how to extend the searches beyond software engineering, thereby increasing the quality of our searches.

Test searches were performed in three search engines, of which two were selected for our review. We selected Lund University Libraries search engine (LUBsearch) and ACM digital library and excluded Google Scholar. LUBsearch provides a broad base since it includes other search engines such as Scopus, IEEE Explore, and ScienceDirect. ACM was selected due to providing a set that suited the scope of our mapping study. ACM provides good coverage of software engineering in general, while complementing the content provided by LUBsearch. In addition, our early test searches with ACM resulted in identifying several articles that are highly relevant to our review.

The search strings were specified through an exploratory process with test searches and customised for each of the two selected search engines depending on their specific search facilities. For LUBsearch, we derived keywords from a smaller set of matching papers. The final search consisted of the search string “Prototyping AND (Fidelity OR Software Prototype OR Agile)” for the subject term, with the options “Peer reviewed” and in English. For ACM, we found that many articles lacked keywords and settled on the search string “prototype OR prototyping ” for the title. The final searches were performed in February 2020.

Study selection was performed through a gradually extended scan of title, abstract, and full text using inclusion and exclusion criteria to guide the selection decisions (see below), which resulted in identifying 33 primary studies. The two last authors each performed this selection on the set from ACM and LUBsearch respectively. They aligned their selection by comparing and reaching consensus on inclusion and exclusion for a random set of 10 publications. We included articles on prototyping from all fields and excluded articles that do not explicitly discuss prototyping methodology or prototype dimensions. We defined the inclusion criteria as articles published before February 2020 that cover meta-level or methodological aspects of prototyping or prototype use, since this is the aim of our main research question RQ1. The exclusion criteria were defined as articles that merely describe the use of prototypes without considering methodological perspectives such as principles and procedures for the prototyping, articles that are not peer reviewed or written in English, and duplicates of studies already included in the systematic map. We provide a demographic overview of our map in Section  6.1 including the number of articles per search engine and for each selection step.

Data extraction, data analysis and classification of general and specific items of the primary studies was performed by the last two authors, and then validated by the first author. The general items extracted were publication year and research field (RQ2), and the primary studies were classified according to these and according to research type (RQ3, Wieringa et al. 2006 ). The resulting classification of the primary studies is reported through describing the demographics of the systematic map in Section 6.1. Furthermore, specific items related to our main research question (RQ1) were extracted. These specific items consisted of the aspects of prototyping covered in the primary studies. We extracted the information specific to our enquiry on prototyping through reading the full text of each primary study and wrote a short summary of how it relates to our study. Initial categories, or aspects of prototyping, emerged and were identified based on these summaries, similar to open coding of grounded theory. When these aspects had been established, each primary study was classified according to these aspects.

3.2 Design of prototyping aspects model including initial validation

Our model was designed iteratively based on the open coding of our systematic mapping study. First, the commonly occurring aspects of purpose (why) and prototype scope (what) were included, followed by exploration strategy and prototype use (how). Each aspect was detailed into further facets through analysis of the primary studies. An additional aspect that was considered, but at this point excluded, was the method used to produce a prototype, e.g., paper prototyping or computer simulation. Since our focus was prototyping methodology, and since computer-simulated prototypes can achieve similar effects as paper prototypes, we decided to exclude this aspect from the first version of our model. However, a similar fifth aspect of prototype media was added in the second version of our model based on the insights gained from our multi-case study. For the first version of the model, an additional design iteration was performed to increase internal validity of our model.

We performed an initial validation of the draft version of the prototyping aspects model by re-analysing the articles in our systematic map (see below). This internal validation mainly led to renaming aspects, adding, and restructuring some facets. The aspect of strategy was renamed exploration strategy and the aspect of review method was renamed prototype use to more clearly reflect what these aspects represent. The facets related to validation & testing were grouped to provide a better overview of the purposes of prototyping. Two additional facets were added to prototype scope , namely interactive & haptic behaviour , and realistic data , both of which were observed in the original analysis, but initially not included. Finally, the facet of usage environment was added to prototype use since the feedback that can be obtained from demonstrating or testing a prototype may vary for, e.g., a loud or a dark environment compared to in a meeting setting.

The design of our model was further refined after subsequent validation with practitioners at Telavox and through a multi-case study of ten software startup companies. These validation steps are described below. The first and second version of the model are described in Section  5 , and the differences between the two versions are described and motivated in Section  6.3 based on our subsequent multi-case study.

3.3 Validation of draft version: Focus Group with Telavox and Reanalysis

The initial draft version of our prototyping aspects model was validated through reanalysis of the primary studies of our systematic map and through a focus group at a case company. Footnote 1 The reanalysis was performed to improve reliability and internal validation of our prototyping aspects model, while the focus group was performed to validate the relevance and usefulness of our model with practitioners. The first author performed triangulation of the model through an independent re-analysis of the primary studies of our systematic map including (re)coding the full text of the articles in NVivo. The differences were then discussed, agreed with the other two authors, and the model updated as described in this article.

The focus group was prepared by designing a focus group protocol structured according to the main activities or stages of a requirements process, namely Concept exploration, Eliciting customer needs, Identifying system scope & requirements, Validate and improve system scope & requirements, and Confirm system scope & requirements. These stages were inspired by Lauesen and correspond to the main activities of requirements engineering common to all projects, both traditional and agile (Lauesen 2002 ). For each stage, we considered the main RE goals and suitable prototyping instances, or scenarios based on two of the aspects of our model, namely purpose and prototype scope. These prototyping scenarios were discussed at the focus group, supported by a set of questions. The protocol was iteratively designed by the last two authors, reviewed with the first author and with a contact person at the company, and then improved upon. The focus group protocol is available in Appendix A .

The focus group was conducted with five practitioners from Telavox’ user experience team. This team elicits and details product requirements through prototyping of the user interface. The participants had degrees in either industrial economics or interaction design; two B.Sc. and three M.Sc. The focus group was managed, led, and moderated by the last two authors. To ensure equal airtime, the participants were given turns to initiate the discussion for each talking point. The focus group was performed on-line due to the ongoing Covid-19 pandemic. The meeting was recorded, transcribed, and analysed using open coding to identify information related to the different stages of requirements, and to the different aspects of the model. The empirical data from the focus group it is treated as confidential since it may contain company information and the participants are promised anonymity to encourage them to speak freely. Furthermore, the description of the case company and the case-specific results have been reviewed by the case company.

3.4 Broader validation: multi-case study

We performed a multi-case study of software startups using semi-structured interviews to gain insights into current prototyping practices. Software startups were selected as our object of study due to these companies frequent use of prototyping to explore and shape their business models and product offerings, as part of the Lean startup method (Ries 2011 ) where new products are designed through continuously building, measuring, and learning what the customers want and are willing to pay for (Olsen 2015 ). Through the case study, we explored the practical applicability of our model by using it to discuss and categorise prototyping instances, thereby validating the model further and identifying improvements to it (RQ4). The case study consisted of four stages, namely preparations , data collection, data analysis, and validation.

In the preparation stage , a case study protocol and an interview guide were designed (available in Appendix B ), and an initial set of case companies and interviewees were recruited. The interview guide is based on the first version of our prototyping aspects model (Bjarnason et al. 2021a ) and on previous research on software startups (progression model (Klotins et al. 2019 ), typical characteristics (Berg et al. 2018 ; Giardino et al. 2015 )). The interviews were designed to investigate how startup companies work with requirements and prototyping, and support a broad exploration of (possibly) influencing factors by covering questions about the interviewee and the company. After the first two interviews, it became clear that the terms prototyping and prototype are not uniformly understood, which caused some initial confusions at these interviews. The interview guide was then extended with a question about what prototyping means to the interviewee. Case companies and interviewees were selected through convenience sampling and consisted of startups recruited through local business incubators; Minc, VentureLab, SmiLe, and Ideon Innovation, of which some were previously involved in our RE course. The main criteria for selecting startups to include in our study, was that they use or plan to use prototyping for developing their business and/or software. In total, eleven startups (Company A-K) were investigated through twelve interviews with thirteen interviewees. The companies and interviewees are described in Section 4.2 . In general, one interview was held per company, except for Company D where the founder and the technical lead were interviewed separately. In addition, the interview at Company H was held with two founders of different profiles (at their request).

The data collection consisted of semi-structured interviews with ample opportunity for interviewees to speak freely and to ask follow-up questions. The interviews were held and recorded in a video conferencing system (Zoom), and each lasted for about one hour. At the interviews, we presented our prototyping aspects model (version 1) and opened up for a discussion of how their prototyping practices relate to the different aspects (purpose, scope, use, and exploration strategy of prototyping). At the beginning of each interview, the participants were informed that their participation is voluntary and that they and their companies can remain anonymise, if they so wish, and that the interview data is treated as confidential since it may contain company information. Furthermore, each participant was sent a draft of the resulting articles for the parts derived from their interview for validation purposes.

In the data analysis stage , the audio recordings were transcribed and analysed by applying thematic coding in several iterations using NVivo. In the first analysis iteration , codes representing the different parts of the interview were used, such as interviewee roles and background, company and product characteristics, startup life-cycle maturity and challenges, RE and prototyping practices. In the second analysis iteration , each interview transcript was re-read and prototyping instances mentioned by the interviewees were identified. The relevant parts of the transcripts for each prototyping instance were coded using codes denoting the case company and a sequence number for each prototyping instance. For example, for Company B for which three prototyping instances were identified, the codes B1-B3 were used to denote and tag the relevant parts of the interview transcripts for each of these instances. In the third analysis iteration , the interview data per prototyping instance was analysed and each prototyping instance was categorized according to the prototyping aspects model (see table in Section 6.3 ). Observations about how well the model corresponded to the described prototyping instances were noted in memos, together with descriptions of each prototyping instance, and illustrative quotes found in the transcripts. These memos were then used to report the results and to adjust the prototyping aspects model as described in Section  6.3 .

Finally, in the validation stage of our case study, the interviewees were asked to validate the descriptions related to their companies and the co-authors discussed and agreed to the revised version of the prototyping aspects model. All interviewees were contacted and asked to validate the information relevant to their companies, and to indicate if they wished their company name to be included in the article or not. The interviewees were provided with the prototyping instances identified for their startup, the memos related to these including the quotes extracted from the transcripts, and the company and interviewee characteristics reported in Section 4.2 . The interviewees responded, mainly by agreeing to the descriptions and by highlighting some minor misunderstandings and additional information, after which the manuscript was revised accordingly.

4 Case companies

The initial draft of the prototyping aspects model was validated through a focus group at Telavox. The first published version of the model was then further validated through the multi-case study of eleven software startup companies where twelve practitioners were interviewed about their use of prototyping.

4.1 Initial case company: Telavox

Telavox offers cloud-based Private Branch Exchange solutions. The company was founded in 2002 and currently has around 250 employees. The initial part of this study, i.e. the mapping study and initial design of the model, was carried out in 2020 at the company’s site in Malmö, Sweden, by the 2nd and 3rd author. At this site there are about 15 development teams for areas such as app development for Android and iOS, user experience, and web development. Product support is coordinated and provided via key account managers that work closely with customers. The company applies Scrum, and thus uses an agile development model. New and improved product ideas are explored and communicated through prototyping and general product statements. Product owners coordinate product development and prioritise product requirements. When an idea is ready for development, the appropriate teams are assigned user stories. The teams apply test-driven development and continuous delivery. Product scope is validated with key account managers and customers before release.

4.2 Case companies A-K: software startups

Eleven software startups (Company A-K) were investigated in our multi-case study, see Table 1 . Most of these companies were in the inception or stabilisation phase (Klotins et al. 2019 ) of their business ventures, while Companies E and J were in the growth stage (Klotins et al. 2019 ). In total, we interviewed thirteen practitioners at the case companies, mostly founders and co-founders, but also some product owners and technical leads. An overview description of our interviewees is provided in Table 2 .

5 Results: Prototyping aspects model (RQ1)

Our model characterizes the methodology of prototyping by five aspects: purpose of prototyping , prototype scope , prototype media, use of prototype, and exploration strategy , and is here described based on related work. Table 3 provides an overview of the model and the primary studies from our initial mapping study for each aspect. In this article, a revised version of the model is presented based on a multi-case study where the model was used to characterise prototyping instances. The changes to the first version of the model are described and motivated in Section  6.3 . Table 3 contains both the first version of the model (published in Bjarnason et al. ( 2021a )) and the revised (second) version (presented herein) to facilitate comparison.

5.1 Purpose of prototyping – why prototype?

The practice of prototyping can achieve many different purposes that often vary throughout the life cycle of a project. We have identified eight purposes, namely exploration , communication , incremental development , quality improvement , and validation & testing of problem–solution fit, product-market fit, technical feasibility, and usability . When prototyping, several purposes may be satisfied simultaneously, e.g., communicating the product idea to potential customers while also validating its market desirability. A project’s purpose of prototyping often evolves from exploration and communication to validation & testing (Ratcliff 1988 ).

5.1.1 Exploration & learning

Prototyping is commonly used to explore the solution space (Dow et al. 2011 ; Lim et al. 2008 ; Rahman et al. 2017 ; Wiberg and Stolterman 2014 ) and learn by experimenting with ideas (Budde and Zullighoven 1990 ; Chen et al. 1994 ; Tronvoll et al. 2017 ), and is a foundational purpose for any prototyping. Problems and new solution directions can be discovered and explored through prototyping and can lead to new ideas and direct further exploration (Lim et al. 2008 ). Such exploration of multiple solutions can mitigate the risk of overinvesting in a single concept (Dow et al. 2011 ). However, Schneider found that when users are left out of the process and prototype use is purely internal, only the developers learn (Schneider 1996 ). Lichter et al. suggest combining the purpose of exploration with communication and testing of product-market fit to clarify requirements (Lichter et al. 1994 ).

5.1.2 Communication: sales, alignment of requirements

Visions and ideas about a product can be communicated through prototyping, which provides a common language between developers and stakeholders (Budde and Zullighoven 1990 ; Ciriello et al. 2017 ; Rahman et al. 2017 ; Zink et al. 2017 ) and acts as an anchor for group communication (Dow et al. 2011 ). Prototyping thus facilitates presenting, discussing, and evaluating a product with external parties, such as customers and investors, and internally within a project, thereby supporting decision making (Budde and Zullighoven 1990 ; Raja 2009 ). Ciriello et al. note that prototypes can support requirements elicitation by clarifying problems early on (Ciriello et al. 2017 ). However, a prototype may convey an overly positive impression of the current status that can lead to unrealistic customer expectations and subsequent requests to evolve the prototype into the final system (Lichter et al. 1994 ).

5.1.3 Incremental development

One purpose of prototyping may be to evolve the prototype into a final product based on user feedback and priorities (Budde and Zullighoven 1990 ; Lichter et al. 1994 ; Schneider 1996 ; Toffolon and Dakhli 2008 ). In these cases, the prototype may be a pilot system or a partial product version such as alpha or beta, or an MVP, that is developed with the expressed intention of exploring or validating a solution option (according to our definition of prototype). In agile development, prototyping is often an integral part of the development process with regular feedback from users and other stakeholders (Fairley and Willshire 2005 ), e.g. through validation of software at end-of-sprint demonstrations. Thus, prototyping is often used within agile development to detail and validate requirements, and to reduce uncertainty (Bellomo et al. 2013 ). Fern et al. propose a prototyping methodology that covers throw-away prototypes as well as prototypes that will be developed further into the deliverable system (Fern and Donaldson 1989 ).

5.1.4 Quality improvement

Prototyping can be used to optimise quality aspects, e.g. by focusing on response times while all other behaviour is retained. Kordon observed that care needs to be taken to avoid measurement overhead and to ensure accuracy in the evaluation (Kordon 1994 ). Arano et al. propose the use of hybrid prototyping for improving quality aspects since this can enable measuring quality in early development stages prior to full implementation (Arano et al. 1993 ).

5.1.5 Validation & testing

One of the main purposes of prototyping is to validate requirements by testing a solution option with internal and/or external stakeholders (Ciriello et al. 2017 ; Tronvoll et al. 2017 ) for perspectives, such as problem–solution fit, product-market fit, technical feasibility, and usability. Problem–solution fit concerns the degree to which the envisioned product solves an actual customer or end-user problem. Prototyping can be used to investigate customer needs, validate and clarify customer requirements and tasks (Budde and Zullighoven 1990 ; Ciriello et al. 2017 ), and thus support a company in designing a solution to address customer problems. Product-market fit is a premise for business viability. Prototyping can be used to explores a product’s business potential from the customer perspective, and to assess the value of the product and likeliness of purchase (McCurdy et al. 2006 ; Zink et al. 2017 ) and the product’s ability to fit within time and budget constraints (Zink et al. 2017 ). The insights gained from such prototyping can be used to support business-related decision making (Ciriello et al. 2017 ). Technical feasibility concerns a system’s technical capabilities and the feasibility of realising requirements in the solution space (Budde and Zullighoven 1990 ; Zink et al. 2017 ), e.g. to operate at scale, to resolve structural uncertainties, or to fulfill security requirements. Proof-of-concept prototypes are built for this purpose (Tronvoll et al. 2017 ) and can be used to evaluate novel technical approaches (Fern and Donaldson 1989 ). Similarly, breadboard prototypes are used to investigate technical aspects, e.g. in circuit design, and to support system specification and coding (Budde and Zullighoven 1990 ; Ciriello et al. 2017 ; Lichter et al. 1994 ; Lim et al. 2008 ). Furthermore, Lichter et al. found that a prototype built to demonstrate feasibility can also support project acquisition (Lichter et al. 1994 ). Finally, Zink et al. found that feasibility testing requires a high share of the time invested in a project (Zink et al. 2017 ). Usability testing was reported by Zink et al. as the least common singular purpose for prototyping though it is often combined with other purposes (Zink et al. 2017 ). User interface design can be validated through prototyping (Derboven et al. 2010 ; Hakim and Spitzer 2000 ; Zink et al. 2017 ) and uncover various usability issues (Lim et al. 2008 ). When prototyping for usability testing, Hakim et al. recommends ensuring that metrics such as task completion time, can be captured and considering the protocol for use (e.g. associated instructions and data) since this may affect the results of the testing (Hakim and Spitzer 2000 ).

5.2 Scope of prototype – what to prototype?

The scope of a prototype describes the extent to which a prototype resembles the final product and is, in our model, represented by the breadth of the prototype’s functionality (Bruegger et al. 2009 ; Budde and Zullighoven 1990 ; Goldman and Narayanaswamy 1992 ; Hakim and Spitzer 2000 ; Lim et al. 2008 ; McCurdy et al. 2006 ; Tronvoll et al. 2017 ), and the facets of functional refinement (Bruegger et al. 2009 ; Budde and Zullighoven 1990 ; Goldman and Narayanaswamy 1992 ; Hakim and Spitzer 2000 ; Lim et al. 2008 ; McCurdy et al. 2006 ; Tronvoll et al. 2017 ), visual appearance (Budde and Zullighoven 1990 ; Goldman and Narayanaswamy 1992 ; Hakim and Spitzer 2000 ; Hendry et al. 2005 ; Jaskiewicz and Helm 2018 ; Lim et al. 2008 ; Liu and Khooshabeh 2003 ; McCurdy et al. 2006 ; Zainuddin and Liu 2012 ), interactive & haptic behaviour (Budde and Zullighoven 1990 ; Goldman and Narayanaswamy 1992 ; Hakim and Spitzer 2000 ; Hendry et al. 2005 ; Lim et al. 2008 ; Liu and Khooshabeh 2003 ; McCurdy et al. 2006 ) and data realism (Lim et al. 2008 ; McCurdy et al. 2006 ). Breadth represents the extent to which a prototype covers a product’s full functionality, e.g., all or only one product features (broad or narrow prototype), while the other facet represents the degree of detailing of this (deep or shallow) regarding functionality, visual appearance, interactive behaviour, and data realism. Functional refinement represents the degree of detailing for each function or feature. The visual appearance of a prototype concerns aesthetics, e.g., layouts, fonts, and user interface elements, and is interrelated to functional refinement since visuals are represented by functionality. The user’s experience is also affected by how well a prototype mimics a product’s interactive & (sometimes) haptic behaviour, and to what degree the available data can simulate normal and realistic use.

The scope affects the cost of producing a prototype and the feedback that can be obtained (Derboven et al. 2010 ; Lim et al. 2008 ; Liu and Khooshabeh 2003 ; McCurdy et al. 2006 ; Tronvoll et al. 2017 ). Thus, prototype scope relates to the purpose of prototyping. A rough paper prototype of one product feature is an example of a prototype with a small functional breadth and a low degree of refinement, i.e. a narrow and shallow prototype. In contrast, a minimal viable product may have a broad functional scope by representing all menu options, but varying degrees of refinement (and depth) depending on the implementation status for each feature and for the overall product, e.g., the user interface, and for the amount of user data provided in the prototype.

Yasar describes three perspective to consider when prototyping: role (of product in a user’s life), look & feel, and implementation (Yasar 2007 ). These perspectives can help pinpoint the purpose of prototyping and the scope of a prototype. In our model, look & feel is captured by the facets of visual and interactive & haptic behaviour. The facet of realistic data relates to look & feel, but primarily to the perspective of role since the degree to which provided data simulates actual use affects the user’s ability to relate to their situation.

There are different views on how the degree of prototype refinement, e.g., for visuals, affects the type and amount of feedback that can be obtained. Sefelin et al. reports on a study on usability testing with prototypes where the number of suggestions and critiques do not significantly differ between prototypes with a high vs. low degree of refinement, i.e. between deep or shallow prototypes. However, the users preferred testing with a prototype with a high degree of interactive refinement since this provided freedom to explore the system by themselves (Sefelin et al. 2003 ). Several researchers report that some usability issues can not be discovered unless the prototype provides a broad functional scope with a high degree of refinement, in particular for visual appearance and interactive behaviour (Liu and Khooshabeh 2003 ; McCurdy et al. 2006 ). However, a mix of low and high degrees of refinement is suggested as more economical, e.g., for usability testing. Yasar even reports that simpler prototypes with narrow functionality scope and a low degree of refinement are cost effective and useful for validation purposes and to capture major issues (Yasar 207 Similarly, Bellomo et al. describe how a company consciously selects the breadth and degree of refinement of prototype scope to match the prototyping purpose, and to validate the concept in focus (Bellomo et al. 2013 ).

The terms hi and lo-fidelity are often used to describe resemblance to the final product. Our terminology is more fine-grained and provides a more precise way of categorising prototypes by their breadth and degree of refinement w.r.t. different facets. We believe that our model can support informed decisions of which prototype scope that is required to meet the desired purposes of prototyping.

5.3 Prototype media – what media to use for prototype?

A prototype can be constructed using a range of techniques and media, including sketching on paper (Hendry et al. 2005 ) or in PowerPoint, using a prototyping tool to produce a wireframe or a mock-up (Liu and Khooshabeh 2003 ), or using an early version of product software as your prototype (McCurdy et al. 2006 ; Sefelin et al. 2003 ). Even a video or a physical model is included in our definition of a prototype, and can be used to, e.g., validate product-market fit, or to communicate with stakeholders.

While there is a correspondence between different kinds of prototype media and the scope of these, the cost of producing prototypes, the gains and types of learning that can be obtain from these varies with the media (McCurdy et al. 2006 ; Zainuddin and Liu 2012 ). The choice of media may also affect the ease with which the prototype can be evaluated in some environments (Hendry et al. 2005 ). Sefelin et al. compared the use of paper sketches to that of computer-based prototypes of identical breadth and level of refinement, i.e. different media was used to represent the same prototype scope. These prototypes were used for usability testing, and the results showed no major differences in the obtained feedback depending on prototype media. However, the computer-based prototypes tended to yield more comments on graphical details, while the paper sketches stimulated participant to “ a greater willingness to draw their suggestions ” (Sefelin et al. 2003 ).

Liu and Khooshabeh made a similar comparison between paper sketches and computer-based (interactive) mock-ups (Zainuddin and Liu 2012 ). They found that while paper sketches provide more flexibility for designers in early stages, this kind of prototype media requires more effort to use with larger sets of users, e.g., to manually simulate interaction. Furthermore, Liu and Khooshabeh also found that interactive computer-based mock-ups yielded more feedback when used for user testing.

While paper sketches are often cheaper and faster to construct, Sefelin et al. suggest selecting prototype media based on a number of factors, including the competence of those constructing the prototype, the abilities of the available prototyping tools, and the envisioned continuation of the prototyping. Finally, Hendry et al. found that paper sketches were a good media for eliciting and validating requirements in actual environments and with some stakeholder groups, such as in public locations where people of all age categories can be accessed (Hendry et al. 2005 ).

5.4 Prototype use – how to use the prototype?

This aspect concerns how a prototype is used to achieve a purpose and covers who the reviewers are (internal, external, or with family-friends-and-foes FFF), if direct prototype interact is used (or not), what review approach that is used (scenario based or free), and in what environment the prototype is presented and reviewed. With these four facets of prototype use the main uses of prototype identified in our literature review can be represented, namely internal prototype use without any user presentation (Budde and Zullighoven 1990 ; Dow et al. 2011 ; Schneider 1996 ), prototype demonstrations (Bellomo et al. 2013 ; Heisler et al. 1989 ), scenario testing (Tronvoll et al. 2017 ), and free testing (Heisler et al. 1989 ; Tronvoll et al. 2017 ). Lichter et al. emphasise the importance of user involvement in prototyping (Lichter et al. 1994 ) and several researchers address challenges with this (Ciriello et al. 2017 ; Lichter et al. 1994 ; Zainuddin and Liu 2012 ). Zainuddin and Liu propose a systematic approach to building prototypes with user involvement and capturing user feedback during prototype use (Zainuddin and Liu 2012 ).

The choice of reviewers affects the learnings that can be obtained from prototyping. Purely internal use of a prototype can support brainstorming and allow designers and engineers to generate and organize ideas. While internal use allows a project to focus on ideas and possibilities rather than on external expectations, there is a risk that knowledge remains with, e.g., the developers. Schneider suggests better capitalization of pure internal prototyping by capturing and documenting this knowledge (Schneider 1996 ). Using a prototype with external parties often increases the cost due to increased expectations and demands on quality. In addition, the pre-knowledge of customers and users affect the feedback that can be obtained, and Cafer and Misra suggest adapting the review method based on the customers’ cognitive abilities (Cafer and Misra 2009 ). Several of the startups included in our multi-case study described showing prototypes to family-friends-and-foes (FFF) before going to potential customers, as a means to obtain friendly feedback.

There is usually no direct reviewer interaction with the prototype during demonstrations. Instead, the presenter shows the prototype by operating it live or by using prepared media such as videos or photos. For more refined prototypes, e.g. computer-simulated mock-ups with a high degree of interactivity reviewers may be encouraged to interact and directly use the prototype, and thereby provide richer insights and user feedback. Such interaction can be achieved also with simpler prototypes, additional human effort is required to simulate, e.g. button presses, when using prototypes with a low degree of refinement for interactive & haptic behaviour (Liu and Khooshabeh 2003 ).

Different approaches can be used when evaluating a prototype, e.g. a scenario-based approach which can support reviewers in relating the prototype to their own problems and usage scenarios. When using prototyping for testing purposes, scenarios can be used to guide users through instructions and steps. Similarly, Ciriello et al. suggests using storytelling to increase customer involvement (Ciriello et al. 2017 ). An alternative approach when testing prototypes with external parties or FFFs, is to encourage free testing as for example for beta testing when minimal instructions are provided.

The usage environment for a prototype can affect the outcome (Grevet and Gilbert 2015 ; Lichter et al. 1994 ; Tronvoll et al. 2017 ) both in the possible feedback and the representation of subjects in the review. If a product’s natural habitat is very different from a lab or meeting room environment, this can affect the feedback, e.g., for a system intended for a loud environment. Using a prototype in conditions similar to that of the final product increases the chance of uncovering new setting-specific requirements (Lichter et al. 1994 ). Hendry et al. performed usability testing among home-less people and concluded that using a prototype in the streets enabled them to reach otherwise inaccessible user segments (Hendry et al. 2005 ). Thus, the environment of use may result in some future users being underrepresented in the prototyping, further limiting the feedback.

5.5 Exploration strategy – how to traverse the solution space?

This aspect concerns strategies used to traverse the solution space over time. The exploration strategy determines which instances to pursue, how resources and decisions are organised, and how uncertainties are managed. The needs and goals may change for each iteration, which can enable focusing on specific product aspects or parts. Within one iteration, several concurrent solution paths can be pursued in parallel, which can stimulate innovation (Dow et al. 2011 ) and avoid imposing unnecessary limitations (Budde and Zullighoven 1990 ), but also cause contradictions between prototypes and complicate decision making (Jaskiewicz and Helm 2018 ). When iterating over multiple parallel prototyping instances, the length of the iterations may impact the extent and quality of the prototyped solution option and thus the prototyping effectiveness (Tronvoll et al. 2017 ). Jaskiewicz et al. found that while longer iterations promote focusing on certain aspects, multiple short iterations can lead to a more diverse but superficial set of solution options (Jaskiewicz and Helm 2018 ).

We model this aspect using the three facets: single vs parallel exploration , iteration focus , and iteration size These facets are partly based on the four strategies identified by Tronvoll et al., namely point-based , parallel (or set-based solution arrays), optimisation (or performance set investigations), and flexible exploration (Tronvoll et al. 2017 ). Our facet of single exploration corresponds to Tronvoll’s point-based strategy where resources are focused on one single solution path and where alternatives are considered before being cemented. If the solution appears unsuitable it may be necessary to discard the prototype and redo the process. In contrast, several potential solution options are pursued simultaneously when using parallel exploration . Decisions are managed by merging prototype variants either continuously or during stage-wise iterations.

Tronvoll’s other two strategies, namely optimisation and flexible exploration can be viewed as either single or parallel exploration , though the focus and size of the iterations varies, which is captured by these two facets of our model. Tronvoll’s optimisation exploration uses parallel exploration with a focus on feature or product level, and with gradually decreasing iteration sizes . For such exploration, the number of simultaneously investigated options are kept to a minimum to reduce cross-contamination and over time the solution converges by applying a systematic performance evaluation of the optimised characteristics. Decisions surrounding the solution options are postponed until they can be validated and when encountering several alternatives, only the most promising one is pursued. Solution options are judged by performance, which affects the evaluation. Such prototypes should be evaluated against a range (weak-strong-good quality) and may involve additional factors that affect each other. For example, a certain design-level requirement may provide an efficient way of saving text, but drastically increase the size of the saved file.

Finally, Tronvoll’s flexible exploration strategy corresponds to a single exploration strategy where the solution options are based on best-guesses and on facilitating the change of quality requirements as the work progresses. This approach is well suitable for agile development where the solution option is iterated, evaluated, and changed as required.

6 Underpinning results: map demographics, focus group and multi-case study

We will now present the results used to design the prototyping model aspects presented in the previous section. In this section, we report on the demographics of our systematic map, the results from the focus group and from the interviews of our multi-case study. The model was initially designed based on previous literature, then validated and adjusted through the empirical data of the focus group and the multi-case study.

6.1 Demographics of Map (RQ2 and RQ3)

Our systematic map consists of thirty-three primary studies published between 1988 and 2018, see Table 4 . The majority of these (17) were within software engineering (in general), 11 within human factors (HF) & design, and 5 within requirements engineering (RE), see Table 5 . The number of articles appear to have increased somewhat over the years from on average 1 to 2 articles a year, except around the year 2000. The increase is mainly within the areas of HF & Design and RE. The research type for each paper was classified according to the categories by Wieringa et al. (Wieringa et al. 2006 ): (1) Evaluation research investigates a problem or technique in practice and provides new knowledge of causal or logical relationships, (2) Solution proposals present a solution without a full-blown validation, (3) Validation research presents a solution proposal validated outside of industrial practice. (4) Philosophical papers sketch new theories or frameworks, (5) Experience papers describe personal experiences and may contain anecdotal evidence.

Around 40% of the articles (14 of 33) are based on in-vivo research and empirical data from industry ( Evaluation) , see Table 6 . Another 40% (12 of 33) propose solutions based on in vitro (only) validation or theoretical reasoning ( Solution proposal and Validation type). Our systematic map indicates an increase of in-vivo research concerning prototyping over the past 10–15 years.

6.2 Focus group results: Validation of Initial Version of Model Footnote 2

Practitioners discussed the use of prototyping to perform requirements-related tasks such as elicitation, specification, and validation at the focus group by reflecting on scenarios based on our prototyping aspects model. In general, the participants agreed to the scenarios, that they correspond to a typical way of working and to their prototyping practices, which they see as a good way of working with agile software development. There was some disagreement about prototyping in early project stages. Several participants said that prototyping is time consuming and difficult, and rarely used during early phases and at the first meetings with customers. Instead, these participants preferred stakeholder interviews and competitor analysis to understand the problem domain. In contrast, two participants said that early prototyping, e.g., through sketching, can be useful in understanding the problem. Another participant said that early prototypes can be beneficial in loosely defined projects and facilitate discussions about user expectations. They had used simple prototypes in the form of sketches, which helped designers and users understand what they were discussing and assisted in framing the role of the system.

6.2.1 Purpose

The participants described several purposes for prototyping. Its value in exploration & learning was described as “ the more you work with design [of the prototype] the more you know what works for you ”. Footnote 3 Several participants described the value of prototyping for internal use and the importance of “ building understanding before you start thinking about solutions .” One participant described creating personal prototypes to get an overview of the future system, essentially working as a specification . Another participant said that prototypes are more useful than a formal specification to communicate requirements. Two participants described the importance of exposing developers to the prototype as a way of testing the technical feasibility and, if possible, adjust the requirements to “ the easiest way … to implement the solution ”. Another participant described the importance of communicating and involving developers by testing prototypes and discussing requirements, although some developers just want information on “ the components I need to build .”

6.2.2 Prototype scope

The focus group provided validation for this aspect of our model. They recognised the relevance of considering breadth and depth of functionality, and refinement of all our identified facets (of the first version of our model) except for data richness, which was not mentioned during the focus group. The participants described that in early project stages, they prefer to use simpler prototypes, either paper sketches with shallow functionality and low refinement or a prototyping tool that supports producing prototypes with shallow functionality and mid to high degree of visual refinement. These simpler prototypes provide “ a way to understand the problem ” and to “ organize and explore … [the solution] space. ” Another participant implied that feedback on visual style could be avoided by using prototypes with a low degree of visual and interactive refinement such as wireframes that encourage feedback on functionality. In contrast, for more complex solutions “ you get value out of being able to click around ” and a high degree of interactive refinement is preferred. Some participants said that prototyping must involve interactivity and that they “ wouldn’t do a good job … [without] interactive prototypes ”. This illustrates the need for a common definition of prototyping. The fact that prototype scope was spontaneously discussed by our participants demonstrates that our model can provide support for this.

6.2.3 Prototype use

The participants confirmed all four review methods included in the first version of our model, i.e., internal use, demonstrations, scenario-based, and free testing. Several participants described using prototyping to test new ideas with colleagues and that they first show the prototype “ within the team and not with users ”. This was especially the case for early prototypes that one participant rarely showed to customers, despite experiencing challenges in discussing with users without any visuals. When prototypes are shown to external stakeholders it is often as a demonstration that takes place after first having discussed their problem and current situation. One participant said that users are only occasionally involved in interacting with prototypes while this is done “ all the time ” within the project. Instead, users are involved in beta testing “ all the time ”, which some, but not all, of the participants see as prototyping. One participant described the use of scenario testing a few times a year, e.g., with new employees or students.

Several participants mentioned the importance of structuring feedback sessions to “ get feedback on the right thing ”, i.e. relevant facets. One participant said that it is easier to get feedback on what is good rather than what needs improving. Another participant described that “ talking with just one person isn’t enough ” since stakeholders may have preconceived solution ideas. One participant suggested using solution-agnostic questions to encourage users to consider what problems the system should solve and to discuss requirements at domain and product level.

6.2.4 Exploration strategy

Our participants validated the four exploration strategies included in the first version model and described that the strategy is varied throughout development stages and for different prototyping purposes. One participant described the importance of creating several parallel prototyping variants to avoid getting stuck in a single solution too soon. She said that using multiple prototypes helps users’ express the direction the system should take, and thereby involves them in determining product scope and requirements. Similarly, another participant described that they “ throw a wide net with many different options ” in initial project stages. In addition, several participants said that building several variants of a prototype is useful for exploring alternative solutions and demonstrating these to stakeholders to “ collect ideas about how to proceed and what path to choose ” and thereby optimise the solution. Furthermore, one participant described that in a later stage, an incremental and flexible exploration strategy is needed to “ fit in [customer requests] with the concept you have in mind ”.

6.3 Multi-case study results: broader validation and adjustments (RQ4) Footnote 4

We validated the prototyping aspects model through our multi-case study with eleven startups. The model was presented to practitioners in the interviews and used in the analysis of the interview data to categorize the prototyping instances described by our interviewees. In total, we identified forty-three prototyping instances among our startups, see Table 7 . Prototyping in these startups ranges from using simple sketches through mock-ups, to using source code to explore and to obtain feedback on early software versions. We have categorised the identified prototyping instances using our prototyping aspects model. In this section, we describe and motivate changes to our model based on our empirical data. The first and the revised (second) version of the model are shown side-by-side in a table in Section  5 . We recommend that the reader consults that table while reading this section. Initial results of the prototyping practices of startup Companies A-D are available in a separate publication (Bjarnason 2021 ).

6.3.1 Purpose

This aspect mostly worked well in discussing and categorising how the startups use prototyping, and only two minor revisions are made to this aspect, namely for the purposes Communication and Validation .

One of the main purposes of prototyping is “ as a tool for selling products to customers ” (Company B) and is thus used in communicating “ about sales… to make them understand that there is a need ” (Company A). For this reason, Company A had invested “quite a lot of work” in building a high-fidelity interactive mockup (prototyping instance A2) which “ is fake, but quite good… communicates that we know and builds trust… Based on that we get investors. ” (Company A). For these reasons, we adjust the detailing of the aspect of Purpose in our model to highlight that communication includes sales and marketing, thus Communication & Alignment is changed to Communication: Sales, Alignment .

For Validation & Testing , the boundaries between Market viability and Business viability were hard to determine for the prototyping instances, and the difference between these two is unclear. Instead, we replace these with the terms Problem–Solution fit and Product-Market fit (Osterwalder et al. 2014 ). This will more clearly separate between prototyping to ensure that the product matches the needs of users and customers by provides a solution that addressing their problem, and prototyping to validate that customers are willing to pay for your product and thus the existence of a viable market and business opportunities. Different prototyping instances may be used for each dimension. For example, the mock-up C2 was used to “explore what is required [by the market]”, i.e. product-market fit, while the beta release C3 “to 2–3 [customers] is used to tune [the implementation] further” (Company C) and thus improve the problem–solution fit. Product-Market Fit is especially important for startups that “ must produce a prototype to be able to confirm that you have buyers, and thus revenue… [before] incurring more costs without knowing that it is sellable ” (Company A).

Furthermore, while we observe that prototyping is not directly used within our startups for the purpose of improving quality, we retain the purpose of Quality Improvement in our model. The lack of support for this prototyping option in our case study is likely due to the nature of early startups, rather than lack of relevance in general of this purpose. In startups, the main focus is on establishing a viable business model with a product solution that matches customer needs and problems, and developing initial product version to realise their solution ideas. Thus, prototyping is primarily used for sales & marketing (Communication), internal exploration of the solution domain, feasibility testing of technical aspects of the solution design, and to validate the product-solution and product-market fit of their business model.

6.3.2 Prototype media

When discussing and analysing the prototyping in the startups it became clear that the media used to represent a prototype, e.g. paper or PowerPoint sketches, computer-generated mock-ups, or source code software, was an important aspect to consider. A similar aspect, related to paper prototyping, was discussed by the authors during our original design of the model, but discarded at that point in time since the aspect of prototype scope appeared to be sufficient. However, we reintroduce this aspect in our model since the multi-case study indicates that the choice of prototype media affects the costs and benefits of prototyping. Our interviewees described that they “ have chosen to produce a mock-up due to cost and time aspects ” (Company A). We believe that this is an important aspect of prototyping that can further support practitioners in making informed decisions about the kind of prototype media to use. In particular for early stages of product development, we want to highlight the cheaper and easier kinds of prototype media such as videos and interviewing, which some of our startups find very useful. For example, in prototyping instances H1, J6, I1 and I2, videos are used for sales & marketing (communication) and for validating product-market fit. One interviewee described videos as a way “ to present products far before they are ready ” (Company I). This interviewee also described prototyping instance I3 and interviews as a means of prototyping “ even without anything to show ” by “ talking to people about how they solve this problem today ” (Company I). Similarly, the now growing startup Company J used videos in social media channels for sales & marketing purposes; “ funny videos that became a bit viral ” (Company J). This startup also uses interviewing for exploring the problem domain. In the early prototyping instance J1, they “ asked people [users] … [and] that became an important signal ” that their solution idea was viable, and later used a similar approach in prototyping instance J5 to validate their revenue model.

6.3.3 Prototype scope

When discussing the scope of a prototype, the degree of refinement of the visual appearance, interactive behaviour, and functionality worked well. For example, one interviewee described that “ we do not focus that much on interactivity at the beginning ” (Company F). In addition, five interviewees expressed the importance of realistic data, e.g. to capture behaviour around “ errors and bad data ” (Company I). Also, “ it is really important to use that [realistic data] otherwise your [customers] probably get stuck on that” (Company F) and that when “ the data is fairly realistic … [it] can be shown to customer without any problems ” (Company G). Another startup that markets advanced AI algorithms described that they “ need accurate data to back up [our algorithm]”, and thus demonstrate their solution by using “ actual [customer] data ” (Company H).

The aggregated dimension of broad vs narrow functional scope was harder to convey, though we saw examples of both broad and narrow prototypes covering many vs a few features. For this reason, we revise this aspect to include Breadth as a stand-alone dimension, alongside the dimensions that can be refined, i.e. functionality , visual appearance , interactive & haptic behaviour , and data realism . We believe this provides a clearer and more easily comprehensible model for describing the scope of a prototype.

6.3.4 Prototype use

This is the aspect where the previous structure was less aligned with how the interviewees described their use of prototypes. The dimension of usage environment worked well, even though many of our startups only tested their prototypes in meeting settings. In part, this is due to challenges in accessing the actual usage environment. For example, for Company F where testing in a live environment is “ difficult since our product targets clinics… varies a lot how open they are with this .” Other reasons for this, are likely cost and awareness of when testing in a live environment is important. One interviewee believed that “[environment] is an important aspect of mobile solutions, to test them in their correct context ” (Company I).

We find that the (previous) dimension of review method is not fine-grained enough to categories the identified prototyping instances. Instead, we replace this dimension in our model with the three dimensions reviewers (who receives or gives feedback on the prototype), prototype interaction (directly with prototype or not, e.g. when demoing), and review approach (scenario-based or free). This corresponds better to how the interviewees describe their use of prototypes.

All interviewees described with whom prototypes were used, i.e. Reviewers, and often the same prototype is used with multiple kinds of reviewers. For example, the sketches of prototyping instance A1 were “ tried out on ourselves and on people in our proximity, then we started developing ” (Company A). The later more refined “ mockup [A2] then becomes a blueprint for the product to be built by developers ” (Company A). Thus, reviewers are often internal within the startups and development team, as well as, external such as customers, funders, or product users. In addition, some startups described using family and friends (so called FFF , Family, Friends, and Foes) for obtaining feedback on prototypes. For example, for Company J, the prototyping instance J3 used “ FFF when we have made something new. What do they think? ” and for J4 they “ grabbed people in the corridors and ask them what they think… [and then] realised that we had missed several things ”(Company J).

There were also variations in how these reviewers could interact with the prototype. When demoing, e.g. of sketches or early mock-ups, there is no reviewer interaction with the prototype. There are also examples of choosing not to allow reviewer interaction with the prototype since “[customer] awareness is too low ” (Company A) and “ the market is still not that aware of their needs…to have the level of understanding to be active [in giving constructive feedback on premature mock-ups]” (Company E). For more refined prototypes w.r.t. functionality, visual appearance, and interactivity, the users may be encouraged to try the prototypes out for themselves. For example, the fully functioning product prototype of B3 was available to the general public and “ customers get to use our stuff… [allowing the startup to] follow up how they are used in the field ” (Company B). Similarly, users are encouraged to interact directly with the mock-up of prototyping instance F2 “ to see reactions…. how they reason, how they select and react to things ” (Company F), which provides the startup with rich feedback. We have added a dimension of Interactivity to our model to cover this dimension.

Finally, while scenario testing was included in our initial version of the model, our interviewees primarily described the use of scenarios as an approach that could be applied both when demoing a prototype, or when allowing users to interact directly with the prototype. For example, when demonstrating the paper mock-ups of prototyping instance F1, the startup would “ try to get them [customers] to think about what they do then [in that scenario] (Company F). Similarly, a startup that provides a technical solution uses scenarios to help non-techie customers “ see the value [in the solution] …[through] presenting these cases to them… and avoid just clicking us through ” (Company G). A similar approach is seen for prototyping instances K4 and K5, where the startup “ goes through a simple scenario, explains the challenges, and shows the solution ” (Company K). Thus, we added Review approach as a dimension with the options scenario-based or free, where free, covers both free testing, e.g. for beta releases, and demos without any clear scenarios.

6.3.5 Exploration strategy

This final aspect was the hardest to discuss with our interviewees, and the one with the least number of responses with the exception of parallel exploration . Using multiple prototype versions in parallel was a clear and understandable strategy, though most of our startups stated that due to resource constraints they tend to focus on one version at a time. As one interviewee said: “ we go with one [option] first since we are a small team ” (Company H). Another interviewee mentioned that “ we always try to sketch different [options] ” (Company F) though without giving any concrete example of this (and therefor not seen in any of the reported prototyping instances.) A third interviewee said: “ focus one thing at a time and learn about. But, in some cases [parallel exploration is useful] … e.g. for a pricing model ” (Company I). Thus, we have modified the aspect of exploration strategy to include three facets: single or parallel exploration , iteration focus ( business , product , feature or optimisation level), and iteration size . The two facets related to prototyping iterations were indicated by a few of our interviewees. For example, one interviewed technical lead advocated the “ need to iterate slowly… [to ensure] what the stakeholder originally wished for ” (Company D). Initially, startups often iterate at the level of the product, as was described by the interviewee from Company E: “ we have an idea and want to see if it works. ” And, then move towards “ more gradual development … and more structured feature growth from the very basic experimental ideas up to the ready-for-market products ” (Company E).

The dimension of iteration size replaces the other three exploration options (of the initial version of our model), namely point-based , optimisation , and flexible exploration . The difference between these previous strategies mainly connects to the size of the change between prototype versions. Prototyping with an optimisation strategy, and thus with small changes between versions, could be conveyed through giving the Purpose of Quality improvement. Similarly, the difference between point-based and flexible could also be deferred to the size of changes and connected to the stage and maturity of a product or an idea. For example, in the early stages of a startup prototyping could be used for broad exploration of a suitable solution approach, in which case flexible exploration appears suitable. As a startup and their solution approach matures, they are more likely to want to explore more fine-grained variations, e.g. in what features or what user-interface design to implement, in which case point-based exploration, or even optimisation exploration is more relevant.

7 Discussion

We have investigated the current body of knowledge regarding prototyping methodology through a systematic mapping study and a multi-case study of eleven startups. Our research identifies five main aspects of prototyping (RQ1) that cover the Why? and How? and What? of prototyping. These aspects can be used to characterize prototyping instances and thereby provide practitioners and researchers with a model (see Section 5 ) that can help them to reflect on, analyse, and improve their prototyping practices. This was the case at our focus group and in the twelve interviews of our multi-case study where the four aspects included in the initial version of our model were covered, namely purpose , prototype scope , prototype use , and exploration strategy . When categorising the prototyping instances described in the interviews of the multi-case study, a fifth aspect emerged, namely the kind of prototype media used, e.g. sketch, mock-up, or source code software.

The systematic map on which our model is based consists of thirty-three primary studies and represents a wide set of papers from the past thirty years from the areas of human factors and design, requirements engineering, and agile software development (RQ2). We observe an increase in publication rate during the past decade and of empirically based research (RQ3).

7.1 The aspects of prototyping (RQ1 and RQ4)

7.1.1 purpose of prototyping.

The purpose of prototyping can vary and may consist of a combination of reasons. At the heart of prototyping, lies exploration of the solution space through experimenting with ideas, gathering feedback, and iteratively detailing, validating, and communicating product requirements. Within agile, prototyping is a known RE practice where requirements are gradually defined, validated, and communicated through prototyping as part of the incremental development process (Ramesh et al. 2010 ). With a prototype, new requirements can be elicited through exploration & learning and validated by testing business viability, market desirability, technical feasibility, or usability.

Prototyping also provides a powerful tool for communicating with customers and for creating a good communication climate within a development team (Dow et al. 2011 ) by showing rather than just telling. In addition, for startups prototypes play a vital role in sales & marketing and in obtaining funding for their venture (Bjarnason 2021 ). However, there is also a risk that prototypes can convey an inaccurate perception of development status and create unrealistic expectations (Lichter et al. 1994 ). This risk should be considered when prototyping to validate product-market fit to avoid making unrealistic business decisions regarding budget and development plans (Ciriello et al. 2017 ; Zink et al. 2017 ).

Prototyping can also support exploring the problem at hand. An additional purpose of prototyping is to specify requirements and to act as a requirements specification, as was mentioned by several of our case companies. This is a topic that requires further research to understand in what contexts a prototype can be used as a specification and what is required of a prototype to fulfil this purpose.

7.1.2 Prototype media

The kind of prototype media used affects the cost of constructing it, but also the learnings and benefits that can obtained from using it. Simpler and cheaper media such as paper or interviewing, or computer-based mock-ups can be used with good benefits, especially in the early stages of ideation and product design and development, such as in software startups (Nguyen-Duc et al. 2017 ). However, our previous research indicates that some startups tend to prefer and strive to use source code software for prototyping since they believe this enables them to demonstrate ability and build trust with customers and investors (Bjarnason 2021 ). This preference for certain types of prototype media is often due to the skills and previous experiences of prototyping technologies and tools of those involved in the startup, which play an important role when selecting the media used for prototyping (Gupta et al. 2021 ; Nguyen-Duc et al. 2017 ). This connection between skill set and prototype media also found in our interview material. For example, while UX designers can quickly produce mock-ups using tools, e.g. Figma, software developers often prefer sketching and exploring their ideas directly in source code software.

While we advocate considering current experience and competence, we also encourage practitioners to consider other factors when selecting the kind of media to use for prototyping, such as what kind of feedback that is sought and the risks involved in using source code software for prototyping. While choosing source code as the prototype media enables quickly getting started with actual development, this also comes with risks related to product quality and to cementing solution ideas too early on. While there may be stakeholders, e.g. sponsors, that are eager to quickly move on to realising and delivering a novel idea, moving too quickly to production source code comes with a risk of having to cancel the project later on (Ciriello et al. 2017 ). For these reasons, throw-away prototyping is often advocated since companies are then forced to separate between identifying the ‘right’ requirements (through prototyping) and implementing the requirements ‘right’.

7.1.3 Scope of a prototype

The prototype scope also affects the cost of prototyping and the type of feedback that can be obtained. Thus, the scope should be selected to match the intended purposes of the prototyping effort. The breadth of a prototype and its degree of refinement for functionality, visual appearance, interactive behaviour, and data realism, can be varied. For example, a high-quality mock-up that covers all the features of a future system with toy examples, has a broad prototype scope covering all major features but with a low degree of functional refinement of these, but with a high degree of refinement for their visual appearance and interactive behaviour, while it provides a low degree of refinement for realistic data (only toy examples). The two dimensions of breadth and refinement (in general) have been described in previous literature using the terms horizontal versus vertical prototyping (Budde and Zullighoven 1990 ), where horizontal prototyping explores the breadth and a vertical prototyping the refinement of the scope of the software being development. Compared to previous literature, our model also distinguishes between the type of facet that is refined, e.g. visuals or functionality, and provides a more fine-grained terminology for describing the scope of a prototype.

While our focus group participants actively related to three of the facets of refinement, namely functionality, visual appearance and interactive behaviour, they did not mention data richness. However, five of our interviewees in the startups could relate to the importance of using realistic data, mainly to convey realistic scenarios and trust to customers, but also for validating error cases connected to missing or badly formatted data. We interpret the omission of mentioning this facet at the focus group and among the remaining seven interviewees as an indication of low awareness of the importance of the facet of data for prototyping. Since it is also the facet with the least number of supporting references in our mapping study, we interpret this as an indication that further knowledge and research on the role of data richness in prototyping is needed. For example, to what degree does the use of realistic data, e.g., in a demonstration, affect what feedback that can be obtained?

Furthermore, we note that studies report contradicting results on the relationship between prototype scope and the feedback that can be obtained, and thus the purpose that can be fulfilled. For example, for usability testing, one study shows no significant difference in feedback for prototyping with a low versus a high degree of refinement (Sefelin et al. 2003 ), while other studies suggest that the most cost-effective scope is either a mix of low and high refinement (Liu and Khooshabeh 2003 ; McCurdy et al. 2006 ), or to use simple prototypes with a low degree of refinement (Yasar 2007 ). These conflicting findings indicate that the matter is complex, and that further empirical research is needed to identify the relevant factors of prototyping practice and the environment, and the relations between these.

7.1.4 Use of a prototype

The aspect of prototype use covers with whom the prototype is presented and reviewed (internal, external, or with family-friends-and-foes), how (with or without user interaction, and based on scenarios or more freely), and in which environment this takes place, e.g., in a lab setting or in the actual environment in which the actual product is to be used. At our initial case company, prototypes are frequently used internally to try out new ideas. Our focus group participants described using prototypes with customers either through a demonstration without any direct user interaction, or as free testing of a beta version of the software. In general, our startup companies describe using prototypes primarily internally in the early stages of their business venture, and gradually extending the use to family-friends-and-foes and then to external stakeholders such as potential customers and investors.

Direct interaction with prototypes is common for internal use and when the prototype is slightly more refined. Several startups also describe using scenarios both when demonstrating a prototype and when asking users to interact with it as a means to enhance understandability and communicating how their product solution can address user problems. Who and how a prototype is used affects the type of knowledge that can be obtained through the interactions that takes place both with the actual prototype and between the people involved in, e.g. a prototype demonstration.

The obtained feedback can be steered to specific aspects by structuring the prototype demo sessions. For example, the focus group participants described using solution-agnostic questions to focus a prototype demonstration on the problem description, rather than on details in the solution. The interactions around prototypes relate to the cognitive abilities (Cafer and Misra 2009 ) and communication skills (Ciriello et al. 2017 ) of those involved and is a complex and interesting area for further research. For example, to investigate how to optimise the feedback obtained from users through communication techniques such as storytelling, and if a smaller, less refined, and thus cheaper, prototype scope can be compensated for by boosting the prototype use through designing extensive and highly realistic usage scenarios.

The environment in which a prototype is used is another facet of use that affects the feedback that can be obtained. Previous research describes the impact of the physical environment (Lichter et al. 1994 ). We suggest that the digital environment also may play an important role here and is a facet to consider for use of a prototype. In our previous research on digital work environments, we have identified systems interplay and work interplay as two important factors to consider in RE for systems and tools intended to be used in the work place, i.e. the interaction with other systems and with current work practices (Håkansson and Bjarnason 2020 ). Further research is needed to investigate in which situations a prototype should be used in the targeted digital and physical environment, and in which situations a lab or meeting setting is sufficient to gain the knowledge needed at a specific stage in the development process and for a specific product domain, type of feature, and prototyping purpose.

7.1.5 The exploration strategy

The strategy used to explore the solution space affects which prototyping instances that are pursued and how resources are utilised. Previous research identifies four main strategies, namely point-based, parallel, optimisation and flexible exploration (Tronvoll et al. 2017 ). While all these strategies are relevant to prototyping, we found that when discussing exploration strategy with practitioners a different set of facets is more suitable. This set of facets can also be used to describe all four strategies included in the first version of the model. For that reason, we modify our model to cover single or parallel exploration where multiple solution paths are explored at once, the size and the focus of the iteration , e.g. business, product, feature, or optimisation level.

Exploring multiple options in parallel enables optimising, e.g. a detail in the design, or different parameters in a revenue model. While this approach requires more resources, it also allows for delaying decisions on alternative requirement options until more knowledge has been obtained for these. Exploring one solution path or quality aspect at a time through prototyping allows for freely selecting solution options based on current knowledge and as requirements changes. Single option exploration is the most common approach among our startups since it is more cost-effective. The interviewees that did mention using a parallel exploration approach tended to have a background in user-interaction design. This is similar to our focus group participants who described only using single exploration in later development stages, while preferring a strategy of parallel exploration in early stages to keep an open mind to alternative solutions and avoid getting locked into one solution option at an early stage. This corresponds to previous research that found that parallel exploration stimulates innovation (Dow et al. 2011 ).

We note that previous research identifies a connection between the length of an iteration and the number and quality of solution options, or requirement possibilities, that are obtained (Jaskiewicz and Helm 2018 ). This highlights an interesting avenue for further research in considering the perspective of time in relation to prototyping scope and considering the costs and benefits of taking multiple short prototyping steps covering only a few requirements for each step, compared to performing fewer iterations on a larger set of requirements for each step.

Furthermore, we note a parallel between prototyping in small optimising iterations with how to manage quality (non-functional) requirements (Berntsson Svensson and Regnell 2015 ). In both cases, the outcome focuses on evaluating against a range of values, as opposed to a simple pass/fail, and the increased complexity of needing to considering influencing factors (Tronvoll et al. 2017 ). This complexity in managing quality requirements has also been observed in the context of another agile RE practice, namely the one of using test cases as requirements (Bjarnason et al. 2016 ). It would be interesting to investigate if this is due to the same underlying characteristics of quality requirements, or if there are other factors in common between prototyping and testing, and, if so, how these two practices relate and can be aligned.

7.2 Threats to validity

We discuss the validity of our study and the presented model in view of descriptive and theoretical validity, and generalizability.

Interpretative and descriptive validity concern how reasonable the conclusions are given the data and the extent to which these are objectively described. We judge that both these aspects are high for our study. Several steps were taken to mitigate the risks of misunderstanding and misinterpreting the literature on which our model is based, the focus group participants, and the interviewees. The primary studies were analysed twice, first as part of the initial design by the 2 nd and 3 rd author who calibrated their views, and then independently by the 1 st author, as part of the initial validation (after the focus group). We interpret the fact that this re-analysis only led to minor modifications and improvements of the presented model (see Section 3.3 ) as an indication of high descriptive validity although further research is required to further strengthen the evidence for our model. The risk of misinterpreting the focus group participants was mitigated using the same kind of independent re-analysis of the transcript. In addition, the results from the literature study and the focus group were presented at the case company, who also reviewed an earlier version of this paper describing the first version of our model without raising any concerns about misinterpretation. Furthermore, the risk of the (single) first researcher mis-interpreting the interview material was partly mitigated by asking the interviewees to read through the summarising memos with selected quotes from the material and the set of identified prototyping instances. The feedback from the interviewees mainly concerned additions, e.g. mentioning more prototyping instances, which were then incorporated in the paper.

Theoretical validity is determined by our ability to capture what was intended. While we believe our model provides a good representation of the existing research on prototyping methodology, there is a risk that we have missed relevant previous research and that there are additional primary studies relevant to the topic of our systematic mapping study. This remains an open threat that could be addressed in future research through triangulating the search results, e.g. through snowballing. Furthermore, there is also a risk that our model does not fully align with current prototyping practices in industry, even though we have taken initial steps to explore and validate this. Thus, there is a risk that our model is not complete and that there are additional aspects of prototyping, and in particular additional facets, that should be included in our model. This is especially relevant to the aspect that were modified in this second version of the model, primarily i.e. Prototype media , Prototype use , and Exploration strategy . While these modified aspects were used in our analysis of the interview material, they were not presented to the interviewees. Thus, further research, in particular of the practical applicability of our model, is needed to strengthen the completeness of our model, e.g. through case studies and other empirical investigations of industrial prototyping practices. Furthermore, our research indicates a number of potential relationships within the entities of the model, e.g. between prototype scope and prototyping purpose, that appear to affect the learnings that can be obtained. Exploring these relationships, e.g. through case studies, quasi-controlled experiments, and extended literature reviews, poses an interesting area for further research that can provide insights into how to optimise prototyping practices from a cost–benefit perspective.

Generalizability of our model beyond our case companies and the area of RE is believed to be medium. Since our model is based on previous research within HF, RE and agile software development, we believe that our model may be valid and relevant beyond the case companies involved in our study. However, further research is required to validate this. In particular, prototyping practices for large and established software product companies need to be investigated since their prototyping practice may differ greatly from those of the smaller and newer companies investigated so far in our research. Furthermore, additional research is needed to understand the lack of empirical data in our case studies on some of the aspects and facets of our model, in particular exploration strategy and quality improvement. The lack of evidence for these in our case studies may be due to contextual difference in how the practice of prototyping is used, but may also be an indication of areas for which increased insight may support practitioners in improving their prototyping practices. For these reasons, we suggest further case studies where our prototyping model is applied to additional companies and organisational contexts to further improve generalizability.

8 Conclusions and future work

While there is a host of research on prototyping within user interaction design, software engineering and agile development, we find only some research on the use of prototyping as a requirement engineering (RE) practice. Also, most of the research on the methodological aspects of the practice of prototyping concerns the scope of a prototype (e.g. horizontal vs. vertical, hi- vs lo-fidelity, paper prototyping, sketching, mockups etc.) rather than considering the overall practice of prototyping (that also includes how and with whom a prototype is used and for what purposes). To address this lack, we have designed a model of prototyping. Our prototyping aspects model (PAM) is based on a systematic mapping study of previous research and has been iteratively validated and improved upon through a focus group at one case company and through interviews at eleven startup companies (RQ4). We have identified five main aspects of prototyping (RQ1) that are included in our model, namely the purpose of prototyping, the scope of a prototype, the prototype media, the method of prototype use , and the exploration strategy used to explore the solution space. In this paper, we provide a description of each aspect and their more detailed facets based on previous research and on empirical data from our case studies. We conclude that research on prototyping methodology has mainly been performed within software engineering (in general) and within human factors & design (RQ2), and that there is roughly the same amount of in-vivo and in-vitro research on this topic (RQ3), although there appears to be an increase of in-vivo research during the past decade.

We believe that our model can support agile development teams in reflecting on their prototyping practices and in making conscious choices regarding how to explore the solution space in an effective way considering their goals and resources. Practitioners are encouraged to consider the following:

Purpose of prototyping : What is to be achieved with prototyping, in general and for a prototyping instance, e.g. learning or validation, communication or optimisation? Select the prototype scope and the method of prototype use to match the intended purpose based on existing knowledge from research and your own experience.

Prototype scope : To what extent does the prototype need to represent the final product to achieve the intended the purpose? Is a broad representation of all future product features needed, or is a narrower scope sufficient? What level of refinement is needed w.r.t. functionality, visual appearance, interactive behaviour, and data? Depending on the aim, focus on detailing a specific feature (narrow and refined prototype scope) or providing an overall system view (broad prototype scope). Balance the cost of a broader and more refined prototype scope against the possible benefits.

Prototype media : Given the purpose of the prototyping and the selected scope , consider what media that will yield the best learnings and benefits with the least amount of effort. If the purpose is to explore an early idea within the development team, consider using simpler media such as sketches either in paper or in digital form. If the purpose is to test the technical feasibility of a new component, this will likely be best achieved by prototyping in actual source code software . We encourage practitioners to consider simpler forms of media, such as paper sketches, interviewing, and videos that enable exploring ideas very early on in the design and development process. Also, consider the available resources and competences within your team and select the prototype media accordingly. If someone is skilled in producing computer-based mock-ups , this is a fast and cheap way to explore and validate product ideas involving user interaction. In contrast, it is quicker for an experienced software developer to validate, e.g. a new algorithm in a technically complex product through prototyping in source code software. Furthermore, consider the overall cost of development from a long-term perspective. In particular, if considering prototyping in source code software, also consider if and how this prototype is to be thrown out or carried on into subsequent development stages with the associated risks to product quality and cementing ideas too early on.

Prototype use : Which stakeholders and user categories can provide the feedback needed to fulfil the purpose? Should they interact directly with the prototype or is a pure demo sufficient? Should a scenarios-based approach be used to increase understandability? Design the review to align with the purpose; to focus on problem or on solution understanding, and on the relevant facets of scope. Consider the usage environment, both physical and digital, and adapt to yield the desired type of feedback, both regarding content and stakeholder representation.

Exploration strategy : How broad is our current potential solution space? Is the main focus currently on product or feature level? What sized changes are suitable to manage between iterations? If in the initial stages of development, consider using a parallel exploration strategy to avoid fixating on a single solution option too early on. Switch to a single exploration strategy as more certainty is gained.

The presented model poses a starting point for further research into prototyping in specific organisational contexts, e.g., for startups, and to explore the relationships between prototyping aspect. The impact of different factors can be studied, and different prototyping practices compared, by categorising prototyping instances using the five aspects of our model and by comparing to other contextual factors. Areas for future research on prototyping practices include the use of prototyping as a specification practice, the effect of realistic data, the interplay between how a prototype is used/reviewed and the obtained feedback, the communication around prototypes, and the influence of cognitive abilities. Through evidence-based guidelines and insight into prototyping, practitioners may be supported in selecting their prototyping practices from a cost–benefit perspective, and thus improve their abilities to effectively elicit, specify, validate, and communicate novel business ideas and requirements. We believe that effective prototyping can help software development organisations to optimise their use of resources to pinpoint and develop successful products.

Finally, since prototyping is used in several areas, such as user interaction design, requirements engineering, software design and development, the practice has the potential to bridge and integrate different development activities throughout the development life-cycle. As such, it is an interesting practice for further research, and in particular, to investigate how prototyping can facilitate a better integration and alignment between different software development activities such as requirements engineering, user interface and software design, implementation, and testing.

Data availability

The dataset for the systematic map of literature and the protocols for the focus group and the case study including interview guide are available on-line (Bjarnason 2021b ). Data collected during the focus group and the interviews are not publicly available due to reasons of confidentiality.

The focus group was performed prior to the reanalysis and thus on initial draft of PAM, as depicted in Fig.  1 .

The aspect of prototype media was not included in the model at this stage of our study and thus not covered in the focus group.

Direct quotes from focus group participants are noted within citations and italicized.

Direct quotes from interviewees are noted within citations and italicized.

Questions primarily for interviewees with the business perspective

Acosta, R.D., Burns, C.L., Rzepka, W.E. and Sidoran, J.L. (1994).A case study of applying rapid prototyping techniques in the Requirements Engineering Environment. Proceedings of IEEE International Conference on Requirements Engineering 66–73

Alves, C., Cunha, J. and Araujo, J. (2020). On the Pragmatics of Requirements Engineering Practices in a Startup Ecosystem. 311–321

Arano, T., Chang, C.K., Mongkolwat, P., Liu, Y. and Shu, X. (1993). An object-oriented prototyping approach to system development. Proc. of 17th Int. Computer Softw. and Applications Conf. 56–62

Batova, T., Clark, D. and Card, D. (2016). Challenges of lean customer discovery as invention. 2016 IEEE International Professional Communication Conference (IPCC) 1–5

Bellomo, S., Nord, R.L. and Ozkaya, I. (2013). Elaboration on an integrated architecture and requirement practice: Prototyping with quality attribute focus. 2nd Int Workshop on TwinPeaks 8–13

Berg V, Birkeland J, Nguyen-Duc A, Pappas IO, Jaccheri L (2018) Software startup engineering: A systematic mapping study. J Syst Softw 144(2018):255–274

Article   Google Scholar  

BerntssonSvensson R, Regnell B (2015) A Case Study Evaluation of the Guideline-Supported QUPER Model for Elicitation of Quality Requirements. REFSQ 15:230–246

Google Scholar  

Bjarnason E, Hess A, BerntssonSvensson R, Regnell B, Doerr J (2014) Reflecting on Evidence-Based Timelines. IEEE Software 31(4):37–43. https://doi.org/10.1109/MS.2014.26

Bjarnason E, Unterkalmsteiner M, Borg M, Engström E (2016) A multi-case study of agile requirements engineering and the use of test cases as requirements. Inf Softw Technol 77(2016):61–79

Bjarnason E, Lang F, Mjöberg A (2021a) A Model of Software Prototyping based on a Systematic Map. ESEM 2021:1–11

Bjarnason, E., Lang, F., Mjöberg, A. (2021b) Data set for systematic mapping study on prototyping aspects and protocol for focus group and case study. https://serg.cs.lth.se/experiment-packages/pam/ . Accessed: 2023-08-14

Bjarnason, E. (2021). Prototyping Practices in Software Startups: Initial Case Study Results. 29th International Requirements Engineering Conference Workshops (REW) 206–211

Block, Z. and MacMillan, I.C. (1985) Milestones for Successful Venture Planning. Harvard Business Review, Sept 1985.

Bruegger, P., Lalanne, D., Lisowska, A. and Hirsbrunner, B. (2009). Tools for designing and prototyping activity-based pervasive application. Proc of 7th Int Conf on Mobile Comp and Multim 129–136

Budde R, Zullighoven H (1990) Prototyping revisited. Proc of COMPEURO 90:418–427

Cafer, F. and Misra, S. (2009). A cognitive requirement specification model. 24th Int Symp on Computer and Information Sciences 518–521

Chen Chen, Porter, A. and Purtilo, J. (1994) Tool support for tailored software prototyping. 3rd Symp on Assessm of Quality Softw Dev Tools 171–181

Ciriello, R.F., Richter, A. and Schwabe, G. (2017). When Prototyping Meets Storytelling: Practices and Malpractices in Innovating Software Firms. Proc 39th Int Conf on Softw Engineering in Practice Track 163–172.

Derboven J, De Roeck D, Verstraete M, Geerts D, Schneider-Barnes J, Luyten K (2010) Comparing user interaction with low and high fidelity prototypes of tabletop surfaces. Proc NordCHI 2010:148–157

Dow, S., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D. and Klemmer, S. (2011) Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. Conf on HFs in Computing Systems 2807–2816

Fairley RE, Willshire MJ (2005) Iterative rework: the good, the bad, and the ugly. Computer. 38(9):34–41

Fern, D.A. and Donaldson, S.E. (1989). Tri-Cycle: a prototype methodology for advanced software development. Proc of 22nd HICSS 377–386

Giardino C, Unterkalmsteiner M, Paternoster N, Gorschek T, Abrahamsson P (2014) What Do We Know about Software Development in Startups? IEEE Software. 31(5):28–32

Giardino C, Bajwa SS, Wang X, Abrahamsson P (2015) Key Challenges in Early-Stage Software Startups. Agile Processes in Software Engineering and Extreme Programming 2015:52–63

Goldman N, Narayanaswamy K (1992) Software Evolution through Iterative Prototyping. Int Conf on Software Engineering 1992:158–172

Grevet, C. and Gilbert, E. (2015). Piggyback Prototyping. 33rd Annual ACM Conf on Human Factors in Computing Systems 4047–4056

Gupta V, Rubalcaba L, Gupta C (2021) Multimedia Prototyping for Early-Stage Startups Endurance: Stage for New Normal? IEEE MultiMedia. 28(4):107–116. https://doi.org/10.1109/MMUL.2021.3122539

Håkansson, E. and Bjarnason, E. (2020) Including Human Factors and Ergonomics in Requirements Engineering for Digital Work Environments.  IEEE First International Workshop on Requirements Engineering for Well-Being, Aging, and Health (REWBAH)  57–66

Hakim, J. and Spitzer, T. (2000).Effective prototyping for usability. 18th Conf on Computer Documentation 47–54

Rex Hartson (2019). The UX Book. Morgan Kaufmann Publishers

Heisler, K.G., Tsai, W.T. and Ramamoorthy, C.V. (1989). Integrating the role of requirements specification into the process of prototyping: the protospec. 22nd Annual Hawaii Int Conf on System Sciences 348–357

Hendry DG, Mackenzie S, Kurth A, Spielberg F, Larkin J (2005) Evaluating paper prototypes on the street. CHI 05:1447–1450

Jaskiewicz, T. and van der Helm, A. (2018) Unlocking the Interactive Office: Concurrent Prototyping Approach. DIS 18 547–558

Käpyaho, M. and Kauppinen, M. (2015) Agile requirements engineering with prototyping: A case study. 23rd Int Requirements Engineering Conf 334–343

Karras, O., Unger-Windeler, C., Glauer, L. and Schneider, K. (2017). Video as a By-Product of Digital Prototyping: Capturing the Dynamic Aspect of Interaction. 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW) 118–124

Klotins E, Unterkalmsteiner M, Chatzipetrou P, Gorschek T, Prikladnicki R, Tripathi N, Pompermaier L (2019) A progression model of software engineering goals, challenges, and practices in start-ups. IEEE TOSEM 2019:1–1

Kordon F (1994) Proposal for a generic prototyping approach. ETFA 94:396–403

Lauesen S (2002) Software Requirements. Addison Wesley, Styles and Techniques

Lauesen, S. (2005). User interface design : a software engineering perspective . Pearson/Addison-Wesley

Lenarduzzi, V. and Taibi, D. MVP ( 2016) Explained: A Systematic Mapping Study on the Definitions of Minimal Viable Product. 42th Euromicro Conf on Softw Engineering and Adv Applications (SEAA) 112–119

Lichter H, Schneider-Hufschmidt M, Zullighoven H (1994) Prototyping in industrial software projects-bridging the gap between theory and practice. IEEE Trans on Software Engineering. 20(11):825–832

Lim, Y.-K., Stolterman, E. and Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Trans on Computer-Human Interaction . 15, 2 7:1–7:27

Liu, L. and Khooshabeh, P (2003) Paper or interactive? : A study on prototyping techniques for ubiquitous computing environments. CHI ’03 Extended Abstracts on Human Factors in Computing Systems 1030–1031

McCurdy, M., Connors, C., Pyrzak, G., Kanefsky, B. and Vera, A. (2006).Breaking the fidelity barrier: an examination of our current characterization of prototypes and an example of a mixed-fidelity success. Proc of SIGCHI Conf on HFs in Computing Systems 1233–1242

Nguyen-Duc A, Wang X, Abrahamsson P (2017) What Influences the Speed of Prototyping? An Empirical Investigation of Twenty Software Startups. Agile Proc in Softw Eng 2017:20–36

Nielsen, J. (1993). Usability engineering . AP Professional

Olsen D (2015) The Lean Product Playbook - How to Innovate with Minimum Viable Products and Rapid Customer Feedback. Wiley

Book   Google Scholar  

Osterwalder A, Pigneur Y, Bernarda G, Smith A, Papadakos T (2014) Value Proposition Design. Wiley

Paternoster N, Giardino C, Unterkalmsteiner M, Gorschek T, Abrahamsson P (2014) Software development in startup companies: A systematic mapping study. Inf & Softw Techn. 56(10):1200–1218

Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: An update. Inf Softw Technol 64(2015):1–18

Rahman, A., Razek, A. and van Husen, C. (2017) Innovation by service prototyping design dimensions and attributes, key design aspects, and toolbox. Int Conf on Engin , Techn and Innovation (ICE/ITMC) 571–576

Raja, U.A. (2009). Empirical studies of requirements validation techniques. 2nd Int Conf on Computer Control & Comm ) 1–9

Ramesh, B., Cao, L. and Baskerville, R. (2010). Agile requirements engineering practices and challenges: an empirical study. information Systems Journal . 20 5

Ratcliff B (1988) Early and not-so-early prototyping-rationale and tool support. Proc COMPSAC 88(1988):127–134

Ries, E. (2011). The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Currency

Rudd J, Stern K, Isensee S (1996) Low vs. high-fidelity prototyping debate. Interactions. 3(1):76–85. https://doi.org/10.1145/223500.223514

Schneider K (1996) Prototypes as assets, not toys. Why and how to extract knowledge from prototypes. (Experience report). ICSE 96:522–531

Sefelin, R., Tscheligi, M. and Giller, V. (2003) Paper prototyping - what is it good for? a comparison of paper- and computer-based low-fidelity prototyping. CHI ’03 Extended Abstracts on HFs in Comp. Systems 778–779

Sergio, L.M. (2015) Why is important prototyping? in Human-Computer Interaction (course material), Universitat d’Alacant: http://desarrolloweb.dlsi.ua.es/cursos/2015/hci/why-is-important-prototyping . Accessed: 2023-08-14

Shepherd, D.A. and Gruber, M. (2020). The Lean Startup Framework: Closing the Academic–Practitioner Divide - Dean A. Shepherd, Marc Gruber,. Entrepreneurship Theory and Practice

Teece DJ (2010) Business Models, Business Strategy and Innovation. Long Range Planning. 43(2):172–194

Terho H, Suonsyrjä S, Systä K (2016) The Developers Dilemma: Perfect Product Development or Fast Business Validation? Product-Focused Software Process Improvement 2016:571–579

Toffolon, C. and Dakhli, S. (2008). An Iterative Meta-Lifecycle for Software Development, Evolution and Maintenance. 3rd Int Conference on Software Engineering Advances 284–289

Tripathi N, Klotins E, Prikladnicki R, Oivo M, Pompermaier LB, Kudakacheril AS, Unterkalmsteiner M, Liukkunen K, Gorschek T (2018) An anatomy of requirements engineering in software startups using multi-vocal literature and case survey. J Syst Softw 146(2018):130–151

Tronvoll SA, Elverum CW, Welo T (2017) Prototype Experiments: Strategies and Trade-offs. Procedia CIRP 2017:554–559

Tullis, T.S. (1990). High-fidelity prototyping throughout the design process. Proceedings of the Human Factors Sciety 34th Annual Meeting 266

Wiberg, M. and Stolterman, E. (2014). What makes a prototype novel? | Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational. Proc of 8th Nordic Conf on HCI 531–540

Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements Engineering. 11(1):102–107

Yasar, A.-U.-H. (2007) Enhancing experience prototyping by the help of mixed-fidelity prototypes. Proc of 4th Int Conf on Mobile Technology 468–473

Zainuddin, F.B. and Liu, S. (2012). An Approach to Low-fidelity Prototyping Based on SOFL Informal Specification. 2012 19th Asia-Pacific Software Engineering Conference 654–663

Zink, L., Hostetter, R., Böhmer, A.I., Lindemann, U. and Knoll, A. (2017).The use of prototypes within agile product development. 2017 Int. Conf. on Engin , Techn and Innovation (ICE/ITMC) 68–77

Download references

Acknowledgements

We thank the participants at Telavox and the interviewees at the startups for good collaboration, and for investing their time and engagement in this study. This work was partly funded by the Swedish strategic research environment ELLIIT.

Open access funding provided by Lund University. The research presented in this article was partly funded by the Swedish strategic research environment ELLIIT.

Author information

Authors and affiliations.

Department of Computer Science, Lund University, Lund, Sweden

Elizabeth Bjarnason, Franz Lang & Alexander Mjöberg

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Elizabeth Bjarnason .

Ethics declarations

Conflicts of interests.

Bjarnason’s contribution to this work was funded by the Swedish strategic research environment ELLIIT.

She has no other financial or non-financial interests, or connection to any of the involved case companies, to disclose. Lang’s and Mjöberg’s initial contribution to this work (the initial mapping study) was part of their MSc project, which was performed at the case company Telavox during 2020. Since completing their MSc project, they have no financial or non-financial interests, or connections to Telavox or any of the other involved case companies, to disclose.

Additional information

Communicated by Maria Teresa Baldassarre, Markos Kalinowski.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on International Symposium on Empirical Software Engineering and Measurement (ESEM)

APPENDIX A: Focus Group Protocol

The following protocol was used to perform an initial validation of our model at a focus group with practitioners at our initial case company Telavox. The aim of the focus group was to discuss prototyping scenarios (categorised by the initial draft of our model) for five stages of RE with generic and stage-specific questions. The following questions were used for all stages:

• What is a good/bad outcome for this stage?

• How do you ensure a good outcome for this stage?

• What is clear/unclear with this stage?

• What knowledge is required for a good outcome of this stage?

• Which people should be involved in this stage?

1.1 Stage 1 – Concept exploration.

Purpose: exploration of problem domain and solutions

Scope: shallow functionality, low degree of refinement

Use: mainly internal usage

Strategy: parallel exploration

Domain- and product-level requirements are elicited by exploring ideas and solution space. Prototypes of shallow functional scope and low refinement are tested with low-cost methods such a paper prototyping. The main focus is on internal learning. Sharing knowledge externally is optional.

• Which are the best ways to brainstorm ideas for a new system? What are the pros and cons with these?

• Pros and cons of not having thought of a new idea prior to talking to a customer about a future system?

1.2 Stage 2 - Eliciting customer needs.

Purpose: Testing market desirability, Exploration

Scope: narrow and shallow functionality, low degree of refinement

Use: any review method with users/customers.

Strategy: flexible exploration

Domain-level requirements are elicited and market desirability tested. The aim is to understand customer needs with a focus on the role of the system from the users’ perspective. Simple prototypes of selected system concepts are designed and presented to users.

• In this stage, a simple paper prototype can be of use. What can you vs. can you not learn from such a prototype?

• It can feel difficult to present early prototypes for discussions. Do you agree, what pros and cons do you see with this?

1.3 Stage 3 – Identify system scope & requirements.

Purpose: Testing market desirability, Exploration (external), Communication (internal)

Use: any review method, internal and external use

Strategy: point-based exploration

Prototyping is used to identify system scope and requirements. A simple prototype with shallow functional scope and a low degree of refinement is used externally to pinpoint requirements that will satisfy customer needs, and internally to communicate and align regarding requirements.

• Requirements are identified in traditional, as well as in agile projects, but noted in different ways and forms. In traditional RE, requirements are documented in an SRS, often kept in a spreadsheet. Do you experience a need for such documentation in agile projects? How do you achieve this?

1.4 Stage 4 – Test and improve system scope & requirements.

Purpose: Usability testing, Test market desirability, Communication (internal & external)

Scope: broad and shallow functionality, low degree of refinement

Use: any review method, external and internal use

Product scope and requirements are communicated, and usability and market desirability is validated. Communication and alignment of requirements between customer and development, and within a project is facilitated by prototypes that act as requirements specifications. Simple prototypes (broad and shallow functionality with low refinement) represent the current understanding. User feedback is captured by demonstrations, scenario testing, or free testing. A flexible exploration strategy is used to develop prototypes based on feedback.

• How early on is it good to test prototypes?

1.5 Stage 5 – Confirm system scope & requirements.

Purpose: Communication, Validation of Market desirability and Usability, Optimisation, Incremental development

Scope: broad and mid/deep functionality, low/mid visual refinement

Use: any review method

Prototyping is used to communicate with customers and to agree on system scope and requirements. The prototype is broader and more refined than in previous stage, particular for functionality, and can be a true (throw-away) prototype or an early version of the system.

• When do you need to perform a more formal validation of the requirements and user-interface design for a product?

• What is the difference between performing a formal validation and having a colleague perform the validation?

APPENDIX B: Interview Guide for Multi-Case Study of Startups

The following interview guide was used with entrepreneurs in our multi-case study of software startup companies. The main aim of the interviews was to explore prototyping practices of software startups and contextual factors that may influence these. The interviews were also designed to validate the prototyping aspects identified in our theoretical model (PAM) and explore to the model’s usefulness in supporting practitioners to describing their prototyping practices.

1.1 Interview introduction – 10 min

1) Present the study (purpose and time frame), main researcher, policy for NDA & confidentiality. GDPR paper, recording etc

2) Interviewee presentation : current role, main area of expertise, #years at startup/in field, current and previous experience of startup ventures

1.2 Contextual characteristics [Business & Tech] – 10 min

4) Company/startup : company origin/history, age (years), size (employees, teams)

5) Product : domain, type of VP [SW-based product, content, service, experiences, user data Teece 2010 )]

5) [B] Footnote 5 Business model : customer type (market/bespoke, B2B), revenue model, channels

6) [B] Show model of start-up life-cycle maturity  (Klotins et al. 2019 )

a) What stage and status are you currently in?

b) Describe current main focus and goals for

i) product development (ideation, building, variations)

ii) marketing/growth

iii) operations & customer support

c) What stages & status have you been through?

7) [B] Startup challenges & characteristics (based on (Berg et al. 2018 ) and (Giardino et al. 2015 ), grouped by Time & Resources, Business vs Technology focus, Organisation)

a) How does your startup relate to these, for each category?

1.3 RE practices – 10 min

For your current stage:

8) What are your main requirements sources : internal/external, Tech/Business focus?

9) Do you currently have any software development? Development model (agile, traditional, hybrid), Size (#engineers & teams)

10) How do you do handle the following? Techniques based on (Klotins et al. 2019 ) and (Lauesen 2002 ) as checklist

a) Elicitation

b) Validation

c) Communication of ideas & requirements (primarily externally to customers and sponsors/investors)

1.4 Prototyping – 25 min

11) What does prototyping mean to you? Simple sketches, mock-ups, MVPs?

12) Describe how you use prototyping, in what stages, for what purposes, with what scope and how.

13) Present prototyping aspects model. How do you relate to each aspect in your prototyping practices :

b) Scope of prototype

c) Use of prototype: review method, environment.

d) Strategy for handling uncertainties

14) How do you reason concerning the cost-benefit balance for prototyping?

15) Do you use any tools for prototyping?

16) Does your prototyping approach vary, if so how and why?

a) for different purposes, such as eliciting, validating, and communicating?

b) for different stakeholders?

c) due to different points-in time, e.g. as your start-up matures?

1.5 Future work – 5 min

17) What would you like to improve around prototyping in your startup?

18) What topics/areas/questions/problems within prototyping would you as a startup want research to address & investigate?

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and Permissions

About this article

Bjarnason, E., Lang, F. & Mjöberg, A. An empirically based model of software prototyping: a mapping study and a multi-case study. Empir Software Eng 28 , 115 (2023). https://doi.org/10.1007/s10664-023-10331-w

Download citation

Accepted : 19 April 2023

Published : 30 August 2023

DOI : https://doi.org/10.1007/s10664-023-10331-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Prototyping
• Requirements engineering
• Systematic mapping study

Advertisement

• Find a journal
• Publish with us

Help | Advanced Search

Computer Science > Cryptography and Security

Title: an empirical study & evaluation of modern captchas.

Abstract: For nearly two decades, CAPTCHAs have been widely used as a means of protection against bots. Throughout the years, as their use grew, techniques to defeat or bypass CAPTCHAs have continued to improve. Meanwhile, CAPTCHAs have also evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots (machines) and humans. Given this long-standing and still-ongoing arms race, it is critical to investigate how long it takes legitimate users to solve modern CAPTCHAs, and how they are perceived by those users. In this work, we explore CAPTCHAs in the wild by evaluating users' solving performance and perceptions of unmodified currently-deployed CAPTCHAs. We obtain this data through manual inspection of popular websites and user studies in which 1,400 participants collectively solved 14,000 CAPTCHAs. Results show significant differences between the most popular types of CAPTCHAs: surprisingly, solving time and user perception are not always correlated. We performed a comparative study to investigate the effect of experimental context -- specifically the difference between solving CAPTCHAs directly versus solving them as part of a more natural task, such as account creation. Whilst there were several potential confounding factors, our results show that experimental context could have an impact on this task, and must be taken into account in future CAPTCHA studies. Finally, we investigate CAPTCHA-induced user task abandonment by analyzing participants who start and do not complete the task.

Submission history

• Download a PDF of the paper titled An Empirical Study & Evaluation of Modern CAPTCHAs, by Andrew Searles and 5 other authors PDF
• Other formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

ORIGINAL RESEARCH article

This article is part of the research topic.

Sustainable Housing and New Perspectives in Urban and Rural Development

How the Rural Infrastructure Construction Drives Rural Economic Development through Rural Living Environment Governance-Case Study of 285 Cities in China

• 1 Heilongjiang University, China

The final, formatted version of the article will be published soon.

With deepening of rural revitalization strategy, rural infrastructure construction plays an important role in living environment management and economic development. Based on the mediation model, this paper takes case study of 285 cities in China from 2017 to 2022 as samples, constructs the explanatory variable, the explained variables and the mediator variable by entropy method, empirically analyzes the impact of rural infrastructure construction on rural living environment governance and economic development, as well as the mediation role played by rural living environment governance. It is found that there is a significant positive impact between rural infrastructure construction and rural economic development, and rural infrastructure construction can promote economic development through rural living environment governance. Further analysis show that the impact of infrastructure on economic development presents heterogeneity, and the impacts of rural infrastructure construction on local economic development and on local economic development through living environment governance in the eastern and central China is stronger than that in the western China. After controlling a series of variables related to rural infrastructure, and performing endogeneity tests and robustness tests such as tail-shrinking regression and principal component analysis, the regression results are still robust. This paper firstly provides scientific empirical evidence for the hypothesis that rural infrastructure construction promotes economic development through living environment governance, and secondly confirms the necessity of strengthening infrastructure construction in China to promote rural revitalization, providing a policy basis for scientific decision-making, and finally finds an important way out to solve the problem of unbalanced economic development in rural areas to some extent.

Keywords: Rural Infrastructure Construction, Rural living environment, rural economic development, Rural revitalization, Transportation

Received: 21 Aug 2023; Accepted: 01 Sep 2023.

Copyright: © 2023 Du and Jiao. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Mx. Xuan Du, Heilongjiang University, Harbin, China

People also looked at