types of systematic literature reviews

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters & Hedges
• Grey Literature
• Managing & Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Sep 12, 2023 2:52 PM
• URL: https://guides.library.ucla.edu/systematicreviews

Systematic Reviews: Types of literature review, methods, & resources

• Types of literature review, methods, & resources
• Protocol and registration
• Search strategy
• Medical Literature Databases to search
• Study selection and appraisal
• Data Extraction/Coding/Study characteristics/Results
• Reporting the quality/risk of bias
• Manage citations using RefWorks This link opens in a new window
• GW Box file storage for PDF's This link opens in a new window

Analytical reviews

GUIDELINES FOR HOW TO CARRY OUT AN ANALYTICAL REVIEW OF QUANTITATIVE RESEARCH

Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network. (Tracking and listing over 550 reporting guidelines for various different study types including Randomised trials, Systematic reviews, Study protocols, Diagnostic/prognostic studies, Case reports, Clinical practice guidelines, Animal pre-clinical studies, etc). http://www.equator-network.org/resource-centre/library-of-health-research-reporting/

When comparing therapies :

PRISMA (Guideline on how to perform and write-up a systematic review and/or meta-analysis of the outcomes reported in multiple clinical trials of therapeutic interventions. PRISMA  replaces the previous QUORUM statement guidelines ):  Liberati, A,, Altman, D,, Moher, D, et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.  Plos Medicine, 6 (7):e1000100. doi:10.1371/journal.pmed.1000100 

When comparing diagnostic methods :

STAndards for the Reporting of Diagnostic accuracy studies (STARD) Statement. (Reporting guidelines for writing up a study comparing the accuracy of competing diagnostic methods)  http://www.stard-statement.org/

When evaluating clinical practice guidelines :

AGREE Research Trust (ART) (2013).  Appraisal of Guidelines for Research & Evaluation (AGREE-II) . (A 23-item instrument for as sessing th e quality of Clinical Practice Guidelines. Used internationally for evaluating or deciding which guidelines could be recommended for use in practice or to inform health policy decisions.)

National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument (2019). (A 15-item instrument using scales of 1-5 to evaluate a guideline's adherence to the Institute of Medicine's standard for trustworthy guidelines. It has good external validity among guideline developers and good interrater reliability across trained reviewers.)

When reviewing genetics studies

Human genetics review reporting guidelines.  Little J, Higgins JPT (eds.). The HuGENet™ HuGE Review Handbook, version 1.0 . 

When you need to re-analyze individual participant data

If you wish to collect, check, and re-analyze individual participant data (IPD) from clinical trials addressing a particular research question, you should follow the  PRISMA-IPD  guidelines as reported in  Stewart, L.A., Clarke, M., Rovers, M., et al. (2015). Preferred Reporting Items for a Systematic Review and Meta-analysis of Individual Participant Data: The PRISMA-IPD Statement. JAMA, 313(16):1657-1665. doi:10.1001/jama.2015.3656 .

When comparing Randomized studies involving animals, livestock, or food:

O’Connor AM, et al. (2010).  The REFLECT statement: methods and processes of creating reporting guidelines for randomized controlled trials for livestock and food safety by modifying the CONSORT statement.  Zoonoses Public Health. 57(2):95-104. Epub 2010/01/15. doi: 10.1111/j.1863-2378.2009.01311.x. PubMed PMID: 20070653.

Sargeant JM, et al. (2010).  The REFLECT Statement: Reporting Guidelines for Randomized Controlled Trials in Livestock and Food Safety: Explanation and Elaboration.  Zoonoses Public Health. 57(2):105-36. Epub 2010/01/15. doi: JVB1312 [pii] 10.1111/j.1863-2378.2009.01312.x. PubMed PMID: 20070652.

GUIDELINES FOR HOW TO WRITE UP FOR PUBLICATION THE RESULTS OF ONE QUANTITATIVE CLINICAL TRIAL

When reporting the results of a Randomized Controlled Trial :

Consolidated Standards of Reporting Trials (CONSORT) Statement. (2010 reporting guideline for writing up a Randomized Controlled Clinical Trial).  http://www.consort-statement.org . Since updated in 2022, see Butcher, M. A., et al. (2022). Guidelines for Reporting Outcomes in Trial Reports: The CONSORT-Outcomes 2022 Extension . JAMA : the Journal of the American Medical Association, 328(22), 2252–2264. https://doi.org/10.1001/jama.2022.21022

Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010). Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research. PLoS Biology, 8(6), e1000412–e1000412. https://doi.org/10.1371/journal.pbio.1000412 (A 20-item checklist, following the CONSORT approach, listing the information that published articles reporting research using animals should include, such as the number and specific characteristics of animals used; details of housing and husbandry; and the experimental, statistical, and analytical methods used to reduce bias.)

Narrative reviews

GUIDELINES  FOR HOW TO CARRY OUT  A  NARRATIVE REVIEW / QUALITATIVE RESEARCH /  OBSERVATIONAL STUDIES

Campbell, M. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ, 368. doi: https://doi.org/10.1136/bmj.l6890  (guideline on how to analyse evidence for a narrative review, to provide a recommendation based on heterogenous study types).

Community Preventive Services Task Force (2021).  The Methods Manual for Community Guide Systematic Reviews . (Public Health Prevention systematic review guidelines)

Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network. (Tracking and listing over 550 reporting guidelines for various different study types including Observational studies, Qualitative research, Quality improvement studies, and Economic evaluations). http://www.equator-network.org/resource-centre/library-of-health-research-reporting/

Cochrane Qualitative & Implementation Methods Group. (2019). Training resources. Retrieved from  https://methods.cochrane.org/qi/training-resources . (Training materials for how to do a meta-synthesis, or qualitative evidence synthesis). 

Cornell University Library (2019). Planning worksheet for structured literature reviews. Retrieved 4/8/22 from  https://osf.io/tnfm7/  (offers a framework for a narrative literature review).

Green, B. N., Johnson, C. D., & Adams, A. (2006).  Writing narrative literature reviews for peer-reviewed journals: secrets of the trade . Journal of Chiropractic Medicine, 5(3): 101-117. DOI: 10.1016/ S0899-3467 (07)60142-6.  This is a very good article about what to take into consideration when writing any type of narrative review.

When reviewing observational studies/qualitative research :

STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement. (Reporting guidelines for various types of health sciences observational studies).  http://www.strobe-statement.org 

Meta-analysis of Observational Studies in Epidemiology (MOOSE)  http://jama.jamanetwork.com/article.aspx?articleid=192614

RATS Qualitative research systematic review guidelines.  https://www.equator-network.org/reporting-guidelines/qualitative-research-review-guidelines-rats/

Methods/Guidance

Right Review , this decision support website provides an algorithm to help reviewers choose a review methodology from among 41 knowledge synthesis methods.

The Systematic Review Toolbox , an online catalogue of tools that support various tasks within the systematic review and wider evidence synthesis process. Maintained by the UK University of York Health Economics Consortium, Newcastle University NIHR Innovation Observatory, and University of Sheffield School of Health and Related Research.

Institute of Medicine. (2011).  Finding What Works in Health Care: Standards for Systematic Reviews . Washington, DC: National Academies  (Systematic review guidelines from the Health and Medicine Division (HMD) of the U.S. National Academies of Sciences, Engineering, and Medicine (formerly called the Institute of Medicine)).

International Committee of Medical Journal Editors (2022).  Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals . Guidance on how to prepare a manuscript for submission to a Medical journal.

Cochrane Handbook of Systematic Reviews of Interventions (International Cochrane Collaboration systematic review guidelines).

Cochrane Methods Support Unit, webinar recordings on methodological support questions 

Cochrane Qualitative & Implementation Methods Group. (2019). Training resources. Retrieved from https://methods.cochrane.org/qi/training-resources . (How to do a meta-synthesis, or qualitative evidence synthesis). 

Center for Reviews and Dissemination (University of York, England) (2009).  Systematic Reviews: CRD's guidance for undertaking systematic reviews in health care . (British systematic review guidelines). 

Agency for Health Research & Quality (AHRQ) (2013). Methods guide for effectiveness and comparative effectiveness reviews . (U.S. comparative effectiveness review guidelines)

Hunter, K. E., et al. (2022). Searching clinical trials registers: guide for systematic reviewers.  BMJ (Clinical research ed.) ,  377 , e068791. https://doi.org/10.1136/bmj-2021-068791

Patient-Centered Outcomes Research Institute (PCORI).  The PCORI Methodology Report . (A 47-item methodology checklist for U.S. patient-centered outcomes research. Established under the Patient Protection and Affordable Care Act, PCORI funds the development of guidance on the comparative effectivess of clinical healthcare, similar to the UK National Institute for Clinical Evidence but without reporting cost-effectiveness QALY metrics). 

Canadian Agency for Drugs and Technologies in Health (CADTH) (2019). Grey Matters: a practical tool for searching health-related grey literature. Retrieved from https://www.cadth.ca/resources/finding-evidence/grey-matters . A checklist of N American & international online databases and websites you can use to search for unpublished reports, posters, and policy briefs, on topics including general medicine and nursing, public and mental health, health technology assessment, drug and device regulatory, approvals, warnings, and advisories.

Hempel, S., Xenakis, L., & Danz, M. (2016). Systematic Reviews for Occupational Safety and Health Questions: Resources for Evidence Synthesis. Retrieved 8/15/16 from http://www.rand.org/pubs/research_reports/RR1463.html . NIOSH guidelines for how to carry out a systematic review in the occupational safety and health domain.

A good source for reporting guidelines is the  NLM's  Research Reporting Guidelines and Initiatives .

Grading of Recommendations Assessment, Development and Evaluation (GRADE). (An international group of academics/clinicians working to promote a common approach to grading the quality of evidence and strength of recommendations.) 

Phillips, B., Ball, C., Sackett, D., et al. (2009). Oxford Centre for Evidence Based Medicine: Levels of Evidence. Retrieved 3/20/17 from https://www.cebm.net/wp-content/uploads/2014/06/CEBM-Levels-of-Evidence-2.1.pdf . (Another commonly used criteria for grading the quality of evidence and strength of recommendations, developed in part by EBM guru David Sackett.) 

Systematic Reviews for Animals & Food  (guidelines including the REFLECT statement for carrying out a systematic review on animal health, animal welfare, food safety, livestock, and agriculture)

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies . Health Information & Libraries Journal, 26(2), 91-108. doi:10.1111/j.1471-1842.2009.00848.x. (Describes 14 different types of literature and systematic review, useful for thinking at the outset about what sort of literature review you want to do.)

Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: exploring review types and associated information retrieval requirements . Health information and libraries journal, 36(3), 202–222. doi:10.1111/hir.12276  (An updated look at different types of literature review, expands on the Grant & Booth 2009 article listed above).

Garrard, J. (2007).  Health Sciences Literature Review Made Easy: The Matrix Method  (2nd Ed.).   Sudbury, MA:  Jones & Bartlett Publishers. (Textbook of health sciences literature search methods).

Zilberberg, M. (2012).  Between the lines: Finding the truth in medical literature . Goshen, MA: Evimed Research Press. (Concise book on foundational concepts of evidence-based medicine).

Lang, T. (2009). The Value of Systematic Reviews as Research Activities in Medical Education . In: Lang, T. How to write, publish, & present in the health sciences : a guide for clinicians & laboratory researchers. Philadelphia : American College of Physicians.  (This book chapter has a helpful bibliography on systematic review and meta-analysis methods)

Brown, S., Martin, E., Garcia, T., Winter, M., García, A., Brown, A., Cuevas H.,  & Sumlin, L. (2013). Managing complex research datasets using electronic tools: a meta-analysis exemplar . Computers, Informatics, Nursing: CIN, 31(6), 257-265. doi:10.1097/NXN.0b013e318295e69c. (This article advocates for the programming of electronic fillable forms in Adobe Acrobat Pro to feed data into Excel or SPSS for analysis, and to use cloud based file sharing systems such as Blackboard, RefWorks, or EverNote to facilitate sharing knowledge about the decision-making process and keep data secure. Of particular note are the flowchart describing this process, and their example screening form used for the initial screening of abstracts).

Brown, S., Upchurch, S., & Acton, G. (2003). A framework for developing a coding scheme for meta-analysis . Western Journal Of Nursing Research, 25(2), 205-222. (This article describes the process of how to design a coded data extraction form and codebook, Table 1 is an example of a coded data extraction form that can then be used to program a fillable form in Adobe Acrobat or Microsoft Access).

Elamin, M. B., Flynn, D. N., Bassler, D., Briel, M., Alonso-Coello, P., Karanicolas, P., & ... Montori, V. M. (2009). Choice of data extraction tools for systematic reviews depends on resources and review complexity .  Journal Of Clinical Epidemiology ,  62 (5), 506-510. doi:10.1016/j.jclinepi.2008.10.016  (This article offers advice on how to decide what tools to use to extract data for analytical systematic reviews).

Riegelman R.   Studying a Study and Testing a Test: Reading Evidence-based Health Research , 6th Edition.  Lippincott Williams & Wilkins, 2012. (Textbook of quantitative statistical methods used in health sciences research).

Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic Reviews, 480. doi:10.1186/s13643-015-0067-6

Guyatt, G., Rennie, D., Meade, M., & Cook, D. (2015). Users' guides to the medical literature (3rd ed.). New York: McGraw-Hill Education Medical.  (This is a foundational textbook on evidence-based medicine and of particular use to the reviewer who wants to learn about the different types of published research article e.g. "what is a case report?" and to understand what types of study design best answer what types of clinical question).

Glanville, J., Duffy, S., Mccool, R., & Varley, D. (2014). Searching ClinicalTrials.gov and the International Clinical Trials Registry Platform to inform systematic reviews: what are the optimal search approaches? Journal of the Medical Library Association : JMLA, 102(3), 177–183. https://doi.org/10.3163/1536-5050.102.3.007

Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan a web and mobile app for systematic reviews.  Systematic Reviews, 5 : 210, DOI: 10.1186/s13643-016-0384-4. http://rdcu.be/nzDM

Kwon Y, Lemieux M, McTavish J, Wathen N. (2015). Identifying and removing duplicate records from systematic review searches. J Med Libr Assoc. 103 (4): 184-8. doi: 10.3163/1536-5050.103.4.004. https://www.ncbi.nlm.nih.gov/pubmed/26512216

Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. (2016). De-duplication of database search results for systematic reviews in EndNote. J Med Libr Assoc. 104 (3):240-3. doi: 10.3163/1536-5050.104.3.014. Erratum in: J Med Libr Assoc. 2017 Jan;105(1):111. https://www.ncbi.nlm.nih.gov/pubmed/27366130

McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol. 2016;75:40–46. doi: 10.1016/j.jclinepi.2016.01.021 . PRESS is a guideline with a checklist for librarians to critically appraise the search strategy for a systematic review literature search.

Clark, JM, Sanders, S, Carter, M, Honeyman, D, Cleo, G, Auld, Y, Booth, D, Condron, P, Dalais, C, Bateup, S, Linthwaite, B, May, N, Munn, J, Ramsay, L, Rickett, K, Rutter, C, Smith, A, Sondergeld, P, Wallin, M, Jones, M & Beller, E 2020, 'Improving the translation of search strategies using the Polyglot Search Translator: a randomized controlled trial',  Journal of the Medical Library Association , vol. 108, no. 2, pp. 195-207.

Journal articles describing systematic review methods can be searched for in PubMed using this search string in the PubMed search box: sysrev_methods [sb] . 

Software tools for systematic reviews

• Covidence GW in 2019 has bought a subscription to this Cloud based tool for facilitating screening decisions, used by the Cochrane Collaboration. Register for an account.
• NVIVO for analysis of qualitative research NVIVO is used for coding interview data to identify common themes emerging from interviews with several participants. GW faculty, staff, and students may download NVIVO software.
• RedCAP RedCAP is software that can be used to create survey forms for research or data collection or data extraction. It has very detailed functionality to enable data exchange with Electronic Health Record Systems, and to integrate with study workflow such as scheduling follow up reminders for study participants.
• Systematic Review Toolbox Select the Healthcare discipline and features you want a tool to support.
• SRDR tool from AHRQ Free, web-based and has a training environment, tutorials, and example templates of systematic review data extraction forms
• RevMan 5 RevMan 5 is the desktop version of the software used by Cochrane systematic review teams. RevMan 5 is free for academic use and can be downloaded and configured to run as stand alone software that does not connect with the Cochrane server if you follow the instructions at https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman/revman-5-download/non-cochrane-reviews
• Rayyan Free, web-based tool for collecting and screening citations. It has options to screen with multiple people, masking each other.
• GradePro Free, web application to create, manage and share summaries of research evidence (called Evidence Profiles and Summary of Findings Tables) for reviews or guidelines, uses the GRADE criteria to evaluate each paper under review.
• DistillerSR Needs subscription. Create coded data extraction forms from templates.
• EPPI Reviewer Needs subscription. Like DistillerSR, tool for text mining, data clustering, classification and term extraction
• SUMARI Needs subscription. Qualitative data analysis.
• Dedoose Needs subscription. Qualitative data analysis, similar to NVIVO in that it can be used to code interview transcripts, identify word co-occurence, cloud based.
• Meta-analysis software for statistical analysis of data for quantitative reviews SPSS, SAS, and STATA are popular analytical statistical software that include macros for carrying out meta-analysis. Himmelfarb has SPSS on some 3rd floor computers, and GW affiliates may download SAS to your own laptop from the Division of IT website. To perform mathematical analysis of big data sets there are statistical analysis software libraries in the R programming language available through GitHub and RStudio, but this requires advanced knowledge of the R and Python computer languages and data wrangling/cleaning.
• PRISMA 2020 flow diagram generator The PRISMA Statement website has a page listing example flow diagram templates and a link to software for creating PRISMA 2020 flow diagrams using R software.

GW researchers may want to consider using Refworks to manage citations, and GW Box to store the full text PDF's of review articles. You can also use online survey forms such as Qualtrics, RedCAP, or Survey Monkey, to design and create your own coded fillable forms, and export the data to Excel or one of the qualitative analytical software tools listed above.

Forest Plot Generators

• RevMan 5 the desktop version of the software used by Cochrane systematic review teams. RevMan 5 is free for academic use and can be downloaded and configured to run as stand alone software that does not connect with the Cochrane server if you follow the instructions at https://training.cochrane.org/online-learning/core-software-cochrane-reviews/revman/revman-5-download/non-cochrane-reviews.
• Meta-Essentials a free set of workbooks designed for Microsoft Excel that, based on your input, automatically produce meta-analyses including Forest Plots. Produced for Erasmus University Rotterdam joint research institute.
• Neyeloff, Fuchs & Moreira Another set of Excel worksheets and instructions to generate a Forest Plot. Published as Neyeloff, J.L., Fuchs, S.C. & Moreira, L.B. Meta-analyses and Forest plots using a microsoft excel spreadsheet: step-by-step guide focusing on descriptive data analysis. BMC Res Notes 5, 52 (2012). https://doi-org.proxygw.wrlc.org/10.1186/1756-0500-5-52
• For R programmers instructions are at https://cran.r-project.org/web/packages/forestplot/vignettes/forestplot.html and you can download the R code package from https://github.com/gforge/forestplot
• << Previous: Home
• Next: Protocol and registration >>

• Last Updated: Oct 27, 2023 10:59 AM
• URL: https://guides.himmelfarb.gwu.edu/systematic_review

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

• University of Michigan Library
• Research Guides

Systematic Reviews

• Types of Reviews
• Work with a Search Expert

Choosing a Review Type

Types of literature reviews.

• Evidence in a Systematic Review
• Information Sources
• Search Strategy
• Managing Records
• Selection Process
• Data Collection Process
• Study Risk of Bias Assessment
• Reporting Results
• For Search Professionals

This guide focuses on the methodology for systematic reviews (SRs), but an SR may not be the best methodology to use to meet your project's goals. Use the articles listed here or in the Types of Literature Reviews box below for information about additional methodologies that could better fit your project. 

• Haddaway NR, Lotfi T, Mbuagbaw L. Systematic reviews: A glossary for public health . Scand J Public Health. 2022 Feb 9:14034948221074998. doi: 10.1177/14034948221074998. Epub ahead of print. PMID: 35139715.
• Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies . Health Info Libr J. 2009 Jun;26(2):91-108. Defines 14 types of reviews and provides a helpful summary table on pp. 94-95.
• Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements . Health Info Libr J . 2019;36(3):202–222. doi:10.1111/hir.12276
• If you're not sure what type of review is right for your quantitative review, use this tool to find the best methodology for your project.:What Review is Right for You? https://whatreviewisrightforyou.knowledgetranslation.net

Meta-Analyses

• Comparative Effectiveness
• systematically and transparently searches for a broad range of information to synthesize, in order to find the effect of an intervention.
• uses a protocol 
• has a clear data extraction and management plan.
• Time-intensive and often take months to a year or more to complete, even with a multi-person team. 

NOTE: The term "systematic review" is also used incorrectly as a blanket term for other types of reviews.

Methodological Guidance

• Finding What Works in Health Care: Standards for Systematic Reviews. 2011. Institute of Medicine. http://books.nap.edu/openbook.php?record_id=13059
• Cochrane Handbook of Systematic Reviews of Interventions, v. 6. 2019. https://training.cochrane.org/handbook
• The Joanna Briggs Reviewers Manual. 2014. https://jbi-global-wiki.refined.site/space/MANUAL
• The Community Guide/Methods/Systematic Review Methods. 2014. The Community Preventive Services Task Force. http://www.thecommunityguide.org/about/methods.html

For issues in systematic reviews, especially in social science or other qualitative research: 

• Some Potential "Pitfalls" in the Construction of Educational Systematic Reviews. https://doi.org/10.1007/s40596-017-0675-7
• Lescoat, A., Murphy, S. L., Roofeh, D., et al. (2021). Considerations for a combined index for limited cutaneous systemic sclerosis to support drug development and improve outcomes. https://doi.org/10.1177/2397198320961967
• DeLong, M. R., Tandon, V. J., Bertrand, A. A. (2021). Review of Outcomes in Prepectoral Prosthetic Breast Reconstruction with and without Surgical Mesh Assistance.  https://pubmed.ncbi.nlm.nih.gov/33177453/
• Carey, M. R., Vaughn, V. M., Mann, J. (2020). Is Non-Steroidal Anti-Inflammatory Therapy Non-Inferior to Antibiotic Therapy in Uncomplicated Urinary Tract Infections: a Systematic Review.  https://pubmed.ncbi.nlm.nih.gov/32270403/
• Statistical technique for combining the findings from disparate  quantitative studies.
• Uses statistical methods to objectively evaluate, synthesize, and summarize results.
• May be conducted independently or as part of a systematic review.
• Cochrane Handbook, Ch 10: Analysing data and undertaking meta-analyses https://training.cochrane.org/handbook/current/chapter-10
• Bauer, M. E., Toledano, R. D., Houle, T., et al. (2020). Lumbar neuraxial procedures in thrombocytopenic patients across populations: A systematic review and meta-analysis. https://pubmed.ncbi.nlm.nih.gov/31810860/ 6
• Mailoa J, Lin GH, Khoshkam V, MacEachern M, et al. Long-Term Effect of Four Surgical Periodontal Therapies and One Non-Surgical Therapy: A Systematic Review and Meta-Analysis. https://pubmed.ncbi.nlm.nih.gov/26110453/

Umbrella Reviews

• Reviews other systematic reviews on a topic. 
• Often defines a broader question than is typical of a traditional systematic review.
• Most useful when there are competing interventions to consider.
• Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses .  https://pubmed.ncbi.nlm.nih.gov/35081993
• Aromataris, E., Fernandez, R., Godfrey, C. M., Holly, C., Khalil, H., & Tungpunkom, P.  2015 Methodology for JBI Umbrella Reviews. https://ro.uow.edu.au/cgi/viewcontent.cgi?articl.
• Gastaldon, C., Solmi, M., Correll, C. U., et al. (2022). Risk factors of postpartum depression and depressive symptoms: umbrella review of current evidence from systematic reviews and meta-analyses of observational studies. https://pubmed.ncbi.nlm.nih.gov/35081993/
• Blodgett, T. J., & Blodgett, N. P. (2021). Melatonin and melatonin-receptor agonists to prevent delirium in hospitalized older adults: An umbrella review.   https://pubmed.ncbi.nlm.nih.gov/34749057/

Comparative effectiveness 

• Systematic reviews of existing research on the effectiveness, comparative effectiveness, and comparative harms of different health care interventions.
•  Intended to provide relevant evidence to inform real-world health care decisions for patients, providers, and policymakers.
• “Methods Guide for Effectiveness and Comparative Effectiveness Reviews.” Methods Guide for Effectiveness and Comparative Effectiveness Reviews https://effectivehealthcare.ahrq.gov/products/collections/cer-methods-guide
• Main document of above guide :  https://effectivehealthcare.ahrq.gov/sites/default/files/pdf/cer-methods-guide_overview.pdf .
• Tanni KA, Truong CB, Johnson BS, Qian J. Comparative effectiveness and safety of eribulin in advanced or metastatic breast cancer: a systematic review and meta-analysis. Crit Rev Oncol Hematol. 2021 Jul;163:103375. doi: 10.1016/j.critrevonc.2021.103375. Epub 2021 Jun 2. PMID: 34087344.
• Rice D, Corace K, Wolfe D, Esmaeilisaraji L, Michaud A, Grima A, Austin B, Douma R, Barbeau P, Butler C, Willows M, Poulin PA, Sproule BA, Porath A, Garber G, Taha S, Garner G, Skidmore B, Moher D, Thavorn K, Hutton B. Evaluating comparative effectiveness of psychosocial interventions adjunctive to opioid agonist therapy for opioid use disorder: A systematic review with network meta-analyses. PLoS One. 2020 Dec 28;15(12):e0244401. doi: 10.1371/journal.pone.0244401. PMID: 33370393; PMCID: PMC7769275.

​ Scoping Review or Evidence Map

Systematically and transparently collect and  categorize  existing evidence on a broad question of  policy or management importance.

Seeks to identify research gaps and opportunities for evidence synthesis rather than searching for the effect of an intervention. 

May critically evaluate existing evidence, but does not attempt to synthesize the results in the way a systematic review would. (see  EE Journal  and  CIFOR )

May take longer than a systematic review.

• For useful guidance on whether to conduct a scoping review or not, see Figure 1 in this article. Pollock, D , Davies, EL , Peters, MDJ , et al. Undertaking a scoping review: A practical guide for nursing and midwifery students, clinicians, researchers, and academics . J Adv Nurs . 2021 ; 77 : 2102 – 2113 . https://doi.org/10.1111/jan.14743For a helpful

Hilary Arksey & Lisa O'Malley (2005) Scoping studies: towards a methodological framework http://10.1080/1364557032000119616

Aromataris E, Munn Z, eds. (2020) . JBI Manual for Evidence Synthesis.  JBI. Chapter 11: Scoping Reviews. https://wiki.jbi.global/display/MANUAL/Chapter+11%3A+Scoping+reviews

Munn Z, Peters MD, Stern C, Tet al. (2018)  Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. https://pubmed.ncbi.nlm.nih.gov/30453902/

Tricco AC, Lillie E, Zarin W, et al.. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018 Oct 2;169(7):467-473. doi: 10.7326/M18-0850. Epub 2018 Sep 4. PMID: 30178033.  https://www.acpjournals.org/doi/epdf/10.7326/M18-0850

Bouldin E, Patel SR, Tey CS, et al. Bullying and Children who are Deaf or Hard-of-hearing: A Scoping Review. https://pubmed.ncbi.nlm.nih.gov/33438758

Finn M, Gilmore B, Sheaf G, Vallières F. What do we mean by individual capacity strengthening for primary health care in low- and middle-income countries? A systematic scoping review to improve conceptual clarity. https://pubmed.ncbi.nlm.nih.gov/33407554/

Hirt J, Nordhausen T, Meichlinger J, Braun V, Zeller A, Meyer G. Educational interventions to improve literature searching skills in the health sciences: a scoping review.  https://pubmed.ncbi.nlm.nih.gov/33013210/

​ Rapid Review

Useful for addressing issues needing timely decisions, such as developing policy recommendations. 

Applies systematic review methodology within a time-constrained setting.

Employs intentional, methodological "shortcuts" (limiting search terms for example) at the risk of introducing bias.

Defining characteristic is the transparency of team methodological choices.

Garritty, Chantelle, Gerald Gartlehner, Barbara Nussbaumer-Streit, Valerie J. King, Candyce Hamel, Chris Kamel, Lisa Affengruber, and Adrienne Stevens. “Cochrane Rapid Reviews Methods Group Offers Evidence-Informed Guidance to Conduct Rapid Reviews.” Journal of Clinical Epidemiology 130 (February 2021): 13–22. https://doi.org/10.1016/j.jclinepi.2020.10.007 .

Klerings I , Robalino S , Booth A , et al. Rapid reviews methods series: Guidance on literature search. BMJ Evidence-Based Medicine. 19 April 2023. https:// 10.1136/bmjebm-2022-112079

WHO. “WHO | Rapid Reviews to Strengthen Health Policy and Systems: A Practical Guide.” World Health Organization. Accessed February 11, 2022. http://www.who.int/alliance-hpsr/resources/publications/rapid-review-guide/en/ .

Dobbins, Maureen. “Steps for Conducting a Rapid Review,” 2017, 25.  https://www.nccmt.ca/uploads/media/media/0001/01/a816af720e4d587e13da6bb307df8c907a5dff9a.pdf

Norris HC, Richardson HM, Benoit MC, et al. (2021) Utilization Impact of Cost-Sharing Elimination for Preventive Care Services: A Rapid Review.   https://pubmed.ncbi.nlm.nih.gov/34157906/

Marcus N, Stergiopoulos V. Re-examining mental health crisis intervention: A rapid review comparing outcomes across police, co-responder and non-police models. Health Soc Care Community. 2022 Feb 1. doi: 10.1111/hsc.13731. Epub ahead of print. PMID: 35103364.

Narrative ( Literature ) Review

A broad term referring to reviews with a wide scope and non-standardized methodology.

See Baethge 2019 below for a method to provide quality assessment,

Search strategies, comprehensiveness, and time range covered will vary and do not follow an established protocol.

It provides insight into a particular topic by critically examining sources, generally over a particular period of time.

Greenhalgh, T., Thorne, S., & Malterud, K. (2018). Time to challenge the spurious hierarchy of systematic over narrative reviews?. https://pubmed.ncbi.nlm.nih.gov/29578574/

• Baethge, C., Goldbeck-Wood, S. & Mertens, S. (2019). SANRA—a scale for the quality assessment of narrative review articles. https://doi.org/10.1186/ s41073-019-0064-8   https:// researchintegrityjournal. biomedcentral.com/articles/10. 1186/s41073-019-0064-8
• Czypionka, T., Greenhalgh, T., Bassler, D., & Bryant, M. B. (2021). Masks and Face Coverings for the Lay Public : A Narrative Update. https://pubmed.ncbi.nlm.nih.gov/33370173/
• Gardiner, F. W., Nwose, E. U., Bwititi, P. T., et al.. (2017). Services aimed at achieving desirable clinical outcomes in patients with chronic kidney disease and diabetes mellitus: A narrative review. https://pubmed.ncbi.nlm.nih.gov/29201367/
•  Dickerson, S. S., Connors, L. M., Fayad, A., & Dean, G. E. (2014). Sleep-wake disturbances in cancer patients: narrative review of literature focusing on improving quality of life outcomes.  https://pubmed.ncbi.nlm.nih.gov/25050080/

Help us improve our Library guides with this 5 minute survey . We appreciate your feedback!

• UOW Library
• Key guides for researchers

Systematic Review

• Five other types of systematic review
• What is a systematic review?
• How is a literature review different?
• Search tips for systematic reviews
• Controlled vocabularies
• Grey literature
• Transferring your search
• Documenting your results
• Support & contact

Five other types of systematic reviews

1. scoping review.

A scoping review is a preliminary assessment of the potential size and scope of available research literature. Aims to identify the nature and extent of research evidence (usually including ongoing research). 

Scoping reviews provide an understanding of the size and scope of the available literature and can inform whether a full systematic review should be undertaken. 

If you're not sure you should conduct a systematic review or a scoping review,  this article  outlines the differences between these review types and could help your decision making.  

2. Rapid review

Rapid reviews are an assessment of what is already known about a policy or practice issue by using systematic review methods to search and critically appraise existing research. 

This methodology utilises several legitimate techniques to shorten the process – careful focus of the research question, using broad or less sophisticated search strategies, conducting a review of reviews, restricting the amount of grey literature, extracting only key variables and performing more simple quality appraisals. 

Rapid reviews have an increased risk of potential bias due to their short timeframe. Documenting the methodology and highlighting its limitations is one way to mitigate bias. 

3. Narrative review

Also called a literature review.  

A narrative, or literature, review synthesises primary studies and explores this through description rather than statistics. Library support for literature review can be found in  this guide . 

4. Meta-analysis

A meta-analysis statistically combines the results of quantitative studies to provide a more precise effect on the results. This type of study examines data from multiple studies, on the same subject, to determine trends. 

Outcomes from a meta-analysis may include a more precise estimate of the effect of treatment or risk factor for disease, or other outcomes, than any individual study contributing to the combined studies being analysed.

5. Mixed methods/mixed studies

Refers to any combination of methods where one significant component is a literature review (usually systematic review). For example, a mixed methods study might include a systematic review accompanied by interviews or by a stakeholder consultation. 

Within a review context, mixed methods studies refers to a combination of review approaches. For example, combining quantitative with qualitative research or outcome with process studies. 

Further reading:

• Duke University, Types of Reviews
• Systematic review types: meet the family  (Covidence)
• Systematic reviews and other types from Temple University
• A typology of reviews: an analysis of 14 review types and associated methodologies  (Grant & Booth, 2009).
• Previous: What is a systematic review?
• Next: How is a literature review different?
• Last Updated: Oct 23, 2023 1:59 PM
• URL: https://uow.libguides.com/systematic-review

Insert research help text here

LIBRARY RESOURCES

Library homepage

Library SEARCH

A-Z Databases

STUDY SUPPORT

Learning Co-Op (academic skills support)

Referencing and citing

Digital Skills Hub

MORE UOW SERVICES

UOW homepage

Student support and wellbeing

IT Services

On the lands that we study, we walk, and we live, we acknowledge and respect the traditional custodians and cultural knowledge holders of these lands.

Copyright & disclaimer | Privacy & cookie usage

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on June 22, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, June 22). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved November 3, 2023, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

Library Services

UCL LIBRARY SERVICES

• Library skills
• Systematic reviews

What are systematic reviews?

• Types of systematic reviews
• Formulating a research question
• Identifying studies
• Searching databases
• Describing and appraising studies
• Synthesis and systematic maps
• Software for systematic reviews
• Online training and support
• Live and face to face training
• Individual support
• Further help

Systematic reviews are a type of literature review of research which require equivalent standards of rigour as primary research. They have a clear, logical rationale that is reported to the reader of the review. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews particularly in the following elements of conduct and reporting.

Systematic reviews: 

• use explicit and transparent methods
• are a piece of research following a standard set of stages
• are accountable, replicable and updateable
• involve users to ensure a review is relevant and useful.

For example, systematic reviews (like all research) should have a clear research question, and the perspective of the authors in their approach to addressing the question is described. There are clearly described methods on how each study in a review was identified, how that study was appraised for quality and relevance and how it is combined with other studies in order to address the review question. A systematic review usually involves more than one person in order to increase the objectivity and trustworthiness of the reviews methods and findings.

Research protocols for systematic reviews may be peer-reviewed and published or registered in a suitable repository to help avoid duplication of reviews and for comparisons to be made with the final review and the planned review.

• History of systematic reviews to inform policy (EPPI-Centre)
• Six reasons why it is important to be systematic (EPPI-Centre)
• Evidence Synthesis International (ESI): Position Statement Describes the issues, principles and goals in synthesising research evidence to inform policy, practice and decisions

On this page

Should all literature reviews be 'systematic reviews', different methods for systematic reviews, reporting standards for systematic reviews.

Literature reviews provide a more complete picture of research knowledge than is possible from individual pieces of research. This can be used to: clarify what is known from research, provide new perspectives, build theory, test theory, identify research gaps or inform research agendas.

A systematic review requires a considerable amount of time and resources, and is one type of literature review.

If the purpose of a review is to make justifiable evidence claims, then it should be systematic, as a systematic review uses rigorous explicit methods. The methods used can depend on the purpose of the review, and the time and resources available.

A 'non-systematic review' might use some of the same methods as systematic reviews, such as systematic approaches to identify studies or quality appraise the literature. There may be times when this approach can be useful. In a student dissertation, for example, there may not be the time to be fully systematic in a review of the literature if this is only one small part of the thesis. In other types of research, there may also be a need to obtain a quick and not necessarily thorough overview of a literature to inform some other work (including a systematic review). Another example, is where policymakers, or other people using research findings, want to make quick decisions and there is no systematic review available to help them. They have a choice of gaining a rapid overview of the research literature or not having any research evidence to help their decision-making. 

Just like any other piece of research, the methods used to undertake any literature review should be carefully planned to justify the conclusions made. 

Finding out about different types of systematic reviews and the methods used for systematic reviews, and reading both systematic and other types of review will help to understand some of the differences. 

Typically, a systematic review addresses a focussed, structured research question in order to inform understanding and decisions on an area. (see the  Formulating a research question  section for examples). 

Sometimes systematic reviews ask a broad research question, and one strategy to achieve this is the use of several focussed sub-questions each addressed by sub-components of the review.  

Another strategy is to develop a map to describe the type of research that has been undertaken in relation to a research question. Some maps even describe over 2,000 papers, while others are much smaller. One purpose of a map is to help choose a sub-set of studies to explore more fully in a synthesis. There are also other purposes of maps: see the box on  systematic evidence maps  for further information. 

Reporting standards specify minimum elements that need to go into the reporting of a review. The reporting standards refer mainly to methodological issues but they are not as detailed or specific as critical appraisal for the methodological standards of conduct of a review.

A number of organisations have developed specific guidelines and standards for both the conducting and reporting on systematic reviews in different topic areas.  

• PRISMA PRISMA is a reporting standard and is an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. The Key Documents section of the PRISMA website links to a checklist, flow diagram and explanatory notes. PRISMA is less useful for certain types of reviews, including those that are iterative.
• eMERGe eMERGe is a reporting standard that has been developed for meta-ethnographies, a qualitative synthesis method.
• ROSES: RepOrting standards for Systematic Evidence Syntheses Reporting standards, including forms and flow diagram, designed specifically for systematic reviews and maps in the field of conservation and environmental management.

Useful books about systematic reviews

Systematic approaches to a successful literature review

An introduction to systematic reviews

Cochrane handbook for systematic reviews of interventions

Systematic reviews: crd's guidance for undertaking reviews in health care.

Finding what works in health care: Standards for systematic reviews

Systematic Reviews in the Social Sciences

Meta-analysis and research synthesis.

Research Synthesis and Meta-Analysis

Doing a Systematic Review

Literature reviews.

• What is a literature review?
• Why are literature reviews important?
• << Previous: Systematic reviews
• Next: Types of systematic reviews >>
• Last Updated: Oct 19, 2023 3:15 PM
• URL: https://library-guides.ucl.ac.uk/systematic-reviews

• Collections
• Research Help
• Teaching & Learning
• Library Home

Systematic Reviews

• Getting Started
• Additional Frameworks
• More Types of Reviews
• Timeline & Resources
• Inclusion/Exclusion Criteria
• Resources & More

SR vs. Meta-Analysis

Can be separate or done together.

SR : The way in which you gather and select your data (studies) is strategic and unbiased as possible.

• Multiple databases
• Selection by title/abstract only
• Not limited by location or language

Meta : Combining different data from different studies to increase population.

• Data does not have to be selected in an unbiased or strategic way

Librarian Profile

Which Review is Right for Your Question

When people say the word "systematic review" there are two things they could mean. "Systematic Review" is an umbrella term for systematically-designed studies that review the evidence on a particular topic. A systematic review is also a type of systematically-designed study that reviews the evidence on a particular topic. Depending on your question and resources, you may actually perform a meta-analysis, scoping review, rapid review, or a mapping review.

To better determine what type of review is right for your study, take a look through the table below detailing the 7 most common types of systematic reviews. 

If none of those seem quite right for your study, there are 7 additional types of reviews . 

Major Types of Reviews

Reproduced from: Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009;26(2):91‐108. doi:10.1111/j.1471-1842.2009.00848.x PMID: 19490148

• << Previous: Additional Frameworks
• Next: More Types of Reviews >>
• Last Updated: Sep 6, 2023 10:34 AM
• URL: https://libguides.wvu.edu/SystematicReviews

Easy guide to conducting a systematic review

Affiliations.

• 1 Discipline of Child and Adolescent Health, University of Sydney, Sydney, New South Wales, Australia.
• 2 Department of Nephrology, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• 3 Education Department, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• PMID: 32364273
• DOI: 10.1111/jpc.14853

A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a predefined search strategy to identify and appraise all available published literature on a specific topic. The meticulous nature of the systematic review research methodology differentiates a systematic review from a narrative review (literature review or authoritative review). This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers.

Keywords: research; research design; systematic review.

© 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

Publication types

• Systematic Review
• Research Design*

Literature Review: Types of literature reviews

• Traditional or narrative literature reviews
• Scoping Reviews
• Systematic literature reviews
• Annotated bibliography
• Keeping up to date with literature
• Finding a thesis
• Evaluating sources and critical appraisal of literature
• Managing and analysing your literature
• Further reading and resources

Types of literature reviews

The type of literature review you write will depend on your discipline and whether you are a researcher writing your PhD, publishing a study in a journal or completing an assessment task in your undergraduate study.

A literature review for a subject in an undergraduate degree will not be as comprehensive as the literature review required for a PhD thesis.

An undergraduate literature review may be in the form of an annotated bibliography or a narrative review of a small selection of literature, for example ten relevant articles. If you are asked to write a literature review, and you are an undergraduate student, be guided by your subject coordinator or lecturer.

The common types of literature reviews will be explained in the pages of this section.

• Narrative or traditional literature reviews
• Critically Appraised Topic (CAT)
• Scoping reviews
• Annotated bibliographies

These are not the only types of reviews of literature that can be conducted. Often the term "review" and "literature" can be confusing and used in the wrong context. Grant and Booth (2009) attempt to clear up this confusion by discussing 14 review types and the associated methodology, and advantages and disadvantages associated with each review.

Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies . Health Information & Libraries Journal, 26 , 91–108. doi:10.1111/j.1471-1842.2009.00848.x

What's the difference between reviews?

Researchers, academics, and librarians all use various terms to describe different types of literature reviews, and there is often inconsistency in the ways the types are discussed. Here are a couple of simple explanations.

• The image below describes common review types in terms of speed, detail, risk of bias, and comprehensiveness:

"Schematic of the main differences between the types of literature review" by Brennan, M. L., Arlt, S. P., Belshaw, Z., Buckley, L., Corah, L., Doit, H., Fajt, V. R., Grindlay, D., Moberly, H. K., Morrow, L. D., Stavisky, J., & White, C. (2020). Critically Appraised Topics (CATs) in veterinary medicine: Applying evidence in clinical practice. Frontiers in Veterinary Science, 7 , 314. https://doi.org/10.3389/fvets.2020.00314 is licensed under CC BY 3.0

• The table below lists four of the most common types of review , as adapted from a widely used typology of fourteen types of reviews (Grant & Booth, 2009).  

Grant, M.J. & Booth, A. (2009).  A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26 (2), 91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

See also the Library's  Literature Review guide.

Critical Appraised Topic (CAT)

For information on conducting a Critically Appraised Topic or CAT

Callander, J., Anstey, A. V., Ingram, J. R., Limpens, J., Flohr, C., & Spuls, P. I. (2017).  How to write a Critically Appraised Topic: evidence to underpin routine clinical practice.  British Journal of Dermatology (1951), 177(4), 1007-1013. https://doi.org/10.1111/bjd.15873 

Books on Literature Reviews

• << Previous: Home
• Next: Traditional or narrative literature reviews >>
• Last Updated: Oct 12, 2023 3:36 PM
• URL: https://libguides.csu.edu.au/review

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

Introduction to Systematic Reviews

In this guide.

• Introduction
• Lane Research Services
• Types of Reviews
• Systematic Review Process
• Protocols & Guidelines
• Data Extraction and Screening
• Resources & Tools
• Systematic Review Online Course

Before You Start Checklist

Are you ready to carry out a knowledge synthesis project such as a systematic review, meta-analysis, or scoping review? Remember that systematic reviews require:

• a team to carry out screening, extraction, and critical appraisal methods
• a significant amount of time to complete
• enough high quality studies to make a systematic review feasible
• a rigorous protocol (that should be registered)
• adherence to transparent and rigorous methods
• a strong project management component with defined goals, responsibilities, deliverables, and timelines 
• financial resources to complete the project 

What Review Is Right For You?

If you're unsure what type of knowledge synthesis best suits your research purposes, follow along this flowchart or complete this short quiz to find your personalized review methodologies: https://whatreviewisrightforyou.knowledgetranslation.net/

Reproduced from  "What type of review could you write?" Yale Medical Library. 

Types of Knowledge Syntheses

Conducting effective reviews is essential to advance the knowledge and understand the breadth of research on a topic; synthesize existing evidence; develop theories or provide a conceptual background for subsequent research; and identify research gaps. However, there are over 100 different kinds of reviews to choose from. The following provides a comparison of common review types.

Reproduced from Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26 (2), 91-108. DOI: 10.1111/J.1471-1842.2009.00848.X

Fifty Shades of Review - Dr Andrew Booth from ScHARR Library on Youtube .

Books on Knowledge Synthesis

• Finding What Works in Health Care by Jill Eden (Editor); Laura Levit (Editor); Alfred Berg (Editor); Sally Morton (Editor); Committee on Standards for Systematic Reviews of Comparative Effectiveness Research; Institute of Medicine; Board on Health Care Services Staff ISBN: 0309164257 Publication Date: 2011

• << Previous: Lane Research Services
• Next: Systematic Review Process >>
• Last Updated: Nov 1, 2023 2:50 PM
• URL: https://laneguides.stanford.edu/systematicreviews

• Correspondence
• Open access
• Published: 10 January 2018

What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences

• Zachary Munn   ORCID: orcid.org/0000-0002-7091-5842 1 ,
• Cindy Stern 1 ,
• Edoardo Aromataris 1 ,
• Craig Lockwood 1 &
• Zoe Jordan 1  

BMC Medical Research Methodology volume  18 , Article number:  5 ( 2018 ) Cite this article

105k Accesses

2019 Citations

409 Altmetric

Metrics details

Systematic reviews have been considered as the pillar on which evidence-based healthcare rests. Systematic review methodology has evolved and been modified over the years to accommodate the range of questions that may arise in the health and medical sciences. This paper explores a concept still rarely considered by novice authors and in the literature: determining the type of systematic review to undertake based on a research question or priority.

Within the framework of the evidence-based healthcare paradigm, defining the question and type of systematic review to conduct is a pivotal first step that will guide the rest of the process and has the potential to impact on other aspects of the evidence-based healthcare cycle (evidence generation, transfer and implementation). It is something that novice reviewers (and others not familiar with the range of review types available) need to take account of but frequently overlook. Our aim is to provide a typology of review types and describe key elements that need to be addressed during question development for each type.

Conclusions

In this paper a typology is proposed of various systematic review methodologies. The review types are defined and situated with regard to establishing corresponding questions and inclusion criteria. The ultimate objective is to provide clarified guidance for both novice and experienced reviewers and a unified typology with respect to review types.

Peer Review reports

Introduction

Systematic reviews are the gold standard to search for, collate, critique and summarize the best available evidence regarding a clinical question [ 1 , 2 ]. The results of systematic reviews provide the most valid evidence base to inform the development of trustworthy clinical guidelines (and their recommendations) and clinical decision making [ 2 ]. They follow a structured research process that requires rigorous methods to ensure that the results are both reliable and meaningful to end users. Systematic reviews are therefore seen as the pillar of evidence-based healthcare [ 3 , 4 , 5 , 6 ]. However, systematic review methodology and the language used to express that methodology, has progressed significantly since their appearance in healthcare in the 1970’s and 80’s [ 7 , 8 ]. The diachronic nature of this evolution has caused, and continues to cause, great confusion for both novice and experienced researchers seeking to synthesise various forms of evidence. Indeed, it has already been argued that the current proliferation of review types is creating challenges for the terminology for describing such reviews [ 9 ]. These fundamental issues primarily relate to a) the types of questions being asked and b) the types of evidence used to answer those questions.

Traditionally, systematic reviews have been predominantly conducted to assess the effectiveness of health interventions by critically examining and summarizing the results of randomized controlled trials (RCTs) (using meta-analysis where feasible) [ 4 , 10 ]. However, health professionals are concerned with questions other than whether an intervention or therapy is effective, and this is reflected in the wide range of research approaches utilized in the health field to generate knowledge for practice. As such, Pearson and colleagues have argued for a pluralistic approach when considering what counts as evidence in health care; suggesting that not all questions can be answered from studies measuring effectiveness alone [ 4 , 11 ]. As the methods to conduct systematic reviews have evolved and advanced, so too has the thinking around the types of questions we want and need to answer in order to provide the best possible, evidence-based care [ 4 , 11 ].

Even though most systematic reviews conducted today still focus on questions relating to the effectiveness of medical interventions, many other review types which adhere to the principles and nomenclature of a systematic review have emerged to address the diverse information needs of healthcare professionals and policy makers. This increasing array of systematic review options may be confusing for the novice systematic reviewer, and in our experience as educators, peer reviewers and editors we find that many beginner reviewers struggle to achieve conceptual clarity when planning for a systematic review on an issue other than effectiveness. For example, reviewers regularly try to force their question into the PICO format (population, intervention, comparator and outcome), even though their question may be an issue of diagnostic test accuracy or prognosis; attempting to define all the elements of PICO can confound the remainder of the review process. The aim of this article is to propose a typology of systematic review types aligned to review questions to assist and guide the novice systematic reviewer and editors, peer-reviewers and policy makers. To our knowledge, this is the first classification of types of systematic reviews foci conducted in the medical and health sciences into one central typology.

Review typology

For the purpose of this typology a systematic review is defined as a robust, reproducible, structured critical synthesis of existing research. While other approaches to the synthesis of evidence exist (including but not limited to literature reviews, evidence maps, rapid reviews, integrative reviews, scoping and umbrella reviews), this paper seeks only to include approaches that subscribe to the above definition. As such, ten different types of systematic review foci are listed below and in Table  1 . In this proposed typology, we provide the key elements for formulating a question for each of the 10 review types.

Effectiveness reviews [ 12 ]

Experiential (Qualitative) reviews [ 13 ]

Costs/Economic Evaluation reviews [ 14 ]

Prevalence and/or Incidence reviews [ 15 ]

Diagnostic Test Accuracy reviews [ 16 ]

Etiology and/or Risk reviews [ 17 ]

Expert opinion/policy reviews [ 18 ]

Psychometric reviews [ 19 ]

Prognostic reviews [ 20 ]

Methodological systematic reviews [ 21 , 22 ]

Effectiveness reviews

Systematic reviews assessing the effectiveness of an intervention or therapy are by far the most common. Essentially effectiveness is the extent to which an intervention, when used appropriately, achieves the intended effect [ 11 ]. The PICO approach (see Table 1 ) to question development is well known [ 23 ] and comprehensive guidance for these types of reviews is available [ 24 ]. Characteristics regarding the population (e.g. demographic and socioeconomic factors and setting), intervention (e.g. variations in dosage/intensity, delivery mode, and frequency/duration/timing of delivery), comparator (active or passive) and outcomes (primary and secondary including benefits and harms, how outcomes will be measured including the timing of measurement) need to be carefully considered and appropriately justified.

Experiential (qualitative) reviews

Experiential (qualitative) reviews focus on analyzing human experiences and cultural and social phenomena. Reviews including qualitative evidence may focus on the engagement between the participant and the intervention, as such a qualitative review may describe an intervention, but its question focuses on the perspective of the individuals experiencing it as part of a larger phenomenon. They can be important in exploring and explaining why interventions are or are not effective from a person-centered perspective. Similarly, this type of review can explain and explore why an intervention is not adopted in spite of evidence of its effectiveness [ 4 , 13 , 25 ]. They are important in providing information on the patient’s experience, which can enable the health professional to better understand and interact with patients. The mnemonic PICo can be used to guide question development (see Table 1 ). With qualitative evidence there is no outcome or comparator to be considered. A phenomenon of interest is the experience, event or process occurring that is under study, such as response to pain or coping with breast cancer; it differs from an intervention in its focus. Context will vary depending on the objective of the review; it may include consideration of cultural factors such as geographic location, specific racial or gender based interests, and details about the setting such as acute care, primary healthcare, or the community [ 4 , 13 , 25 ]. Reviews assessing the experience of a phenomenon may opt to use a mixed methods approach and also include quantitative data, such as that from surveys. There are reporting guidelines available for qualitative reviews, including the ‘Enhancing transparency in reporting the synthesis of qualitative research’ (ENTREQ) statement [ 26 ] and the newly proposed meta-ethnography reporting guidelines (eMERGe) [ 27 ].

Costs/economic evaluation reviews

Costs/Economics reviews assess the costs of a certain intervention, process, or procedure. In any society, resources available (including dollars) have alternative uses. In order to make the best decisions about alternative courses of action evidence is needed on the health benefits and also on the types and amount of resources needed for these courses of action. Health economic evaluations are particularly useful to inform health policy decisions attempting to achieve equality in healthcare provision to all members of society and are commonly used to justify the existence and development of health services, new health technologies and also, clinical guideline development [ 14 ]. Issues of cost and resource use may be standalone reviews or components of effectiveness reviews [ 28 ]. Cost/Economic evaluations are examples of a quantitative review and as such can follow the PICO mnemonic (see Table 1 ). Consideration should be given to whether the entire world/international population is to be considered or only a population (or sub-population) of a particular country. Details of the intervention and comparator should include the nature of services/care delivered, time period of delivery, dosage/intensity, co-interventions, and personnel undertaking delivery. Consider if outcomes will only focus on resource usage and costs of the intervention and its comparator(s) or additionally on cost-effectiveness. Context (including perspective) can also be considered in these types of questions e.g. health setting(s).

Prevalence and/or incidence reviews

Essentially prevalence or incidence reviews measure disease burden (whether at a local, national or global level). Prevalence refers to the proportion of a population who have a certain disease whereas incidence relates to how often a disease occurs. These types of reviews enable governments, policy makers, health professionals and the general population to inform the development and delivery of health services and evaluate changes and trends in diseases over time [ 15 , 29 ]. Prevalence or incidence reviews are important in the description of geographical distribution of a variable and the variation between subgroups (such as gender or socioeconomic status), and for informing health care planning and resource allocation. The CoCoPop framework can be used for reviews addressing a question relevant to prevalence or incidence (see Table 1 ). Condition refers to the variable of interest and can be a health condition, disease, symptom, event of factor. Information regarding how the condition will be measured, diagnosed or confirmed should be provided. Environmental factors can have a substantial impact on the prevalence or incidence of a condition so it is important that authors define the context or specific setting relevant to their review question [ 15 , 29 ]. The population or study subjects should be clearly defined and described in detail.

Diagnostic test accuracy reviews

Systematic reviews assessing diagnostic test accuracy provide a summary of test performance and are important for clinicians and other healthcare practitioners in order to determine the accuracy of the diagnostic tests they use or are considering using [ 16 ]. Diagnostic tests are used by clinicians to identify the presence or absence of a condition in a patient for the purpose of developing an appropriate treatment plan. Often there are several tests available for diagnosis. The mnemonic PIRD is recommended for question development for these types of systematic reviews (see Table 1 ). The population is all participants who will undergo the diagnostic test while the index test(s) is the diagnostic test whose accuracy is being investigated in the review. Consider if multiple iterations of a test exist and who carries out or interprets the test, the conditions the test is conducted under and specific details regarding how the test will be conducted. The reference test is the ‘gold standard’ test to which the results of the index test will be compared. It should be the best test currently available for the diagnosis of the condition of interest. Diagnosis of interest relates to what diagnosis is being investigated in the systematic review. This may be a disease, injury, disability or any other pathological condition [ 16 ].

Etiology and/or risk reviews

Systematic reviews of etiology and risk are important for informing healthcare planning and resource allocation, and are particularly valuable for decision makers when making decisions regarding health policy and prevention of adverse health outcomes. The common objective of many of these types of reviews is to determine whether and to what degree a relationship exists between an exposure and a health outcome. Use of the PEO mnemonic is recommended (see Table 1 ). The review question should outline the exposure, disease, symptom or health condition of interest, the population or groups at risk, as well as the context/location, the time period and the length of time where relevant [ 17 ]. The exposure of interest refers to a particular risk factor or several risk factors associated with a disease/condition of interest in a population, group or cohort who have been exposed to them. It should be clearly reported what the exposure or risk factor is, and how it may be measured/identified including the dose and nature of exposure and the duration of exposure, if relevant. Important outcomes of interest relevant to the health issue and important to key stakeholders (e.g. knowledge users, consumers, policy makers, payers etc.) must be specified. Guidance now exists for conducting these types of reviews [ 17 ]. As these reviews rely heavily on observational studies, the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) [ 30 ] reporting guidelines should be referred to in addition to the PRISMA guidelines.

Expert opinion/policy reviews

Expert opinion and policy analysis systematic reviews focus on the synthesis of narrative text and/or policy. Expert opinion has a role to play in evidence-based healthcare, as it can be used to either complement empirical evidence or, in the absence of research studies, stand alone as the best available evidence. The synthesis of findings from expert opinion within the systematic review process is not well recognized in mainstream evidence-based practice. However, in the absence of research studies, the use of a transparent systematic process to identify the best available evidence drawn from text and opinion can provide practical guidance to practitioners and policy makers [ 18 ]. While a number of mnemonics have been discussed previously that can be used for opinion and text, not all elements necessarily apply to every text or opinion-based review, and use of mnemonics should be considered a guide rather than a policy. Broadly PICo can be used where I can refer to either the intervention or a phenomena of interest (see Table 1 ). Reviewers will need to describe the population, giving attention to whether specific characteristics of interest, such as age, gender, level of education or professional qualification are important to the question. As with other types of reviews, interventions may be broad areas of practice management, or specific, singular interventions. However, reviews of text or opinion may also reflect an interest in opinions around power, politics or other aspects of health care other than direct interventions, in which case, these should be described in detail. The use of a comparator and specific outcome statement is not necessarily required for a review of text and opinion based literature. In circumstances where they are considered appropriate, the nature and characteristics of the comparator and outcomes should be described [ 18 ].

Psychometric reviews

Psychometric systematic reviews (or systematic reviews of measurement properties) are conducted to assess the quality/characteristics of health measurement instruments to determine the best tool for use (in terms of its validity, reliability, responsiveness etc.) in practice for a certain condition or factor [ 31 , 32 , 33 ]. A psychometric systematic review may be undertaken on a) the measurement properties of one measurement instrument, b) the measurement properties of the most commonly utilized measurement instruments measuring a specific construct, c) the measurement properties of all available measurement instruments to measure a specific construct in a specific population or d) the measurement properties of all available measurement instruments in a specific population that does not specify the construct to be measured. The COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) group have developed guidance for conducting these types of reviews [ 19 , 31 ]. They recommend firstly defining the type of review to be conducted as well as the construct or the name(s) of the outcome measurement instrument(s) of interest, the target population, the type of measurement instrument of interest (e.g. questionnaires, imaging tests) and the measurement properties on which the review investigates (see Table 1 ).

Prognostic reviews

Prognostic research is of high value as it provides clinicians and patients with information regarding the course of a disease and potential outcomes, in addition to potentially providing useful information to deliver targeted therapy relating to specific prognostic factors [ 20 , 34 , 35 ]. Prognostic reviews are complex and methodology for these types of reviews is still under development, although a Cochrane methods group exists to support this approach [ 20 ]. Potential systematic reviewers wishing to conduct a prognostic review may be interested in determining the overall prognosis for a condition, the link between specific prognostic factors and an outcome and/or prognostic/prediction models and prognostic tests [ 20 , 34 , 35 , 36 , 37 ]. Currently there is little information available to guide the development of a well-defined review question however the Quality in Prognosis Studies (QUIPS) tool [ 34 ] and the Checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS Checklist) [ 38 ] have been developed to assist in this process (see Table 1 ).

Methodology systematic reviews

Systematic reviews can be conducted for methodological purposes [ 39 ], and examples of these reviews are available in the Cochrane Database [ 40 , 41 ] and elsewhere [ 21 ]. These reviews can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses. There is limited guidance for conducting these reviews, although there does exist an appendix in the Cochrane Handbook focusing specifically on methodological reviews [ 39 ]. They suggest following the SDMO approach where the types of studies should define all eligible study designs as well as any thresholds for inclusion (e.g. RCTS and quasi-RCTs). Types of data should detail the raw material for the methodology studies (e.g. original research submitted to biomedical journals) and the comparisons of interest should be described under types of methods (e.g. blinded peer review versus unblinded peer review) (see Table 1 ). Lastly both primary and secondary outcome measures should be listed (e.g. quality of published report) [ 39 ].

The need to establish a specific, focussed question that can be utilized to define search terms, inclusion and exclusion criteria and interpretation of data within a systematic review is an ongoing issue [ 42 ]. This paper provides an up-to-date typology for systematic reviews which reflects the current state of systematic review conduct. It is now possible that almost any question can be subjected to the process of systematic review. However, it can be daunting and difficult for the novice researcher to determine what type of review they require and how they should conceptualize and phrase their review question, inclusion criteria and the appropriate methods for analysis and synthesis [ 23 ]. Ensuring that the review question is well formed is of the utmost importance as question design has the most significant impact on the conduct of a systematic review as the subsequent inclusion criteria are drawn from the question and provide the operational framework for the review [ 23 ]. In this proposed typology, we provide the key elements for formulating a question for each of the 10 review types.

When structuring a systematic review question some of these key elements are universally agreed (such as PICO for effectiveness reviews) whilst others are more novel. For example, the use of PIRD for diagnostic reviews contrasts with other mnemonics, such as PITR [ 43 ], PPP-ICP-TR [ 44 ] or PIRATE [ 45 ]. Qualitative reviews have sometimes been guided by the mnemonic SPIDER, however this has been recommended against for guiding searching due to it not identifying papers that are relevant [ 46 ]. Variations on our guidance exist, with the additional question elements of ‘time’ (PICOT) and study types (PICOS) also existing. Reviewers are advised to consider these elements when crafting their question to determine if they are relevant for their topic. We believe that based on the guidance included in this typology, constructing a well-built question for a systematic review is a skill that can be mastered even for the novice reviewer.

Related to this discussion of a typology for systematic reviews is the issue of how to distinguish a systematic review from a literature review. When searching the literature, you may come across papers referred to as ‘systematic reviews,’ however, in reality they do not necessarily fit this description [ 21 ]. This is of significant concern given the common acceptance of systematic reviews as ‘level 1’ evidence and the best study design to inform practice. However, many of these reviews are simply literature reviews masquerading as the ideal product. It is therefore important to have a critical eye when assessing publications identified as systematic reviews. Today, the methodology of systematic reviews continues to evolve. However, there is general acceptance of certain steps being required in a systematic review of any evidence type [ 2 ] and these should be used to distinguish between a literature review and a systematic review. The following can be viewed as the defining features of a systematic review and its conduct [ 1 , 2 ]:

Clearly articulated objectives and questions to be addressed

Inclusion and exclusion criteria, stipulated a priori (in a protocol), that determine the eligibility of studies

A comprehensive search to identify all relevant studies, both published and unpublished

A process of study screening and selection

Appraisal of the quality of included studies/ papers (risk of bias) and assessment of the validity of their results/findings/ conclusions

Analysis of data extracted from the included research

Presentation and synthesis of the results/ findings extracted

Interpret the results, potentially establishing the certainty of the results and making and implications for practice and research

Transparent reporting of the methodology and methods used to conduct the review

Prior to deciding what type of review to conduct, the reviewer should be clear that a systematic review is the best approach. A systematic review may be undertaken to confirm whether current practice is based on evidence (or not) and to address any uncertainty or variation in practice that may be occurring. Conducting a systematic review also identifies where evidence is not available and can help categorize future research in the area. Most importantly, they are used to produce statements to guide decision-making. Indications for systematic reviews:

uncover the international evidence

confirm current practice/ address any variation

identify areas for future research

investigate conflicting results

produce statements to guide decision-making

The popularity of systematic reviews has resulted in the creation of various evidence review processes over the last 30 years. These include integrative reviews, scoping reviews [ 47 ], evidence maps [ 48 ], realist syntheses [ 49 ], rapid reviews [ 50 ], umbrella reviews (systematic reviews of reviews) [ 51 ], mixed methods reviews [ 52 ], concept analyses [ 53 ] and others. Useful typologies of these diverse review types can be used as reference for researchers, policy makers and funders when discussing a review approach [ 54 , 55 ]. It was not the purpose of this article to describe and define each of these diverse evidence synthesis methods as our focus was purely on systematic review questions. Depending on the researcher, their question/s and their resources at hand, one of these approaches may be the best fit for answering a particular question.

Gough and colleagues [ 9 ] provided clarification between different review designs and methods but stopped short of providing a taxonomy of review types. The rationale for this was that in the field of evidence synthesis ‘the rate of development of new approaches to reviewing is too fast and the overlap of approaches too great for that to be helpful.’ [ 9 ] They instead provide a useful description of how reviews may differ and more importantly why this may be the case. It is also our view that evidence synthesis methodology is a rapidly developing field, and that even within the review types classified here (such as effectiveness [ 56 ] or experiential [qualitative [ 57 ]]) there may be many different subsets and complexities that need to be addressed. Essentially, the classifications listed above may be just the initial level of a much larger family tree. We believe that this typology will provide a useful contribution to efforts to sort and classify evidence review approaches and understand the need for this to be updated over time. A useful next step might be the development of a comprehensive taxonomy to further guide reviewers in making a determination about the most appropriate evidence synthesis product to undertake for a particular purpose or question.

Systematic reviews of animal studies (or preclinical systematic reviews) have not been common practice in the past (when comparing to clinical research) although this is changing [ 58 , 59 , 60 , 61 ]. Systematic reviews of these types of studies can be useful to inform the design of future experiments (both preclinical and clinical) [ 59 ] and address an important gap in translation science [ 5 , 60 ]. Guidance for these types of reviews is now emerging [ 58 , 60 , 62 , 63 , 64 ]. These review types, which are often hypothesis generating, were excluded from our typology as they are only very rarely used to answer a clinical question.

Systematic reviews are clearly an indispensable component in the chain of scientific enquiry in a much broader sense than simply to inform policy and practice and therefore ensuring that they are designed in a rigorous manner, addressing appropriate questions driven by clinical and policy needs is essential. With the ever-increasing global investment in health research it is imperative that the needs of health service providers and end users are met. It has been suggested that one way to ensure this occurs is to precede any research investment with a systematic review of existing research [ 65 ]. However, the only way that such a strategy would be effective would be if all reviews conducted are done so with due rigour.

It has been argued recently that there is mass production of reviews that are often unnecessary, misleading and conflicted with most having weak or insufficient evidence to inform decision making [ 66 ]. Indeed, asking has been identified as a core functional competency associated with obtaining and applying the best available evidence [ 67 ]. Fundamental to the tenets of evidence-based healthcare and, in particular evidence implementation, is the ability to formulate a question that is amenable to obtaining evidence and “structured thinking” around question development is critical to its success [ 67 ]. The application of evidence can be significantly hampered when existing evidence does not correspond to the situations that practitioners (or guideline developers) are faced with. Hence, determination of appropriate review types that respond to relevant clinical and policy questions is essential.

The revised JBI Model of Evidence-Based Healthcare clarifies the conceptual integration of evidence generation, synthesis, transfer and implementation, “linking how these occur with the necessarily challenging dynamics that contribute to whether translation of evidence into policy and practice is successful” [ 68 ]. Fundamental to this approach is the recognition that the process of evidence-based healthcare is not prescriptive or linear, but bi-directional, with each component having the potential to affect what occurs on either side of it. Thus, a systematic review can impact upon the types of primary research that are generated as a result of recommendations produced in the review (evidence generation) but also on the success of their uptake in policy and practice (evidence implementation). It is therefore critical for those undertaking systematic reviews to have a solid understanding of the type of review required to respond to their question.

For novice reviewers, or those unfamiliar with the broad range of review types now available, access to a typology to inform their question development is timely. The typology described above provides a framework that indicates the antecedents and determinants of undertaking a systematic review. There are several factors that may lead an author to conduct a review and these may or may not start with a clearly articulated clinical or policy question. Having a better understanding of the review types available and the questions that these reviews types lend themselves to answering is critical to the success or otherwise of a review. Given the significant resource required to undertake a review this first step is critical as it will impact upon what occurs in both evidence generation and evidence implementation. Thus, enabling novice and experienced reviewers to ensure that they are undertaking the “right” review to respond to a clinical or policy question appropriately has strategic implications from a broader evidence-based healthcare perspective.

Systematic reviews are the ideal method to rigorously collate, examine and synthesize a body of literature. Systematic review methods now exist for most questions that may arise in healthcare. This article provides a typology for systematic reviewers when deciding on their approach in addition to guidance on structuring their review question. This proposed typology provides the first known attempt to sort and classify systematic review types and their question development frameworks and therefore it can be a useful tool for researchers, policy makers and funders when deciding on an appropriate approach.

Abbreviations

CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies

Condition, Context, Population

COnsensus-based Standards for the selection of health Measurement Instruments

• Evidence-based healthcare

Meta-ethnography reporting guidelines

Enhancing transparency in reporting the synthesis of qualitative research

Joanna Briggs Institute

Meta-analysis Of Observational Studies in Epidemiology

Population, Exposure, Outcome

Population, Prognostic Factors (or models of interest), Outcome

Population, Intervention, Comparator, Outcome

Population, Phenomena of Interest, Context

Population, Intervention, Comparator/s, Outcomes, Context

Population, Index Test, Reference Test, Diagnosis of Interest

Quality in Prognosis Studies

Randomised controlled trial

Studies, Data, Methods, Outcomes

Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ (Clinical research ed). 2009;339:b2700.

Article   Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. AJN. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Munn Z, Porritt K, Lockwood C, Aromataris E, Pearson A. Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol. 2014;14:108.

Article   PubMed   PubMed Central   Google Scholar  

Pearson A. Balancing the evidence: incorporating the synthesis of qualitative data into systematic reviews. JBI Reports. 2004;2:45–64.

Pearson A, Jordan Z, Munn Z. Translational science and evidence-based healthcare: a clarification and reconceptualization of how knowledge is generated and used in healthcare. Nursing research and practice. 2012;2012:792519.

Steinberg E, Greenfield S, Mancher M, Wolman DM, Graham R. Clinical practice guidelines we can trust: National Academies Press 2011.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Chalmers I, Hedges LV, Cooper HA. Brief history of research synthesis. Eval Health Prof. 2002;25(1):12–37.

Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Systematic Reviews. 2012;1:28.

Munn Z, Tufanaru C, Aromataris EJBI. S systematic reviews: data extraction and synthesis. Am J Nurs. 2014;114(7):49–54.

Pearson A, Wiechula R, Court A, Lockwood C. The JBI model of evidence-based healthcare. International Journal of Evidence-Based Healthcare. 2005;3(8):207–15.

PubMed   Google Scholar  

Tufanaru C, Munn Z, Stephenson M, Aromataris E. Fixed or random effects meta-analysis? Common methodological issues in systematic reviews of effectiveness. Int J Evid Based Healthc. 2015;13(3):196–207.

Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Healthc. 2015;13(3):179–87.

Gomersall JS, Jadotte YT, Xue Y, Lockwood S, Riddle D, Preda A. Conducting systematic reviews of economic evaluations. Int J Evid Based Healthc. 2015;13(3):170–8.

Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc. 2015;13(3):147–53.

Campbell JM, Klugar M, Ding S, et al. Diagnostic test accuracy: methods for systematic review and meta-analysis. Int J Evid Based Healthc. 2015;13(3):154–62.

Moola S, Munn Z, Sears K, et al. Conducting systematic reviews of association (etiology): the Joanna Briggs Institute's approach. Int J Evid Based Healthc. 2015;13(3):163–9.

McArthur A, Klugarova J, Yan H, Florescu S. Innovations in the systematic review of text and opinion. Int J Evid Based Healthc. 2015;13(3):188–95.

Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.

Dretzke J, Ensor J, Bayliss S, et al. Methodological issues and recommendations for systematic reviews of prognostic studies: an example from cardiovascular disease. Systematic reviews. 2014;3(1):1.

Campbell JM, Kavanagh S, Kurmis R, Munn Z. Systematic Reviews in Burns Care: Poor Quality and Getting Worse. Journal of Burn Care & Research. 9000;Publish Ahead of Print.

France EF, Ring N, Thomas R, Noyes J, Maxwell M, Jepson RA. Methodological systematic review of what’s wrong with meta-ethnography reporting. BMC Med Res Methodol. 2014;14(1):1.

Stern C, Jordan Z, McArthur A. Developing the review question and inclusion criteria. Am J Nurs. 2014;114(4):53–6.

Higgins J, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. ed: The Cochrane Collaboration 2011.

Hannes K, Lockwood C, Pearson AA. Comparative analysis of three online appraisal instruments' ability to assess validity in qualitative research. Qual Health Res. 2010;20(12):1736–43.

Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12:181.

France EF, Ring N, Noyes J, et al. Protocol-developing meta-ethnography reporting guidelines (eMERGe). BMC Med Res Methodol. 2015;15:103.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Shemilt I, Mugford M, Byford S, et al. In: JPT H, Green S, editors. Chapter 15: incorporating economics evidence. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration: In; 2011.

Google Scholar  

Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. Int J Health Policy Manag. 2014;3(3):123–8.

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA. 2000;283(15):2008–12.

Article   CAS   PubMed   Google Scholar  

COSMIN: COnsensus-based Standards for the selection of health Measurement INstruments. Systematic reviews of measurement properties. [cited 8th December 2016]; Available from: http://www.cosmin.nl/Systematic%20reviews%20of%20measurement%20properties.html

Terwee CB, HCWd V, CAC P, Mokkink LB. Protocol for systematic reviews of measurement properties. COSMIN: Knowledgecenter Measurement Instruments; 2011.

Mokkink LB, Terwee CB, Stratford PW, et al. Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Qual Life Res. 2009;18(3):313–33.

Hayden JA, van der Windt DA, Cartwright JL, CÃ P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6.

The Cochrane Collaboration. Cochrane Methods Prognosis. 2016 [cited 7th December 2016]; Available from: http://methods.cochrane.org/prognosis/scope-our-work .

Rector TS, Taylor BC, Wilt TJ. Chapter 12: systematic review of prognostic tests. J Gen Intern Med. 2012;27(Suppl 1):S94–101.

Peters S, Johnston V, Hines S, Ross M, Coppieters M. Prognostic factors for return-to-work following surgery for carpal tunnel syndrome: a systematic review. JBI Database of Systematic Reviews and Implementation Reports. 2016;14(9):135–216.

Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744.

Clarke M, Oxman AD, Paulsen E, Higgins JP, Green S, Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. In: Higgins JP, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 ed: The Cochrane Collaboration 2011.

Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007;2:MR000016.

Djulbegovic B, Kumar A, Glasziou PP, et al. New treatments compared to established treatments in randomized trials. Cochrane Database Syst Rev. 2012;10:MR000024.

PubMed   PubMed Central   Google Scholar  

Thoma A, Eaves FF 3rd. What is wrong with systematic reviews and meta-analyses: if you want the right answer, ask the right question! Aesthet Surg J. 2016;36(10):1198–201.

Deeks JJ, Wisniewski S, Davenport C. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Chapter 4: guide to the contents of a Cochrane diagnostic test accuracy protocol. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy The Cochrane Collaboration: In; 2013.

Bae J-M. An overview of systematic reviews of diagnostic tests accuracy. Epidemiology and Health. 2014;36:e2014016.

White S, Schultz T. Enuameh YAK. Lippincott Wiliams & Wilkins: Synthesizing evidence of diagnostic accuracy; 2011.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi SPICO. PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. International journal of evidence-based healthcare. 2015;13(3):141–6.

Hetrick SE, Parker AG, Callahan P, Purcell R. Evidence mapping: illustrating an emerging methodology to improve evidence-based practice in youth mental health. J Eval Clin Pract. 2010;16(6):1025–30.

Wong G, Greenhalgh T, Westhorp G, Pawson R. Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses - Evolving Standards) project. Southampton UK: Queen's Printer and Controller of HMSO 2014. This work was produced by Wong et al. under the terms of a commissioning contract issued by the secretary of state for health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR journals library, National Institute for Health Research, evaluation, trials and studies coordinating Centre, alpha house, University of Southampton Science Park, Southampton SO16 7NS, UK. 2014.

Munn Z, Lockwood C, Moola S. The development and use of evidence summaries for point of care information systems: a streamlined rapid review approach. Worldviews Evid-Based Nurs. 2015;12(3):131–8.

Aromataris E, Fernandez R, Godfrey CM, Holly C, Khalil H, Tungpunkom P. Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach. Int J Evid Based Healthc. 2015;13(3):132–40.

Pearson A, White H, Bath-Hextall F, Salmond S, Apostolo J, Kirkpatrick PA. Mixed-methods approach to systematic reviews. Int J Evid Based Healthc. 2015;13(3):121–31.

Draper PA. Critique of concept analysis. J Adv Nurs. 2014;70(6):1207–8.

Grant MJ, Booth A. A Typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Tricco AC, Tetzlaff J, Moher D. The art and science of knowledge synthesis. J Clin Epidemiol. 2011;64(1):11–20.

Bender R. A practical taxonomy proposal for systematic reviews of therapeutic interventions. 21st Cochrane Colloquium Quebec, Canada 2013.

Kastner M, Tricco AC, Soobiah C, et al. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12:114.

Leenaars M, Hooijmans CR, van Veggel N, et al. A step-by-step guide to systematically identify all relevant animal studies. Lab Anim. 2012;46(1):24–31.

de Vries RB, Wever KE, Avey MT, Stephens ML, Sena ES, Leenaars M. The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR J. 2014;55(3):427–37.

Hooijmans CR, Ritskes-Hoitinga M. Progress in using systematic reviews of animal studies to improve translational research. PLoS Med. 2013;10(7):e1001482.

Mignini LE, Khan KS. Methodological quality of systematic reviews of animal studies: a survey of reviews of basic research. BMC Med Res Methodol. 2006;6:10.

van Luijk J, Bakker B, Rovers MM, Ritskes-Hoitinga M, de Vries RB, Leenaars M. Systematic reviews of animal studies; missing link in translational research? PLoS One. 2014;9(3):e89981.

Vesterinen HM, Sena ES, Egan KJ, et al. Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods. 2014;221:92–102.

CAMARADES. Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies. 2014 [cited 8th December 2016]; Available from: http://www.dcn.ed.ac.uk/camarades/default.htm#about

Moher D, Glasziou P, Chalmers I, et al. Increasing value and reducing waste in biomedical research: who's listening? Lancet. 2016;387(10027):1573–86.

Ioannidis J. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly. 2016;94(3):485–514.

Rousseau DM, Gunia BC. Evidence-based practice: the psychology of EBP implementation. Annu Rev Psychol. 2016;67:667–92.

Jordan Z, Lockwood C, Aromataris E. Munn Z. The Joanna Briggs Institute: The updated JBI model for evidence-based healthcare; 2016.

Cooney GM, Dwan K, Greig CA, et al. Exercise for depression. Cochrane Database Syst Rev. 2013;9:CD004366.

Munn Z, Jordan Z. The patient experience of high technology medical imaging: a systematic review of the qualitative evidence. JBI Libr. Syst Rev. 2011;9(19):631–78.

de Verteuil R, Tan WS. Self-monitoring of blood glucose in type 2 diabetes mellitus: systematic review of economic evidence. JBI Libr. Syst Rev. 2010;8(7):302–42.

Munn Z, Moola S, Lisy K, Riitano D, Murphy F. Claustrophobia in magnetic resonance imaging: a systematic review and meta-analysis. Radiography. 2015;21(2):e59–63.

Hakonsen SJ, Pedersen PU, Bath-Hextall F, Kirkpatrick P. Diagnostic test accuracy of nutritional tools used to identify undernutrition in patients with colorectal cancer: a systematic review. JBI Database System Rev Implement Rep. 2015;13(4):141–87.

Australia C. Risk factors for lung cancer: a systematic review. NSW: Surry Hills; 2014.

McArthur A, Lockwood C. Maternal mortality in Cambodia, Thailand, Malaysia and Sri Lanka: a systematic review of local and national policy and practice initiatives. JBI Libr Syst Rev. 2010;8(16 Suppl):1–10.

Peek K. Muscle strength in adults with spinal cord injury: a systematic review of manual muscle testing, isokinetic and hand held dynamometry clinimetrics. JBI Database of Systematic Reviews and Implementation Reports. 2014;12(5):349–429.

Hayden JA, Tougas ME, Riley R, Iles R, Pincus T. Individual recovery expectations and prognosis of outcomes in non-specific low back pain: prognostic factor exemplar review. Cochrane Libr. 2014. http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD011284/full .

Download references

Acknowledgements

No funding was provided for this paper.

Availability of data and materials

Not applicable

Author information

Authors and affiliations.

The Joanna Briggs Institute, The University of Adelaide, 55 King William Road, North Adelaide, Soueth Australia, 5005, Australia

Zachary Munn, Cindy Stern, Edoardo Aromataris, Craig Lockwood & Zoe Jordan

You can also search for this author in PubMed   Google Scholar

Contributions

ZM: Led the development of this paper and conceptualised the idea for a systematic review typology. Provided final approval for submission. CS: Contributed conceptually to the paper and wrote sections of the paper. Provided final approval for submission. EA: Contributed conceptually to the paper and reviewed and provided feedback on all drafts. Provided final approval for submission. CL: Contributed conceptually to the paper and reviewed and provided feedback on all drafts. Provided final approval for submission. ZJ: Contributed conceptually to the paper and reviewed and provided feedback on all drafts. Provided approval and encouragement for the work to proceed. Provided final approval for submission.

Corresponding author

Correspondence to Zachary Munn .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

All the authors are members of the Joanna Briggs Institute, an evidence-based healthcare research institute which provides formal guidance regarding evidence synthesis, transfer and implementation.

The authors have no other competing interests to declare.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Cite this article.

Munn, Z., Stern, C., Aromataris, E. et al. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol 18 , 5 (2018). https://doi.org/10.1186/s12874-017-0468-4

Download citation

Received : 29 May 2017

Accepted : 28 December 2017

Published : 10 January 2018

DOI : https://doi.org/10.1186/s12874-017-0468-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic reviews
• Question development

BMC Medical Research Methodology

ISSN: 1471-2288

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• UNC Libraries
• HSL Academic Process
• Systematic Reviews
• Step 1: Complete Pre-Review Tasks

Systematic Reviews: Step 1: Complete Pre-Review Tasks

Created by health science librarians.

Watch Systematic Review Workshop videos

Recruit team members, choose review tools, develop and refine research question, specify eligibility criteria.

• Pre-Review FAQs
• Step 2: Develop a Protocol
• Step 3: Conduct Literature Searches
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

  Chat with us (during business hours)

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

Search the FAQs

About step 1: complete pre-review tasks.

This step will help you prepare to conduct your systematic review. You will:

• Develop your research question.
• Look at literature to decide if you need to do a systematic review.
• Build your research team.
• Decide which citation manager and systematic review software you will use.

This page has information about research questions and systematic review teams. Librarians can help you edit your research question based on the literature.

Click an item below to see how it applies to Step 1: Pre-Review Tasks.

Reporting your review with PRISMA

The PRISMA (Preferred Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines detail what you should report about your systematic review methods.

• Checklist - your guideline for essential reporting in the final manuscript.
• Flow Diagram - your visualization of the exclusion/inclusion process.
• Statement - the standards of systematic review reporting.
• Explanation and Elaboration - additional details and examples of items.

For this step of the process, you can review the PRISMA checklist and flow diagram and download the PRISMA flow diagram template that matches your review type and sources, then list out the databases and other sources you plan to search.

Managing your review with Covidence

Covidence is the tool we use at UNC to help manage the systematic review screening process. For this step, you can create a Covidence account , set up a review, add reviewers, list inclusion and exclusion criteria, and edit other review settings.

How a librarian can help with Step 1

Before you begin conducting a systematic review, a librarian can help you :

• Develop and refine your research question framework 
• Determine if any prior reviews have been published on the same or similar topics
• Determine how much literature might be available on your topic

The Introduction to Conducting a Systematic Review workshop, offered in October 2020, covered recommended standards, methods, and tools for completing a systematized, scoping, or systematic review at UNC. This workshop recording is available as a series of short videos on the process of conducting a review. It is recommended for those who have not yet conducted such a review, but are planning to do so. 

• Workshop - Part 1 (General Introduction to Systematic Review Methodology)
• Workshop - Part 2 (Developing Research Questions)
• Workshop - Part 3 (Developing a Review Protocol and Creating Your Team)
• Workshop - Part 4 (Overview of Literature Searching for Systematic Reviews)
• Workshop - Part 5 (Screening)
• Workshop - Part 6 (Overview of Quality Assessment, Data Extraction, and Writing the Review)
• SR Workshop slides
• Systematic Review Summer Workshop Series 2023 Slides

There are also a number of free systematic review methods courses you can take online.

• Systematic Reviews and Meta-Analysis — Open & Free (Open Learning Initiative) The course follows guidelines and standards developed by the Campbell Collaboration, based on empirical evidence about how to produce the most comprehensive and accurate reviews of research
• Introduction to Systematic Review and Meta-Analysis (Coursera) We will introduce methods to perform systematic reviews and meta-analysis of clinical trials. We will cover how to formulate an answerable research question, define inclusion and exclusion criteria, search for the evidence, extract data, assess the risk of bias in clinical trials, and perform a meta-analysis.

• Odd number simplifies tie-breaking process
• Depending on the size of the literature, you may want to add additional team members. A team of up to ten or twelve people is not unusual for a large systematic review.
• Conscientiously create a diverse team to ensure different voices and perspectives are represented.
• Collaborate with a librarian to develop a search strategy.
• Work with a statistician if conducting a meta-analysis.
• Define roles and expectations early in the review process.

Citation manager software

Citation managers are recommended to collect citations, remove duplicates, and manage your systematic review citations.  UNC offers Sciwheel for free and Endnote Desktop at a discount. Endnote Basic and Zotero are free for anyone to use.

While citation managers are not required to complete a systematic review, we highly recommend using one, as they can assist with organizing citations and screening levels, deduplication, and finding PDFs of articles for full text screening.

Click to view the citation manager comparison table

Systematic review software.

There are many tools that can be useful for organizing the screening process including Covidence, Rayyan, Abstrackr, and HAWC.   

UNC currently has an institutional subscription to Covidence making it available for free to UNC-affiliated users. HSL can provide classes and support for Covidence. To learn more visit the Covidence LibGuide .

HSL does not currently offer in-house support for screening tools other than Covidence.

Click to view the systematic review software comparison table

If the tools above don't meet your needs, you can also try this Excel tool called the VonVille Method.   

Information about screening tools and features: Van der Mierden, S., Tsaioun, K., Bleich, A., & Leenaars, C. H. C. (2019). Software tools for literature screening in systematic reviews in biomedical research. ALTEX : Alternativen zu Tierexperimenten, 36 (3), 508-517.

Screening tool feature definitions

Quality Assessment: The tool supports risk of bias and quality assessment. 

Data Extraction: The tool supports data extraction. 

Allows Multiple User Support: It is possible for two or more users to work on the same project at the same time, without seeing how others have voted.

Reference Allocation: It is possible to assign references to reviewers. 

Discrepancy Resolving: There is official support for resolving conflict between reviewers. 

Show Project Progress: the tool can display the progress of each reviewer and the overall project.

Attaching Comments: It is possible to add comments to a reference while screening.

Attaching PDFs: It is possible to upload PDFs for full-text screening. 

Keyword Highlighting: It is possible to display highlighted keywords during screening.

Deduplication: The tool will identify and remove duplicate citations. 

PRISMA Diagram: The tool can automatically generate a PRISMA flow diagram. 

Import Multiple File Types: It is possible to import formatted references and the tool supports multiple file types. 

Interrater Reliability: The tool can calculate and display interrater reliability.

Mobile Friendly Version: It is possible to screen on a mobile device. 

Export Results File Types: References can be exported from the screening tool into the listed file types. 

Systematic reviews aim to answer a specific research question. There are frameworks to help in question development and identification of search terms. PICO is the most popular framework utilized for clinical research topics.

The PICO question framework is useful for quantitative research topics. PICO questions identify four concepts: population, intervention, comparison, and outcome.

Research question : In infants diagnosed with necrotizing enterocolitis (NEC), what is the effect of early enteral re-feeding on NEC recurrence compared to late enteral re-feeding?

Did you know?

Did you know there are at least 25 other question frameworks besides variations of PICO?  Frameworks like PEO, SPIDER, SPICE, ECLIPSE, and others  can help you formulate a focused research question. The table and example below were created by the  Medical University of South Carolina (MUSC) Libraries .

Click on a framework below to learn more and see an example of its use.

The PEO question framework is useful for qualitative research topics. PEO questions identify three concepts: population, exposure, and outcome.

Research question : What are the daily living experiences of mothers with postnatal depression?

The SPIDER question framework is useful for qualitative or mixed methods research topics focused on "samples" rather than populations.

SPIDER questions identify five concepts: sample, phenomenon of interest, design, evaluation, and research type.

Research question : What are the experiences of young parents in attendance at antenatal education classes?

The SPICE question framework is useful for qualitative research topics evaluating the outcomes of a service, project, or intervention. SPICE questions identify five concepts: setting, perspective, intervention/exposure/interest, comparison, and evaluation.

Research question : For teenagers in South Carolina, what is the effect of provision of Quit Kits to support smoking cessation on number of successful attempts to give up smoking compared to no support ("cold turkey")?

The ECLIPSE framework is useful for qualitative research topics investigating the outcomes of a policy or service. ECLIPSE questions identify six concepts: expectation, client group, location, impact, professionals, and service.

Research question : How can I increase access to wireless internet for hospital patients?

In order to reduce bias, eligibility criteria (also known as inclusion and exclusion criteria) refer to what you plan to include and exclude from your systematic review. These criteria are decided after the research question is developed but before searches are completed. Below are examples of criteria that may be used to determine inclusion or exclusion.

• << Previous: Home
• Next: Step 2: Develop a Protocol >>
• Last Updated: Oct 17, 2023 9:42 AM
• URL: https://guides.lib.unc.edu/systematic-reviews

Search & Find

• E-Research by Discipline
• More Search & Find

Places & Spaces

• Places to Study
• Book a Study Room
• Printers, Scanners, & Computers
• More Places & Spaces
• Borrowing & Circulation
• Request a Title for Purchase
• Schedule Instruction Session
• More Services

Support & Guides

• Course Reserves
• Research Guides
• Citing & Writing
• More Support & Guides
• Mission Statement
• Diversity Statement
• Staff Directory
• Job Opportunities
• Give to the Libraries
• News & Exhibits
• Reckoning Initiative
• More About Us

• Search This Site
• Privacy Policy
• Accessibility
• Give Us Your Feedback
• 208 Raleigh Street CB #3916
• Chapel Hill, NC 27515-8890
• 919-962-1053

• Chester Fritz Library
• Library of the Health Sciences
• Thormodsgard Law Library

Literature Reviews

• Get started

Literature Reviews within a Scholarly Work

Literature reviews as a scholarly work.

• Finding Literature Reviews
• Your Literature Search
• Library Books
• How to Videos
• Communicating & Citing Research
• Bibliography

Literature reviews summarize and analyze what has been written on a particular topic and identify gaps or disagreements in the scholarly work on that topic.

Within a scholarly work, the literature review situates the current work within the larger scholarly conversation and emphasizes how that particular scholarly work contributes to the conversation on the topic. The literature review portion may be as brief as a few paragraphs focusing on a narrow topic area.

When writing this type of literature review, it's helpful to start by identifying sources most relevant to your research question. A citation tracking database such as Web of Science can also help you locate seminal articles on a topic and find out who has more recently cited them. See "Your Literature Search" for more details.

A literature review may itself be a scholarly publication and provide an analysis of what has been written on a particular topic without contributing original research. These types of literature reviews can serve to help keep people updated on a field as well as helping scholars choose a research topic to fill gaps in the knowledge on that topic. Common types include:

Systematic Review

Systematic literature reviews follow specific procedures in some ways similar to setting up an experiment to ensure that future scholars can replicate the same steps. They are also helpful for evaluating data published over multiple studies. Thus, these are common in the medical field and may be used by healthcare providers to help guide diagnosis and treatment decisions. Cochrane Reviews are one example of this type of literature review.

Semi-Systematic Review

When a systematic review is not feasible, a semi-systematic review can help synthesize research on a topic or how a topic has been studied in different fields (Snyder 2019). Rather than focusing on quantitative data, this review type identifies themes, theoretical perspectives, and other qualitative information related to the topic. These types of reviews can be particularly helpful for a historical topic overview, for developing a theoretical model, and for creating a research agenda for a field (Snyder 2019). As with systematic reviews, a search strategy must be developed before conducting the review.

Integrative Review

An integrative review is less systematic and can be helpful for developing a theoretical model or to reconceptualize a topic. As Synder (2019) notes, " This type of review often re quires a more creative collection of data, as the purpose is usually not to cover all articles ever published on the topic but rather to combine perspectives and insights from di ff erent fi elds or research traditions" (p. 336).

Source: Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research. 104. 333-339. doi: 10.1016/j.jbusres.2019.07.039

• << Previous: Get started
• Next: Finding Literature Reviews >>
• Last Updated: Oct 27, 2023 12:59 PM
• URL: https://libguides.und.edu/literature-reviews

Literature Reviews

Overview of literature reviews.

• Systematic Reviews
• Systematic Reviews for Social Sciences
• Checklists, Guides

What Type of Review is Right for You?

• Decision tree for review types from Cornell University Library

'Literature review' is a generic term that is often used to describe a range of different review types. For a class assignment, you may be required to review academic literature based on a topic of interest and write about the sources you selected. Or, if your are working on a research project, you may need to conduct a comprehensive search of the literature to write a literature review or a literature review chapter for a thesis or dissertation.

Listed below are common review types with brief descriptions for a quick comparison of characteristics. Citations are included for follow up and more details.

• Traditional Review (Integrative/Narrative) Grant & Booth (2009) describe 14 review types. They note the aim of this type of literature review is to examine the current/recent literature, so it may not include comprehensive searches and often it describes only a group of selected sources. 

Grant, M. J., & Booth, A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26 (2), 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x

• Systematic Review  Aims to be comprehensive, adheres to transparent procedures, and provides evidence synthesis that can be used in intervention decisions and policy making.
• Systematized Review   Incorporates some systematic review procedures that can be included as part of a narrative (traditional) review.
• Meta-Analysis   Uses statistical methods to evaluate relevant research studies and may be part of a systematic review.
• Rapid Review   Applies systematic review methods but sets a time limit on locating and appraising sources for a shorten timeframe.
• Scoping Review   Explores research questions to map key concepts, evidence, and gaps in the literature and may take longer to complete than a systematic review,
• Umbrella Review  
• Compiles evidence from multiple reviews based on a broad problem for which there are competing interventions.
• Next: Systematic Reviews >>
• Last Updated: Nov 1, 2023 3:10 PM
• URL: https://guides.ucf.edu/literaturereviews

Systematic Reviews

• The Research Question
• Full Service
• Basic Service
• Inclusion and Exclusion Criteria
• Translating
• Deduplication
• Project Management Tools
• Useful Resources
• Other Review Types

Additional resources on other review types

• Systematic literature reviews

Literature Reviews

Search methods for different literature review types.

All searching should be conducted thoroughly. However, some research questions do not require systematic review methodology. Some research questions are better suited to different review methodologies based on the realistic landscape of how much and what quality of literature is published on a topic at a given time. Additionally, some researchers may have limited time requirements that makes the planning and conduction of a systematic review impractical. In these instances, researchers may want to consider alternatives such as rapid reviews or  systematized literature reviews . 

Before beginning any review type, researchers should do sufficient preliminary searching to determine what review type is suitable for the research topic/question.

Typology of Reviews

There are other types of reviews, and some are often mistaken for systematic reviews. Some may even call themselves 'systematic reviews.' However, understanding the scope of other reviews and methods can help one distinguish between them and a systematic review proper. Here are some common review types:

• Technique that statistically combines the results of quantitative studies to provide a more precise effect of the results. May be a component of a systematic review.
• Generic term: published materials that provide examination of recent or current literature. Can cover a wide range of subjects at various levels of completeness and comprehensiveness. May include research findings.
• Preliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence (usually including ongoing research).
• Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research.
• Specifically refers to reviews compiling evidence from multiple reviews into one accessible and usable document. Focuses on broad condition or problem for which there are competing interventions and highlights reviews that address these interventions and their results.
• Attempts to include elements of the systematic review process while stopping short of a systematic review. Typically conducted as a postgraduate student assignment.

The above definitions are taken from  A typology of reviews: an analysis of 14 review types and associated methodologies. The document is listed below. 

• A typology of reviews: an analysis of 14 review types and associated methodologies

Meta-Analysis

Meta-analysis is the use of statistical methods to summarise the results of independent studies. By combining information from all relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. Meta-analyses also facilitate investigations of the consistency of evidence across studies, and the exploration of differences across studies ( Cochrane Handbook, 1.2.2 ). More information on meta-analyses can be found in   Cochrane Handbook, Chapter 9 .

A  meta-analysis  goes beyond critique and integration and conducts secondary statistical analyses on the outcomes of similar studies. Systematic reviews may use quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings. Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted. In that case, an integrative review is an appropriate strategy.  

Literatures reviews focus on the existing literature of a subject. They lack the rigorous systematic methodology of systematic reviews. They rarely conduct exhaustive search strategies and do not publish the search strategy (although there are exceptions due to the general nature of literature reviews.) Literature reviews may examine the literature that is the most commonly cited within a certain time frame. Synthesis according to some criteria is typically employed. Literature reviews can take many forms: theses, dissertations, a component within a research paper, or lab report. Please see the University of North Carolina at Chapel Hill's information on literature reviews here.  

Scoping Review or (Mapping Review)

In general, scoping reviews are commonly used for ‘reconnaissance’ – to clarify working definitions and conceptual boundaries of a topic or field. Scoping reviews are useful for when a body of literature has not yet been comprehensively reviewed, or exhibits a complex or heterogeneous nature not amenable to a more precise systematic review of the evidence. While scoping reviews may be conducted to determine the value and probable scope of a full systematic review, they may also be undertaken as exercises in and of themselves to summarize and disseminate research findings, to identify research gaps, and to make recommendations for future research.

From  Peters, MD, Godfrey, CM,  Khalil , H, McInerney, P, Parker, D & Soares , CB 2015, ' Guidance for conducting systematic scoping reviews', International Journal of Evidence-Based Healthcare, vol. 13, no. 3, pp. 141-146 :

• Guidance for conducting systematic scoping reviews

• PRISMA for Scoping Reviews The PRISMA extension for scoping reviews was published in 2018. The checklist contains 20 essential reporting items and 2 optional items to include when completing a scoping review. Scoping reviews serve to synthesize evidence and assess the scope of literature on a topic. Among other objectives, scoping reviews help determine whether a systematic review of the literature is warranted. more... less... Check out the Statement/Explanatory paper by Tricco et al. (2018) and the additional Tip Sheets for Items 1-22 in the PRISMA checklist for Scoping Reviews

Rapid reviews

Rapid reviews utilize systematic review methodology, but they have a more streamlined process for possible time constraints. Defining the limitations and the drawbacks of implementing a streamlined process (and a process that may not incorporate all the components of a systematic review for transparency and systematization) must be described. To learn more about rapid reviews, check out the link below. 

• A scoping review of rapid review methods

Umbrella Review

An Umbrella review is a synthesis of existing reviews, only including the highest level of evidence such as systematic reviews and meta-analyes. It specifically refers to a review that compiles evidence from multiple reviews into one accessible and usable document. Umbrella reviews focus on either a broad condition or problem for which there are competing interventions. These reviews can highlight the different interventions and their results.

Methodology paper :  Aromataris , E, Fernandez, R, Godfrey, CM, Holly, C,  Khalil , H &  Tungpunkom , P 2015, 'Summarizing systematic reviews: Methodological development, conduct and reporting of an umbrella review approach',  Int J Evid Based Healthc , vol. 13, no. 3, pp. 132-140.

• Summarizing systematic reviews

Systematized reviews

A systematized review attempts to include elements of the systematic review process while stopping short of the systematic review. Systematized reviews are typically conducted as a postgraduate student assignment, in recognition that they are not able to draw upon the resources required for a full systematic review (such as having two reviewers for extensive literature screening). 

• << Previous: Useful Resources
• Last Updated: Nov 3, 2023 10:07 AM
• URL: https://libguides.library.tmc.edu/Systematic_Reviews

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Start your free trial

Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.

REQUEST A FREE TRIAL

• Thought for Food Blog

What is the difference between a systematic review and a systematic literature review?

By Carol Hollier on 07-Jan-2020 12:42:03

For those not immersed in systematic reviews, understanding the difference between a systematic review and a systematic literature review can be confusing.  It helps to realise that a “systematic review” is a clearly defined thing, but ambiguity creeps in around the phrase “systematic literature review” because people can and do use it in a variety of ways. 

A systematic review is a research study of research studies.  To qualify as a systematic review, a review needs to adhere to standards of transparency and reproducibility.  It will use explicit methods to identify, select, appraise, and synthesise empirical results from different but similar studies.  The study will be done in stages:  

• In stage one, the question, which must be answerable, is framed
• Stage two is a comprehensive literature search to identify relevant studies
• In stage three the identified literature’s quality is scrutinised and decisions made on whether or not to include each article in the review
• In stage four the evidence is summarised and, if the review includes a meta-analysis, the data extracted; in the final stage, findings are interpreted. [1]

Some reviews also state what degree of confidence can be placed on that answer, using the GRADE scale.  By going through these steps, a systematic review provides a broad evidence base on which to make decisions about medical interventions, regulatory policy, safety, or whatever question is analysed.   By documenting each step explicitly, the review is not only reproducible, but can be updated as more evidence on the question is generated.

Sometimes when people talk about a “systematic literature review”, they are using the phrase interchangeably with “systematic review”.  However, people can also use the phrase systematic literature review to refer to a literature review that is done in a fairly systematic way, but without the full rigor of a systematic review. 

For instance, for a systematic review, reviewers would strive to locate relevant unpublished studies in grey literature and possibly by contacting researchers directly.  Doing this is important for combatting publication bias, which is the tendency for studies with positive results to be published at a higher rate than studies with null results.  It is easy to understand how this well-documented tendency can skew a review’s findings, but someone conducting a systematic literature review in the loose sense of the phrase might, for lack of resource or capacity, forgo that step. 

Another difference might be in who is doing the research for the review. A systematic review is generally conducted by a team including an information professional for searches and a statistician for meta-analysis, along with subject experts.  Team members independently evaluate the studies being considered for inclusion in the review and compare results, adjudicating any differences of opinion.   In contrast, a systematic literature review might be conducted by one person. 

Overall, while a systematic review must comply with set standards, you would expect any review called a systematic literature review to strive to be quite comprehensive.  A systematic literature review would contrast with what is sometimes called a narrative or journalistic literature review, where the reviewer’s search strategy is not made explicit, and evidence may be cherry-picked to support an argument.

FSTA is a key tool for systematic reviews and systematic literature reviews in the sciences of food and health.

The patents indexed help find results of research not otherwise publicly available because it has been done for commercial purposes.

The FSTA thesaurus will surface results that would be missed with keyword searching alone. Since the thesaurus is designed for the sciences of food and health, it is the most comprehensive for the field. 

All indexing and abstracting in FSTA is in English, so you can do your searching in English yet pick up non-English language results, and get those results translated if they meet the criteria for inclusion in a systematic review.

FSTA includes grey literature (conference proceedings) which can be difficult to find, but is important to include in comprehensive searches.

FSTA content has a deep archive. It goes back to 1969 for farm to fork research, and back to the late 1990s for food-related human nutrition literature—systematic reviews (and any literature review) should include not just the latest research but all relevant research on a question. 

You can also use FSTA to find literature reviews.

FSTA allows you to easily search for review articles (both narrative and systematic reviews) by using the subject heading or thesaurus term “REVIEWS" and an appropriate free-text keyword.

On the Web of Science or EBSCO platform, an FSTA search for reviews about cassava would look like this: DE "REVIEWS" AND cassava.

On the Ovid platform using the multi-field search option, the search would look like this: reviews.sh. AND cassava.af.

In 2011 FSTA introduced the descriptor META-ANALYSIS, making it easy to search specifically for systematic reviews that include a meta-analysis published from that year onwards.

On the EBSCO or Web of Science platform, an FSTA search for systematic reviews with meta-analyses about staphylococcus aureus would look like this: DE "META-ANALYSIS" AND staphylococcus aureus.

On the Ovid platform using the multi-field search option, the search would look like this: meta-analysis.sh. AND staphylococcus aureus.af.

Systematic reviews with meta-analyses published before 2011 are included in the REVIEWS controlled vocabulary term in the thesaurus.

An easy way to locate pre-2011 systematic reviews with meta-analyses is to search the subject heading or thesaurus term "REVIEWS" AND meta-analysis as a free-text keyword AND another appropriate free-text keyword.

On the Web of Science or EBSCO platform, the FSTA search would look like this: DE "REVIEWS" AND meta-analysis AND carbohydrate*

On the Ovid platform using the multi-field search option, the search would look like this: reviews .s h. AND meta-analysis.af. AND carbohydrate*.af.  

Related resources:

• Literature Searching Best Practise Guide
• Predatory publishing: Investigating researchers’ knowledge & attitudes
• The IFIS Expert Guide to Journal Publishing

Library image by  Paul Schafer , microscope image by Matthew Waring , via Unsplash.

• FSTA - Food Science & Technology Abstracts
• IFIS Collections
• Resources Hub
• Diversity Statement
• Sustainability Commitment
• Company news
• Frequently Asked Questions
• Privacy Policy
• Terms of Use for IFIS Collections

Ground Floor, 115 Wharfedale Road,  Winnersh Triangle, Wokingham, Berkshire RG41 5RB

Get in touch with IFIS

© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved     |     Charity Reg. No. 1068176     |     Limited Company No. 3507902     |     Designed by Blend

• Essay Writing
• Extended Essays
• IB Internal Assessment
• Theory of Knowledge
• Literature Review
• Dissertations
• Research Writing
• Assignment Help
• Capstone Projects
• College Application
• Online Class
• Order Assignment

Literature Review vs Systematic Paper: A Complete Comparison

by  Antony W

September 11, 2022

A literature review and systematic paper can be confusing since they both give a summary of existing research on a given subject. 

Despite serving a similar objective, the two vary significantly. In our literature review vs systematic paper,   we look at the differences between the two. 

Key Takeaways

• Systematic and literature reviews enable researchers to find gaps and advance research and implementation in the field
• A literature review gives you a summary of already existing research, and a systematic review answers a specific question without instances of biasness.
• From a value point of view, systematic review connects researchers to evidence-based practices while a literature review focuses strictly on giving a summary of already existing studies.
• While a literature review can have only one participant, a systematic review requires more than one author. 

Literature Review Writing Help

Are you currently stuck on writing your literature review chapter for your dissertation or research paper project and would like to get professional help?

Maybe you have daily assignments to complete and can’t schedule enough time for the literature?

Hire our  literature review writing service   and benefit from the convenience and flexibility of professional help. 

We’ll do complete custom writing for you, edit and proofread your work, and deliver it to your email or Help for Assessment’s account before the due date.

What is Literature Review? 

We can define a literature review as a qualitative summary of evidence on an issue. One uses subjective and informal approaches to collect, analyze, and interpret findings

A literature review is critical in form. Ideally, it gives an unbiased analysis of already existing research. While it’s a standalone publication, it doesn’t feature new data or experiment in any form.

It’s easy to write and usually takes a short period. You can learn how to write a literature review by reading this post .

What is a Systematic Paper?

A systematic paper is applied in evidence-based medicine.

Unlike a literature review, a systematic paper goes deeper into analysis and provides a comprehensive review of already published work on a specific topic.

It focuses on answering a specific question and demand systematic and effective methodology to answer the question asked.

When writing a systematic paper, you have to give a high-level analysis of the primary research on the given research question .  Given the depth of information required, you have to identify, select, synthesize, and appraise all research evidence linked to the question asked

Unlike literature review, a systematic paper can include gray literature such as ongoing clinical trials, unpublished studies ,  reports, dissertations ,  government research, and abstracts.

Literature Review Vs Systematic Paper: What are the Differences?

The table below shows the difference between literature review and systematic paper review:

Literature Review Vs Systematic Paper: The Scope of the Review Question

Literature reviews are usually broad in scope

Literature review even allows an author to place their knowledge within existing research, or give more preferences to information that favors a specific viewpoint

On the other hand, a systematic review begins with a specific clinical research question. You have to find all existing evidence to the research question, and then present your review in a transparent and reproducible way.

Literature Review Vs Systematic Paper: Research  

With A Literature review, an author does research only when necessary (or needed).

Searches tend not to be exhaustive and may not be fully comprehensive. The author tends to base their review on what they’re familiar with

In systematic review, research is exhaustive.

Authors find all the best possible sources of information, from published information to unpublished findings, to answer the research question. 

Also, the research and review process is professionally documented and presented.

Literature Review Vs Systematic Paper: Reason for the Study

A literature review fails to explain why an author included or excluded studies from the review.

In a systematic review, authors give explicit and informed reasons why they why they included or excluded studies on the issue under investigation.

Literature Review Vs Systematic Paper: Qualitative Assessment

Because a literature does not consider the quality of a study, the study design may be biased.

Systematic review on the other hand evaluates the risks of biasness in individual studies and the quality of the evidence provided in the study.

Also, systematic review paper asses the sources of heterogeneity of the study results

Literature Review Vs Systematic Paper: Research Synthesis

While conclusions in literature review are qualitative, the results isn’t an exact indication of the quality of the study conducted

Systematic review does more than just base conclusions on the quality of the study.

In addition to identifying and acknowledging gaps in the clinical studies, the review also addresses the gaps and recommends the best practices.

Related Readings

• Literature Review vs Research Paper
• The Significance of a Literature Review

Frequently Asked Questions

1. can you use systematic review in a literature review.

You cannot use a systematic review in a literature review because the review is work based on original research articles.

Note that any material considered as a secondary source, including but not limited to narrative reviews, meta analyses, and systematic reviews, isn’t fit for a literature review assignment.

2. What Makes a Literature Review Systematic Review?

A literature review becomes a systematic review if the aim is to find relevant research on a selected research question.

Within this context, there must exist explicit methods that identify what you can reliably express based on the studies.

3. Why is a Literature Review Different than a Systematic Review?

Whereas a systematic review uses an analytical approach to collect and evaluate secondary data, a literature review presents a summary of current theories and knowledge on a given topic. 

About the author 

Antony W is a professional writer and coach at Help for Assessment. He spends countless hours every day researching and writing great content filled with expert advice on how to write engaging essays, research papers, and assignments.

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

A Guide to Using AI Tools to Summarize Literature Reviews

Table of Contents

Needless to say, millions of scientific articles are getting published every year making it difficult for a researcher to read and comprehend all the relevant publications.

Back then, researchers used to manually conduct literature reviews by sifting through hundreds of research papers to get the significant information required for the research.

Fast forward to 2023 — things have turned out quite distinct and favorable. With the inception of AI tools, the literature review process is streamlined and researchers can summarize hundreds of research articles in mere moments. They can save time and effort by using AI tools to summarize literature reviews.

This article articulates the role of the top AI tools used to summarize literature reviews. You can also learn how AI is used as a powerful tool for summarizing scientific articles and understanding the impact of AI on academic research.

Understanding the Role of AI Tools in Literature Reviews

Before we talk about the benefits of AI tools to summarize literature reviews, let’s understand the concept of AI and how it streamlines the literature review process.

Artificial intelligence tools are trained on large language models and they are programmed to mimic human tasks like problem-solving, making decisions, understanding patterns, and more. When Artificial Intelligence and machine learning algorithms are implemented in literature reviews, they help in processing vast amounts of information, identifying highly relevant studies, and generating quick and concise summaries — TL;DR summaries.

AI has revolutionized the process of literature review by assisting researchers with powerful AI-based tools to read, analyze, compare, contrast, and extract relevant information from research articles.

By using natural language processing algorithms, AI tools can effectively identify key concepts, main arguments, and relevant findings from multiple research articles at once. This assists researchers in quickly understanding the overview of the existing literature on a respective topic, saving their valuable time and effort.

Key Benefits of Using AI Tools to Summarize Literature Review

1. best alternative to traditional literature review.

Traditional literature reviews or manual literature reviews can be incredibly time-consuming and often require weeks or even months to complete. Researchers have to sift through myriad articles manually, read them in detail, and highlight or extract relevant information. This process can be overwhelming, especially when dealing with a large number of studies.

However, with the help of AI tools, researchers can greatly save time and effort required to discover, analyze, and summarize relevant studies. AI tools with their NLP and machine learning algorithms can quickly analyze multiple research articles and generate succinct summaries. This not only improves efficiency but also allows researchers to focus on the core analysis and interpretation of the compiled insights.

2. AI tools aid in swift research discovery!

AI tools also help researchers save time in the discovery phase of literature reviews. These AI-powered tools use semantic search analysis to identify relevant studies that might go unnoticed in traditional literature review methods. Also, AI tools can analyze keywords, citations , and other metadata to prompt or suggest pertinent articles that align and correlate well with the researcher’s search query.

3. AI Tools ensure to stay up to date with the most research ideas!

Another advantage of using AI-powered tools in literature reviews is their ability to handle the ever-increasing volume of published scientific research. With the exponential growth of scientific literature, it has become increasingly challenging for researchers to keep up with the latest scientific research and biomedical innovations.

However, AI tools can automatically scan and discover new publications, ensuring that researchers stay up-to-date with the most recent developments in their field of study.

4. Improves efficiency and accuracy of Literature Reviews

The use of AI tools in literature review reduces the occurrences of human errors that may occur during traditional literature review or manual document summarization. So, literature review AI tools improve the overall efficiency and accuracy of literature reviews, ensuring that researchers can access relevant information promptly by minimizing human errors.

List of AI Tools to Streamline Literature Reviews

We have several AI-powered tools to summarize literature reviews. They utilize advanced algorithms and natural language processing techniques to analyze and summarize lengthy scientific articles.

Let's take a look at some of the most popular AI tools to summarize literature reviews.

SciSpace Literature Review

Semantic scholar, paper digest.

SciSpace Literature Review is the best AI tool for summarizing literature review. It is the go-to tool that summarizes articles in seconds. It uses natural language processing models GPT 3.5 and GPT 4.0 to generate concise summaries. It is an effective and efficient AI-powered tool to streamline the literature review process and summarize multiple research articles at once. Once you enter a keyword, research topic, or question, it initiates your literature review process by providing instant insights from the top 5 highly relevant papers at the top.

These insights are backed by citations that allow you to refer to the source. All the resultant relevant papers appear in an easy-to-digest tabular format explaining each of the sections used in the paper in different columns. You can also customize the table by adding or removing the columns according to your research needs. This is the unique feature of this literature review AI tool.

SciSpace Literature review stands out as the best AI tool to summarize literature review by providing concise TL;DR text and summaries for all the sections used in the research paper. This way, it makes the review process easier for any researcher, and could comprehend more research papers in less time.

Try SciSpace Literature Review now!

Semantic Scholar is an AI-powered search engine that helps researchers find relevant research papers based on the keyword or research topic. It works similar to Google Scholar.It helps you discover and understand scientific research by providing suitable research papers. The database has over 200 million research articles, you can filter out the results based on the field of study, author, date of publication, and journals or conferences.

They have recently released the Semantic Reader — an AI-powered tool for scientific readers that enhances the reading process. This is available in the beta version.

Try Semantic Scholar here

Paper Digest — another valuable text summarizer tool (AI-powered tool) that summarizes the literature review and helps you get to the core insights of the research paper in a few minutes! This powerful tool works pretty straightforwardly and generates summaries of research papers. You just need to input the article URL or DOI and click on “Digest” to get the summaries. It comes for free and is currently in the beta version.

You can access Paper Digest here !

SciSummary is another AI tool that summarizes scientific articles and literature review. It uses natural language processing algorithm to generate concise summaries. You need to upload the document on the dashboard or send the article link via email and your summaries will be generated and delivered to your inbox. This is the best AI-powered tool that helps you read and understand lengthy and complicated research papers. It has different pricing plans (both free and premium) which start at $4.99/month, you can choose the plans according to your needs.

You can access SciSummary here

Step-by-Step Guide to Using AI Tools to Summarize Literature Reviews

Here’s a short step-by-step guide that clearly articulates how to use AI tools for summary generation!

• Select the AI-powered tool that best suits your research needs.
• Once you've chosen a tool, you must provide input, such as an article link, DOI, or PDF, to the tool.
• The AI tool will then process the input using its algorithms and techniques, generating a summary of the literature.
• The generated summary will contain the most important information, including key points, methodologies, and conclusions in a succinct format.
• Review and assess the generated summaries to ensure accuracy and relevance.

Challenges of using AI tools for summarization

AI tools are designed to generate precise summaries, however, they may sometimes miss out on important facts or misinterpret specific information.

Here are the potential challenges and risks researchers should be wary of when using AI tools to summarize literature reviews!

1. Lack of contextual intelligence

AI-powered tools cannot ensure that they completely understand the context of the research papers. This leads to inappropriate or misleading summaries of similar academic papers.

To combat this, researchers should feed additional context to the AI prompt or use AI tools with more advanced training models that can better understand the complexities of the research papers.

2. AI tools cannot ensure foolproof summaries

While AI tools can immensely speed up the summarization process, but, they may not be able to capture the complete essence of a research paper or accurately decrypt complex concepts.

Therefore, AI tools are just to be considered as technology aids rather than replacements for human analysis or understanding of key information.

3. Potential bias in the generated summaries

AI-powered tools are largely trained on the existing data, and if the training data is biased, it can eventually lead to biased summaries.

Researchers should be cautious and ensure that the training data is diverse and representative of various sources, different perspectives, and research domains.

4. Quality of the input article affects the summary output

The quality of the research article that we upload or input data also has a direct effect on the accuracy of the generated summaries.

If the input article is poorly written or contains errors, the AI tool might not be able to generate coherent and accurate summaries. Researchers should select high-quality academic papers and articles to obtain reliable and informative summaries.

Concluding!

AI summarization tools have a substantial impact on academic research. By leveraging AI tools, researchers can streamline the literature review process, enabling them to stay up-to-date with the latest advancements in their field of study and make informed decisions based on a comprehensive understanding of current knowledge.

By understanding the role of AI tool to summarize literature review, exploring different AI tools for summarization, following a systematic review process, and assessing the impact of these tools on their academic research, researchers can harness AI tools in enhancing their literature review processes.

If you are also keen to explore the best AI-powered tool for summarizing the literature review process, head over to SciSpace Literature Review and start analyzing the research papers right away — SciSpace Literature Review

Few More Insightful Articles — Just for you!

10 best ai for essay writing, role of ai in systematic literature review, how to use ai tools for conducting a literature review, research paper summarizer | an overview of the best ai summarizers, you might also like.

AI for Essay Writing — Exploring Top 10 Essay Writers

• Open access
• Published: 21 October 2023

Dairy consumption in adults in China: a systematic review

• Shuhua Yang 1 , 2 , 3 ,
• Nupur Bhargava 1 , 2 ,
• Aileen O’Connor 1 , 2 ,
• Eileen R. Gibney 1 , 2 , 3 &
• Emma L. Feeney 1 , 2 , 3  

BMC Nutrition volume  9 , Article number:  116 ( 2023 ) Cite this article

366 Accesses

10 Altmetric

Metrics details

Research on dairy consumption in China is lacking, however, some evidence has demonstrated significant changes in recent years, with a reported increase in the overall consumption of dairy products. To fully understand these changes, a systematic review was conducted to examine reported dairy intakes and differences between dairy consumption in different population groups in China. Methods: Web of Science, Embase, and PubMed databases were searched for studies published from January 2000 to September 2022. The China National Knowledge Infrastructure (CNKI) was used to retrieve papers available in Chinese. Papers reporting dietary intakes of dairy consumption across age, sex, and geographical location sub-groups were considered for inclusion in this review. In addition, this review includes the consumption of different types of dairy foods and changes in dairy intake over time. Results: Forty-seven papers were included in the present study. Twelve papers examined dairy consumption across age groups, showing that middle-aged adults tend to consume less dairy than other age groups. Studies comparing across location-specific cohorts reported dairy intakes among urban populations were higher than rural, as well as being higher than the national average. Coastal, Northern and Eastern residents consumed more dairy products than those living in other regions of China, and people in larger cities had higher reported intakes than smaller cities. Milk was the primary dairy product reportedly consumed by Chinese population, followed by yogurt. Concerning sex, evidence showed that females generally reported a greater daily dairy intake than males. Conclusions: This review shows that, in China, several different population groups displayed significant differences in the amount and type of dairy consumed. When considering the incorporation of dairy products into healthy eating guidelines or positioning specific dairy products on the market, it is important to consider the differences and variations in consumption patterns within population groups.

Peer Review reports

Introduction

Dairy foods such as milk, cheese and yogurt are recognized as important sources of beneficial nutrients, including vitamins D, B5 [ 1 ] and B12 [ 1 , 2 ], and minerals such as calcium [ 3 ], phosphorus, and potassium [ 1 ]. Many health benefits of dairy products are acknowledged [ 4 ], such as an impact on anthropometric measurements (i.e. weight, and waist circumference) [ 5 , 6 ]. Reduced risk of hypertension (HTN) linked to dairy consumption has also been reported, whereby peptides contained within milk have been shown to reduce blood pressure through inhibition of the angiotensin pathway [ 7 ]. One study, conducted in the USA, found that each additional serving of yogurt (227 g) was associated with a 6% reduced risk of incident HTN [ 8 ]. Similarly, in a large epidemiological study of Chinese adults, a significant association between a higher frequency of dairy consumption and reduced HTN was noted [ 9 ]. Higher intake of dairy was also reported to be associated with lower blood pressure levels in a sample of Chinese young women [ 10 ]. In addition, a study in China found that regular dairy consumption (≥ 4 days/week) was associated with a lower risk of ischemic heart disease (IHD) in males [ 11 ]. Evidence has also shown that consumption of dairy may offer protection against risk of other diseases such as metabolic syndrome [ 12 , 13 ], cardiovascular disease (CVD) [ 14 , 15 , 16 ], stroke [ 17 ], obesity [ 13 , 18 , 19 ], type 2 diabetes [ 20 ] and colorectal cancer [ 21 ]. However, although dairy products contain numerous beneficial nutrients, and their consumption may have a positive impact on health, there are still some concerns regarding the consumption of some dairy foods. Much of this concern is related to the saturated fatty acid (SFA) content, present in dairy products [ 22 ], known to be related to the risk of coronary heart disease (CHD) [ 23 ].

Recommendations concerning dairy consumption are given in many national nutrition and healthy eating guidelines [ 24 , 25 , 26 , 27 ]. In Ireland, as an example, the recommendation is 3 servings each day from the food group “milk, yoghurt and cheese” [ 24 ]. In the US, 3 daily servings of dairy products are recommended for US adults [ 25 ]. However, in Asian countries, recommendations for the consumption of dairy are lower than in western countries [ 28 , 29 , 30 ]. In China, a variety of dairy products, equivalent to 300ml of liquid milk per day, are recommended in the 2022 Chinese Dietary Guidelines CDGs [ 30 ].

Dietary patterns in China are known to differ quite significantly from those reported in other global regions including Europe and the US [ 31 , 32 , 33 , 34 , 35 ]. Traditional Chinese dietary patterns are represented by ‘Rice, vegetables, and meat’, while the ‘modern’ Chinese dietary pattern is represented by ‘fast food, milk and deep-fried food’ [ 34 ]. Similar differences are seen within the US, where two major dietary patterns has been identified from national surveys, one was ‘nonwhole grain, white potatoes, cheese, meat, discretionary oil and fat, and added sugar’, and another one was ‘whole grains, vegetable, fruits, fish, nuts and seeds’ [ 35 ]. Researchers in the US also compared Chinese dietary intakes to American diets, reporting that the Chinese diet had a lower daily intake of fiber, vitamins and some micronutrients than the American diet [ 33 ]. In China, whilst dairy products have been available and intakes of dairy have been rising in the past decades dairy consumption remains low compared to the recommended dietary guidelines for Chinese [ 36 , 37 ]. This low consumption is attributed to several factors, such as lack of refrigeration, limited supply and high prices and a traditional plant-based diet [ 38 , 39 ]. As a result of low intakes, in one study, dairy foods were found to contribute only 4.3% of calcium intake, with “vegetable, bean and bean products” as the main source of calcium [ 40 ]. This was relatively low compared to other countries. For instance, in Ireland, dairy contribute 38.8% of calcium to the total diet [ 41 ]. And in Poland, the contribution from dairy to total calcium intake was 54.7% in the average Polish diet [ 42 ]. However, another survey, conducted among an elderly cohort in Beijing, found that dairy products were the main contributor to calcium, contributing 34.5% among older adults aged 60 years and over [ 43 ], indicating that whilst overall consumption is low, considerable variance exists within the population.

In recent decades, the dairy industry in China has grown steadily, prompted by economic factors including the growth in household income, consumer preferences and the provision of financial support from the government [ 44 ]. However, due to existing eating habits, consumer preferences, and other historical factors such as traditional agricultural practices and dietary practices in different regions in China, variations in the consumption of dairy products exist in different sub-groups e.g. gender, location groups, which has been reported in several studies to date [ 45 , 46 , 47 , 48 ]. Understanding the variations in consumption may help to elucidate factors influencing intake, and support the development of strategies to increase consumption among specific population groups, in accordance with dietary recommendations [ 49 , 50 ]. For instance, in the US, food based recommendations have been developed for various age and gender groups providing food choices that will help the population group to meet nutritional recommendations [ 50 ].

The purpose of this paper was to systematically review existing literature reporting dairy consumption among the Chinese population, living in mainland China. The objectives of the study were to summarise the available literature providing information on dairy intakes in the Chinese population, to examine the differences in the consumption of dairy across different population sub-groups and to further identify the factors which contribute to the differences in consumption.

The present systematic review was carried out following the updated Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA 2020) guidelines [ 51 ]. The protocol of this review was previously registered on PROSPERO (International Prospective Register of Systematic Reviews) (registration number: CRD42021285208).

Search strategy

Within this review, the term ‘dairy product’ is defined as milk, yogurt, milk powder, cheese, butter, cream or ice cream. The search strategy of this review followed the PICO framework, focusing on the differences in dairy consumption among different ages, geographic location sub-groups, sex groups among Chinese adults in mainland China, as well as the difference in consumption of the different types of dairy products and the overall changes in dairy consumption over time. The following search terms were used: Dairy OR Milk OR Cheese OR Yogurt OR Yoghurt OR Yoghourt OR Butter OR Cream OR Milk powder OR Food AND Intake OR Consumption OR Market OR Diet OR Dietary AND China OR Chinese OR Asian. The search was limited to studies carried out in human adults (≥ 18 years), written in English or Chinese languages. A manual search of references from included studies was also conducted. We used Google Scholar to retrieve papers where applicable. The China National Knowledge Infrastructure (CNKI) was also used to retrieve papers when the full-text papers were only available in Chinese. Two authors (S.Y. and N.B.) independently performed the literature search in Web of Science, Embase, and PubMed databases for papers published between January, 2000 and October, 2021. To ensure a focus on the most recent research regarding dairy consumption status, papers published before the year 2000 were not searched. An updated search of all the datasets was completed by one researcher (S.Y.) on 06 September 2022.

Study screening and eligibility criteria

Published papers examining dairy intake by considering mean intake, median intake, frequency of consumption, and/or percentage of Chinese adult consumers living in mainland China were included. Study designs that were considered in this review included but not limited to dietary intake assessment study, intervention study but reporting dairy intake of control group at baseline, and consumer behaviour papers that reported findings of dairy intakes. Papers reporting the findings related to comparison of dairy consumption across age, sex, and geographical location sub-groups, different types of dairy products, and different years were included in the analysis in the present review. Papers were excluded if the original study was conducted in Chinese group living in other countries except for China. Papers were excluded if there were only children and/or teenagers involved in the study. Papers that assessed intake of human milk only were excluded. Papers reporting intakes of dairy food groups but including irrelevant food such as egg were excluded. Papers, involving intervention studies but did not report dairy intake data of participants in general good health in control group at the baseline, were excluded. For papers that reported data for those aged < 18 and ≥ 18 years, only data from those over 18 years were considered in the analysis of this review where applicable. Two authors (S.Y. and N.B.) independently screened papers for eligibility firstly based on titles, then abstracts and finally full texts based on the predefined inclusion and exclusion criteria. In the case of disagreement, a third researcher (E.R.G.) was involved, and consensus on inclusion or exclusion was reached after discussion.

Data extraction and quality assessment

Papers included in the present review reported dairy consumption in varies ways. The following information were extracted by one author (S.Y.) firstly from the all the papers reporting dairy intakes: study characteristics (first author, publication year, sample size, study location, year of data collection, dietary assessment method); population demographic characteristics (age, sex); type of reported dairy food (total and / or individual food products if reported). For the studies using data from national survey (i.e. China Health and Nutrition Survey) without specifying study location, the survey location information was searched and taken from the survey website [ 52 ] or presented as national according to the dataset used in papers. Dietary assessment method for those papers missing relative information were taken from survey website [ 11 ] or other papers which used same survey dataset and provided more detailed information. Following, studies where they reported findings of intake differences between age groups were summarized together. Age groups in each study included in the present review were further specified and presented for the comparison within and between studies. Population size, and age details of total population and groups were displayed where applicable. Similarly, information of geographical location sub-groups, sex groups and consumption of different types of dairy products were extracted and summarized for comparison, and the changes of dairy consumption over time were also compared and presented. Basic calculation, such as counting the percentage of consumer based on the number given in papers, was conducted in this review for easier presenting and comparing of findings. Depending on the methods and analysis operated in published papers, the dairy intakes were reported in percentage of consumers, frequency of intake, mean/median intakes (g/d, kg/y, ml/d), range of intakes or descriptive sentences without statistical results in the key findings. The intake presented in this review was absolute amount of intake, not energy-adjusted. If more than one papers used data from the same study or dataset, data from the publication with the greatest detail of information were presented in this review. During the data collection, two authors (E.R.G. and E.L.F.) were involved when a paper needed to be discussed.

To assess risk of bias, the quality of the studies included in this review was examined. S.Y. performed the quality assessment. Given the various of study methods in those studies, the Critical Appraisal Skills Programme (CASP) checklist for Cohort Studies [ 53 ] was applied. The CASP checklist for Cohort Studies consists of several domains that evaluate key aspects of cohort study design, including the clarity of the research question, cohort selection, measurement of variables, consideration of confounding factors, follow-up periods, statistical analysis, and quality of results. 12 questions in the cohort study checklist was used. Two of the questions was scored up to 2 points. Total of 14 points was given if a study met all the criteria.

Literature search results and study characteristics

A total of 10,685 papers were searched from three databases after removing duplicates. Studies identified were screened based on titles and abstracts, and finally full texts of 375 papers including the 54 papers which were identified from the reference lists were assessed according to the inclusion and exclusion criteria. Ultimately, 47 papers were included in the present study. Full details of the search are outlined in Fig.  1 .

PRISMA flow diagram

Full characteristics of the papers and CASP scores from quality assessment are shown in Table  1 . Within the included papers, 24 papers reported findings on total dairy consumption. 16 papers investigated milk only. The remaining 7 papers investigated sub-groups of dairy products. Dairy intake data from 21 papers were draw from several national surveys conducted in China [ 46 , 47 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 ]. Within papers that reported the number of participants, sample sizes ranged from 117 to over 90,000. With respect to reported dietary intake assessment methodology, 24-hour dietary recalls [ 46 , 47 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 67 , 68 , 69 , 70 , 73 , 74 , 75 , 76 ], Food Frequency Questionnaires [ 71 , 72 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 , 88 ], Questionnaires or in-person interview [ 38 , 45 , 72 , 89 , 90 , 91 , 92 ], and Internet-based dietary questionnaire for Chinese (IDQC) [ 93 , 94 , 95 ] were used in the data collection in reported studies to assess overall diet.

Dairy consumption in different age groups

Of the 47 studies included in the final review, 12 reported dairy consumption across different age groups [ 45 , 46 , 47 , 55 , 62 , 63 , 64 , 65 , 69 , 73 , 79 , 89 ] (Table  2 ). In three studies, dairy consumption in those aged or average age over 60 were compared with other age cohorts [ 45 , 46 , 73 ]. Four studies focused on older cohorts aged over 50, with one reporting the differences in dairy intakes in those aged 50–70 [ 79 ], one that compared individuals aged 60–79 and 80 over [ 62 ], and three that compared ages 60–69 and 70 over [ 55 , 65 , 69 ]. One didn’t compare intakes between age groups but reported and compare the median age at low, high and non-consumer groups [ 64 ]. The other remaining studies included dairy consumption of working-age adults (20–59 years) [ 47 ], (18–59 years) [ 63 ], while one study used just 3 age groups to cover all ages (< 30, 30–50 and > 50) [ 89 ].

Of the three studies that compared dairy consumption in population groups aged under and over 60 years, two of these studies showed that people aged over 60 years reported consuming higher amounts [ 45 ], while had lower frequency of milk intake [ 46 ], compared to other age groups. Ba et al. [ 73 ] found that older adults had higher intakes of milk than younger adults with daily intakes reported in older adults (≥ 60 years) of 163.4 g/d, which was significantly greater than intakes reported in those aged 18–44 years and 45–59 years, with reported milk intakes of 75.8 and 96.6 g/d, respectively.

Focusing on people aged over 50 years, dairy consumption was reported in four studies. Xu et al. [ 55 ] reported that the median dairy intakes in males aged 60–69 years who consumed dairy in 2009 was 200 g/d, while the number in males aged 70 years and over was only 162 g/d. Likewise, Zong et al. [ 79 ] found that, within the age group 50–70 years, participants with higher intakes of dairy products were more likely to be of a younger age. In addition, Liu et al. [ 69 ] and Wang et al. [ 65 ] both found that people aged 70 years and over had significantly higher dairy intakes than those aged 60–69 years ( P  < 0.001 and P  < 0.05 separately), with average intakes in these two age groups of 39.57 and 28.49 g/d, respectively. Similarly, Huang et al. [ 62 ] compared differences in dairy consumption between the age groups 60–79 years and 80 years and over, reporting that people aged over 80 years consumed significantly more dairy. One of the largest studies, Tian et al. [ 47 ] assessed dietary intake in residents from 12 cities and provinces in 2004, 2006, 2009 and 2011, and analysed intakes across two age groups (20–39 years, 40–59 years). Within this study, those aged 40–59 years reported higher mean daily dairy intakes than those aged 20–39 years, with intakes of 14.2 ± 55.8 g/d and 13.0 ± 47.1 g/d in each age group, respectively. However, this difference was not significant ( P  > 0.05). Similarly, results from the survey of Bai et al. [ 89 ], conducted in Qingdao city in 2005, showed that the people aged over 50 years consumed more milk than other age groups. However, these differences were not statistically tested, and only reported descriptively. Additionally, Wang et al. [ 63 ] analysed the national dairy consumption data from 1989 to 2011, finding that dairy consumers aged 40–59 years had higher average dairy intakes than adults aged 18–39 years in most of the years except in 1989, 1997 and 2011. Although, this difference was not significant ( P  > 0.05).

Dairy consumption in different geographical location groups

Of the 13 studies reporting on dairy consumption across location-specific cohorts comparing people living in different cities or provinces, two papers focused on dairy consumption in individual cities [ 38 , 45 ], and eleven papers reported on dairy consumption in different regions of China classified by urban, rural; North, South, costal, inland; East, West, central; the size of city or economic status of rural area [ 46 , 47 , 56 , 63 , 64 , 65 , 67 , 68 , 71 , 72 , 79 ]. Table  3 summarises the characteristics and key findings of these papers.

Nine of the 11 papers examined dairy consumption between urban and rural areas, and reported higher intakes of dairy products in urban populations compared to those living in rural areas [ 47 , 56 , 63 , 64 , 65 , 67 , 68 , 72 , 79 ]. For example, Tian et al. [ 47 ] examined milk intakes from 12 cities or provinces in 2004, 2006, 2009, 2011 in China, and reported a greater mean intake of 30.9 g/d in urban populations, compared to only 5.1 g/d in rural residents. Zhang et al. [ 68 ] reported lower mean daily dairy intakes in a rural area in 2002 of 11.4 g/d, compared to 65.8 g/d in urban residents in the same study.

Wang et al. examined differences in reported dairy intakes from urban and rural areas from 1989 to 2011 using data from CHNS. The authors reported that urban residents had a significantly higher consumption than people living in rural areas across these years ( P  < 0.0001) [ 63 ]. Most recently, He et al. reported a significant difference in high milk consumption in urban and rural areas among 31 provinces in China, with a high percentage of consumers (74.17%) are living in urban areas. The high milk consumption in this study was classified as ≥ 200 ml/day and ≥ 5 day/week [ 72 ]. In addition, one paper analyzed dietary intake data from national survey CHNS in 1991, 2000 and 2015, reporting a significant difference of mean daily intake between urban and rural residents with 40.4 g in urban areas and 10.6 g in rural areas ( P  < 0.05) [ 65 ].

Of the papers that examining dairy consumption in other geographical location groups, Li et al. [ 67 ] compared milk intakes between coastal and inland areas, reporting that people living in coastal areas had higher milk intakes than those living in inland, reporting mean intakes of 32.65 and 25.62 g/d, respectively. Research also found that those living in Northern China reported higher milk intakes than those living in Southern China in three separate studies [ 38 , 67 , 79 ]. For example, Li et al. [ 67 ] found that, at a national level in 2002, those in northern regions consumed more milk than people living in southern regions, with reported intakes of 33.38 g/d and 22.24 g/d, respectively. A difference in dairy consumption was also found among people living in Eastern, Central and Western areas, where it was reported that people living in Eastern cities had significant higher intakes than people living in the other two areas [ 64 , 71 ]. Furthermore, only one study compared milk consumption according to the size of the city and type of rural area, demonstrating that people living in big cities consumed much more milk than those living in smaller sized cities and normal rural areas, with 64.3 g/d in big cities, 24.2 and 9.1 g/d in other areas respectively [ 46 ].

Dairy consumption in different sex groups

Table  4 summarises the results from 16 papers that considered differences in dairy consumption across reported sex groups (male and female). All but four of these papers reported higher dairy consumption in females than males [ 47 , 54 , 55 , 60 , 61 , 63 , 64 , 65 , 69 , 72 , 78 , 79 , 84 , 86 , 92 , 95 ]. Within those papers, eleven studies analysed data at the national level. Specifically, 8 papers analysed data from the national survey CHNS, while focusing on the different age groups and/or different collection years [ 47 , 54 , 55 , 60 , 61 , 63 , 64 , 65 ]. One used data from the CNHS study in 2010–2012 [ 69 ]. One study analyzed the data from CNSSPP [ 72 ]. In addition, one study conducted across different regions in China [ 92 ]. The other five studies were conducted in individual cities (Beijing, Shanghai or Guangzhou) [ 78 , 79 , 86 ], Tibet [ 84 ] or regional locations (northern China) [ 95 ].

At the national level, papers that studied the survey data in different years from 1989 to 2011 reported higher dairy intakes among females, with significant differences found by Wang et al. [ 63 ] and Tian et al. [ 47 ], whilst the difference between sexes was either not significant or not statistically tested in other papers [ 54 , 60 , 61 , 64 ].

Mirroring the findings from these national studies, Zong et al. [ 79 ] examined dairy consumption in males and females aged 50–70 years in Beijing and Shanghai in 2005, and found that females in this age group consumed higher amounts of dairy than males, with only 25.8% of those who consumed more than one serving of dairy foods per day being male. Sun et al. [ 78 ] collected information on milk consumption in older Chinese (aged over 50 years) in Guangzhou across two time periods ((Phase 1 (2003–2004) and Phase 2 (2005–2006)), and reported a slight difference between males and females, with 27% females and 25% males consuming over 250 ml whole cow’s milk per week, however, the results were not statistically analysed, and thus are observational. Guo et al. [ 95 ] examined the proportion of sexes across quartiles of reported dairy consumption in people living in northern China, finding that females had higher dairy intakes than males, with 47.23% males in Q1 (mean intake 6.42 ml/d), compared to 35.02% males in Q4 (mean intake 227.89 ml/d).

Four of the 16 papers examining differences in reported dairy intake across sex groups found that, for those who consumed dairy products, males had higher dairy intake compared to females with only one study reported significant difference [ 55 , 65 , 84 , 92 ]. Xu et al., who examined reported intakes using data from CHNS 2009 [ 55 ] reported that more males met the recommended intakes for dairy than females in older adults, with median intakes in males and females aged 60–69 years 200 g/d and 167 g/d respectively. However, the differences were not statistically tested, and only provided as descriptive figures. Another study, which collected data during the COVID-19 lockdown period from March to April 2020, which examined dietary behavior across China showed that males consumed milk more frequently ( P  < 0.001) and more dairy in general compared to females [ 92 ]. Finally, another study examining intakes in the Tibetan plateau, showed greater consumption of dairy foods in males compared to females, however the amount of intake was not reported and statistically tested [ 84 ].

Consumption of different types of dairy products

Differences in the consumption of the different types of dairy products were reported in six papers [ 38 , 62 , 90 , 91 , 95 , 96 ] (Table  5 ). Two of the six studies reported the mean amount consumed or the range on intakes for milk, yogurt, milk powder and ice cream [ 38 , 90 ]. One reported the percentage of consumers of each product among people aged 60 years and over with a focus on milk, yogurt, milk powder and other dairy products [ 62 ]. The other three focused on specific products, namely; milk, yogurt and milk powder [ 95 ], milk and yogurt [ 91 ] and only milk and butter [ 96 ].

All six papers showed that participants had highest intake of milk among these types of dairy products in China. Fuller et al. [ 38 ], examining intakes in Beijing, Shanghai and Guangzhou in 2001, reported that of the annual dairy products consumed, milk consumption was highest in these three cities, with yogurt consumption ranked second, followed by ice cream and milk powder. They also reported that younger, more educated participants consumed more yogurt, whilst elderly participants tended to consume more milk powder. Similarly, the other three studies also reported much higher milk consumption than other types of dairy products (yogurt, milk powder, butter) [ 91 , 95 , 96 ]. Silanikove et al. [ 96 ] reported remarkably lower annual intakes of butter than milk in 2011 with 0.1 kg/y of butter and 9.1 L/y of milk. More recently, Huang et al. [ 62 ] investigated the dairy consumption in 4921 participants aged 60 years and over, and reported the percentage of consumers of each type of dairy product, finding that milk and yogurt were the main dairy products consumed in this group. Yang et al. [ 91 ] who examined the dairy consumption among adults in China during the COVID-19 lockdown, reported that the median intakes of milk and yogurt were 71.5 ml/d and 17.8 ml/d separately.

Changes in dairy consumption over time

Seven papers report analysis of dairy consumption over time at a national level using data from CHNS [ 47 , 56 , 58 , 63 , 65 ], CNNHS [ 68 ] and NBS [ 66 ]. Of the five papers that analysed data from CHNS, one examined dairy intakes in adults aged 18–45 across 6 survey years (1989, 1991, 1993, 1997, 2000, 2004) [ 56 ], and one studied dairy intakes across four survey years (2004, 2006, 2009, 2011) among people aged 20–59 years [ 47 ]. Batis et al. [ 58 ] reported the percentage of consumers of animal-based milk during survey years 1991–2009. The other reported the dairy consumption data of adults aged 18–59 years, covering all of nine survey years (1989–2011) [ 63 ]. In addition, Wang et al. [ 65 ] examined dietary intake data in 1991, 2000 and 2015 among people aged ≥ 60 years in China. Data from these studies showed an increase in dairy intakes. For example, during the period 1989–2004, consumption of dairy products was reported to increase six-fold from 2 g/d to 12 g/d [ 56 ]. From 2004 to 2009, consumption of milk and its products then appeared to experience a decreasing trend, reaching its lowest consumption in 2009, of 25 g/d. However, from 2009 to 2011, reported intakes increased to 35 g/d, which was higher than that of the previous year [ 47 ]. Additionally, from 1991 to 2015, the average intake of dairy foods among elders had significant increase, with 8.0 g/d in 1991, 14.1 g/d in 2000 and 20.3 g/d in 2015 ( P  < 0.001) [ 65 ].Of the other two papers, Fu et al. [ 66 ] reported increasing consumption of dairy products from NBS for both urban and rural areas from 1990 to 2010, with reported dairy intakes from 0.64 kg/y to 3.55 kg/y in rural area, 4.60 kg/y to 18.10 kg/y in urban area, whereas the dairy intakes in urban residents experienced a significant decline from 22.54 to 18.10 kg/y from 2006 to 2010. The remaining paper using the data from CNNHS reported a similar increase in reported intakes of dairy products from 1982, to 1992 and 2002, reporting intakes of 8.1, 14.9, and 26.5 g/d separately [ 68 ]. It also further reported the specific changes in urban and rural areas. Compared to rural areas, urban residents reported a significantly greater increase in dairy consumption during this period, with 9.9 and 65.8 g/d reported in 1992 and 2002 in urban groups, compared to 7.3 and 11.4 g/d in rural groups. When considering differences within individual provinces, one paper reported changes in dietary intakes from 1982 to 2012 in the Hunan province, reporting that dairy intakes experienced a rapid increase from 1982 (5.9 g/d) to 2002 (95.5 g/d), but this then decreased to 16.6 g/d in 2012 [ 75 ].

In addition, researchers examined the changes of eating habits in elderly residents during COVID-19 lockdown in March 2020 in Wuhan city in China, finding that dairy consumption was reduced during this period [ 85 ]. Specifically, a 24.5% reduction was observed among males, and 45.3% among females. Considering age groups, dairy consumption reduced by 38.8% in 60–69 year old, 40.0% in 70–79 year old and 25% in those aged 80 and over.

Based on published literature between 2000 and 2022, which reported the consumption of total or individual dairy foods in China, some consumption patterns of dairy can be observed. Our review found noteworthy differences in dairy consumption across population groups of age, geographic location and sex, as well as differences by type of dairy. Specifically, milk and yogurt were reported to be the main dairy foods consumed in China with milk powder playing an important role in the intake of dairy in older adults. In terms of sex-related differences in dairy consumption, evidence showed that females had higher intakes than males. Clear patterns of dairy emerged across different geographical locations. The intake of dairy products among the urban population was higher than rural areas and also greater than the national average. Furthermore, coastal citizens and those in northern and eastern regions consumed more dairy products than others. Meanwhile, residents in larger cities had higher intakes than smaller cities or rural area. To the best of our knowledge, this is the first systematic review to summarise reported dairy intakes to determine factors that influence the consumption of dairy in different groups in China.

When examining dairy intake in the studies, both total dairy and also the following individual dairy foods were considered: milk, yogurt, ice cream, milk powder, butter. Much of the reporting considered total dairy and did not break down reported intakes into these individual dairy foods. From studies included in this review, milk, yogurt and milk powder were the main dairy foods reported among Chinese adults. In contrast, consumption of butter and cheese were particularly low, albeit data on these dairy products is limited. It is important to note that comparisons of reported intake of total and specific dairy products across studies are often challenging due to the manner in which dairy can be grouped and/or reported in many studies. For example, in a previous study in Poland, the main reported dairy foods were ‘Milk’, ‘Cheese and cottage cheese’, and ‘Yoghurt and milk drinks’ [ 97 ]. Similarly, a study in America grouped milk, cheese and yogurt into ‘total dairy’, excluding other dairy products [ 98 ]. In Korea, one study analysed the national data (from 2007–2009) and defined dairy products as a ‘combination of milk and yogurt’, without cheese being included, due to the extremely low consumption of cheese [ 99 ]. With such differences in the definition of dairy and grouping of dairy foods, caution must be given to comparisons across studies, since the intakes of dairy are dependent on the definition used within each study. To overcome these issues, the present review also reported on individual dairy foods when possible.

In terms of the individual dairy foods consumed, this review showed that milk was the largest contributor to dairy consumption in China, similar to other countries such as Australia [ 100 ] and Spain [ 101 ]. The present review also found that intake of yogurt was the second highest of dairy consumption, with younger and more educated consumers purchasing more yogurt than others [ 38 ]. This is different to intakes reported in other countries, where for example yogurt and fermented milk consumed among people aged 18–64 years in Spain, was less than older adults (64–75 years) [ 102 ]. In addition, data from the National Health and Nutrition Examination survey 1999–2004 in the US showed that consumption of cheese instead of yogurt ranked second among adults [ 103 ]. In contrast to western countries, we found that the consumption of cheese and butter was exceedingly low and was hardly examined in reported dairy intakes in China. One possible reason is that cheese and butter are relatively new to the market, and mostly imported, which may lead to the higher price than milk and other dairy products [ 44 ]. This may go some way to explain why consumers of these products are mostly limited to the younger and wealthier population [ 44 ]. However, more work is needed to fully understand this finding. Significant differences in the consumption of milk powder were also noticeable in the papers reported in this review. Within three identified studies reporting milk powder consumption in different survey years and locations, and among different age groups, we found that milk powder played a particular role in the diet of the Chinese population. Evidence showed that milk powder was consumed by many older adults. Before the purchase of milk and yogurt became convenient and modern refrigeration availability improved, milk powder was the most practical dairy product for consumption in China [ 38 ].

This review identified 16 papers that reported differences in dairy intakes across sex groups. Most of the available evidence showed the females had higher intakes of dairy foods than males, although not all the studies reported or conducted statistical analysis. The association between gender and dairy consumption was also observed in other recent studies examining dietary intakes in Europe [ 104 , 105 ]. One study evaluated dairy intake pattern in older adults across Europe including 16 European countries, and reported that males had lower intakes of dairy than females [ 104 ]. In addition, Pellay et al. [ 105 ] analysed the socio-demographic characteristics and dietary intake among the elderly in France, finding that women were more likely to have the highest frequency of consumption of dairy foods, including milk and fresh dairy products, which also indicates that sex was a factor associated with dairy consumption. Sex has been noted as a factor which is related to dietary habits. A previous study of dietary status in China found that male participants had significantly higher consumption of vegetables, cereal, meat and legumes than females [ 47 ]. Interestingly, there was one study that reported higher dairy consumption in males than females and found that more males met the recommended intake of dairy, but these differences were not found to be significant. Since this paper didn’t give additional details of the two sex groups, we were not able to identify the reason for this result [ 55 ]. The factors that contributed to the difference of dairy consumption in females and males still need to be further investigated, but it’s clear that sex differences exist in dairy consumption in China. It is also important to note that the results in the included papers were not energy-adjusted. Therefore with the findings showing that females tend to consume higher amount of dairy than males, this need to be taken into consideration.

Associations between different regions and dairy consumption in China are considered in this present review. Based on the available papers’ comparisons across different location sub-groups including urban v rural, north south east and west, costal vs. inland, and size of city were examined. One of the main findings was that people living in urban areas had a significantly higher consumption of dairy than those living in rural areas, and this gap appears to have existed for a long time period. For example, data from a national survey in 2002 reported that the mean dairy intakes among urban residents were 65.8 g/d, whereas the amount in rural was only 11.4 g/d [ 67 ]. More recently, in 2011, the dairy intake in urban population was 52.52 ± 115.47 g/d while it was only 8.53 ± 43.38 g/d in rural area [ 63 ], suggesting no change in either of these areas. Similarly, people living in a large or even a small size city had a much higher consumption of dairy compared to those in rural areas. There are many possible reasons behind these findings such as differences in income, education level and convenience [ 38 ], which need to be explored further. People living in urban areas usually have higher incomes and are more likely to have higher education, which may have contributed to the rapid increase in consumption of dairy [ 44 ]. More supermarkets and therefore, availability of dairy products in urban areas means more choice and availability of high-quality dairy products for these population groups, which may have contributed to this difference [ 106 ]. In addition, lack of knowledge of the importance or impact of dairy products on health (or risk of disease) may also be a contributor to low dairy consumption behavior in people living in rural area [ 107 ]. The evidence also demonstrated that northern and costal populations consumed more dairy than those living in southern areas and inland cities. Compared to eastern and central regions, people living in western cities had lower dairy consumption. These differences might be due to the difference in geographic environment, food resources, social culture, and economic disparities in these regions [ 71 ]. For example, coastal and northern cities were opened to foreigners in the nineteenth century, and evidence has shown that greater exposure to western culture had a positive influence on dairy consumption [ 108 ]. Therefore, the impact of western culture on dietary patterns in those regions could be in part responsible for these differences.

Knowledge of these differences in the amount (and type) of dairy products consumed across regions, sex and age groups are of importance, as it is known that the type, and amount of dairy products consumed, can have different effects on human health [ 109 , 110 ]. Dairy foods vary considerably in their nutrient compositions [ 109 ] and, evidence shows that health effects are substantially modified by the food matrix. For example, one previous study found that, dairy fat consumed in the matrix of cheese resulted in significantly lower low density lipoprotein (LDL) and total cholesterol compared with the same components eaten in the matrix of butter [ 110 ]. Many of the studies identified in the present review only considered the consumption of total dairy. The studies which did examine individual dairy foods reported considerable differences in consumption of these products within China, which merits further investigation [ 38 , 90 , 94 ]. We would therefore recommend that future studies capture and report details of intakes of individual dairy foods. Although dairy intakes in China have increased greatly [ 47 ], much of the data was old and more recent data was not found in published papers. With the constant change in dietary habits and more choices within food products within China, such as non-dairy plant-based milk alternatives, which are being adopted by a growing number of consumers, it is possible that a reduction of some dairy products in the Chinese population may be observed.

Whilst this review comprehensively examined the available literature, due to the complexities in reporting discussed previously, and the limited number of papers for the question being considered, the findings reported here are limited and merit further investigation. This review only presented the findings from existing comparison within the studies, therefore no analysis was conducted to compare across the studies. And there might be some published studies not identified for inclusion in this review due to the search terms used in our search. Furthermore, although limited to papers published since 2000, many of the studies use older datasets, and it is likely that dairy intakes have changed considerably and work on more recently collected data is needed. Therefore, there is a need for a detailed analysis of more recent intake data, to determine if the trends reported here are a true reflection of the current status. In addition, in this present review, we only focused on the influence of key factors - age, gender and regions which were most frequently studied and reported in published studies to investigate the difference in dairy consumption in the population group. Many other factors could be examine in future reviews.

Regardless of these limitations, this review demonstrates clear differences in consumption of different types of dairy products, and in population groups (such as males and females, age groups, urban and rural residents). When considering incorporation of dairy consumption into healthy guidelines, it is important to note these differences, and adapt recommendations and promotions accordingly. Furthermore, more detail on how dairy is specifically consumed within the diet is needed, which would support further development of nutrition recommendations through modelling scenarios for differing population groups.

This review has shown deviations in dairy intake across different population groups in China, including age, sex, and geographic location as well as across the different types of dairy products. The main findings of this review demonstrate that middle-aged adults tend to consume less dairy than other age groups, females in generally had higher intakes of dairy foods than males, and that milk and yogurt and milk powder are the main types of dairy products consumed in China. Whilst this review highlighted some novel and interesting findings, it also highlights a detailed lack of understanding of the use of dairy within the diet, and differences in the dairy consumption among different population groups.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

BC. The contribution of dairy products to micronutrient intakes in France. J Am Coll Nutr. 2011;30(5 Suppl 1):410s–4. https://doi.org/10.1080/07315724.2011.10719984 .

Vissers PA, Streppel MT, Feskens EJ, de Groot LC. The contribution of dairy products to micronutrient intake in the Netherlands. J Am Coll Nutr. 2011;30(5 Suppl 1):415s–421. https://doi.org/10.1080/07315724.2011.10719985 .

Article   CAS   PubMed   Google Scholar  

Gaucheron F. Milk and dairy products: a unique micronutrient combination. J Am Coll Nutr. 2011;30(5 Suppl 1):400s–409. https://doi.org/10.1080/07315724.2011.10719983 .

Tunick MH, Van Hekken DL. Dairy products and health: recent insights. J Agric Food Chem. 2015;63(43):9381–8. https://doi.org/10.1021/jf5042454 .

Schwingshackl L, Hoffmann G, Schwedhelm C, Kalle-Uhlmann T, Missbach B, Knüppel S, Boeing H. Consumption of dairy products in relation to changes in anthropometric variables in adult populations: a systematic review and Meta-analysis of Cohort studies. PLoS ONE. 2016;11(6): e0157461. https://doi.org/10.1371/journal.pone.0157461 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Rozenberg S, Body JJ, Bruyère O, Bergmann P, Brandi ML, Cooper C, Devogelaer JP, Gielen E, Goemaere S, Kaufman JM, Rizzoli R, Reginster JY. Effects of dairy products consumption on Health: benefits and Beliefs–A commentary from the Belgian bone Club and the European Society for Clinical and Economic Aspects of Osteoporosis, Osteoarthritis and Musculoskeletal Diseases. Calcif Tissue Int. 2016;98(1):1–17. https://doi.org/10.1007/s00223-015-0062-x .

Dugan CE, Fernandez ML. Effects of dairy on metabolic syndrome parameters: a review. Yale J Biol Med. 2014;87(2):135–47.

CAS   PubMed   PubMed Central   Google Scholar  

Wang H, Fox CS, Troy LM, McKeown NM, Jacques PF. Longitudinal association of dairy consumption with the changes in blood pressure and the risk of incident Hypertension: the Framingham Heart Study. Br J Nutr. 2015;114(11):1887–99. https://doi.org/10.1017/s0007114515003578 .

Wang H, Chen L, Cao Y, Xie K, Wang C, Pei P, Guo Y, Bragg F, Yu M, Chen Z, Li L. Association between frequency of dairy product consumption and Hypertension: a cross-sectional study in Zhejiang Province, China. Nutr Metab (Lond). 2022;19(1):67. https://doi.org/10.1186/s12986-022-00703-2 .

Article   PubMed   Google Scholar  

Lu Y, Wiltshire HD, Baker JS, Wang Q, Ying S. Associations between dairy consumption, physical activity, and blood pressure in Chinese young women. Front Nutr. 2023;10: 1013503. https://doi.org/10.3389/fnut.2023.1013503 .

Song J, Pan C, Li F, Guo Y, Pei P, Tian X, Wang S, Gao R, Pang Z, Chen Z, Li L. Association between dairy consumption and Ischemic Heart Disease among Chinese adults: a prospective study in Qingdao. Nutr Metabolism. 2022;19(1):11. https://doi.org/10.1186/s12986-022-00645-9 .

Article   CAS   Google Scholar  

Chen GC, Szeto IM, Chen LH, Han SF, Li YJ, van Hekezen R, Qin LQ. Dairy products consumption and metabolic syndrome in adults: systematic review and meta-analysis of observational studies. Sci Rep. 2015;5: 14606. https://doi.org/10.1038/srep14606 .

Lee KW, Cho W. The consumption of dairy products is Associated with reduced risks of obesity and metabolic syndrome in Korean women but not in men. Nutrients. 2017;9(6):630. https://doi.org/10.3390/nu9060630 .

Crichton GE, Bogucki OE, Elias MF. Dairy food intake, diet patterns, and health: findings from the Maine-Syracuse Longitudinal Study. Int Dairy J. 2019;91:64–70. https://doi.org/10.1016/j.idairyj.2018.12.009 .

Article   PubMed   PubMed Central   Google Scholar  

Crichton GE, Alkerwi A. Dairy food intake is positively associated with cardiovascular health: findings from Observation of Cardiovascular Risk factors in Luxembourg study. Nutr Res. 2014;34(12):1036–44. https://doi.org/10.1016/j.nutres.2014.04.002 .

Lordan R, Tsoupras A, Mitra B, Zabetakis I. Dairy Fats and Cardiovascular Disease: do we really need to be concerned? Foods. 2018;7(3): 29. https://doi.org/10.3390/foods7030029 .

Umesawa M, Iso H, Ishihara J, Saito I, Kokubo Y, Inoue M, Tsugane S. Dietary calcium intake and risks of Stroke, its subtypes, and coronary Heart Disease in Japanese: the JPHC Study Cohort I. Stroke. 2008;39(9):2449–56. https://doi.org/10.1161/strokeaha.107.512236 .

Kratz M, Baars T, Guyenet S. The relationship between high-fat dairy consumption and obesity, cardiovascular, and metabolic Disease. Eur J Nutr. 2013;52(1):1–24. https://doi.org/10.1007/s00394-012-0418-1 .

Marques-Vidal P, Gonçalves A, Dias CM. Milk intake is inversely related to obesity in men and in young women: data from the Portuguese Health Interview Survey 1998–1999. Int J Obes (Lond). 2006;30(1):88–93. https://doi.org/10.1038/sj.ijo.0803045 .

Elwood PC, Pickering JE, Givens DI, Gallacher JE. The consumption of milk and dairy foods and the incidence of vascular Disease and Diabetes: an overview of the evidence. Lipids. 2010;45(10):925–39. https://doi.org/10.1007/s11745-010-3412-5 .

Elwood PC, Givens DI, Beswick AD, Fehily AM, Pickering JE, Gallacher J. The survival advantage of milk and dairy consumption: an overview of evidence from cohort studies of vascular Diseases, Diabetes and cancer. J Am Coll Nutr. 2008;27(6):723s–734. https://doi.org/10.1080/07315724.2008.10719750 .

Camfield DA, Owen L, Scholey AB, Pipingas A, Stough C. Dairy constituents and neurocognitive health in ageing. Br J Nutr. 2011;106(2):159–74. https://doi.org/10.1017/s0007114511000158 .

Hu FB, Stampfer MJ, Manson JE, Ascherio A, Colditz GA, Speizer FE, Hennekens CH, Willett WC. Dietary saturated fats and their food sources in relation to the risk of coronary Heart Disease in women. Am J Clin Nutr. 1999;70(6):1001–8. https://doi.org/10.1093/ajcn/70.6.1001 .

The national dairy council. https://ndc.ie/nutrition/food-pyramid-and-dairy-guidelines/ . Accessed 11 Oct 2022.

The Dietary Guidelines for Americans., 2020–2025. https://www.dietaryguidelines.gov/resources/2020-2025-dietary-guidelines-online-materials . Accessed 11 Oct 2022.

The Eatwell Guide. National Health Service (NHS). https://www.nhs.uk/live-well/eat-well/food-guidelines-and-food-labels/the-eatwell-guide/ . Accessed 06 June 2023.

Healthy Diet Fact Sheet. World Health Organization (WHO). https://www.who.int/news-room/fact-sheets/detail/healthy-diet . Accessed 06 June 2023.

Sirichakwal PP, Sranacharoenpong K, Tontisirin K. Food based dietary guidelines (FBDGs) development and promotion in Thailand. Asia Pac J Clin Nutr. 2011;20(3):477–83.

PubMed   Google Scholar  

Eating and activity guidelines. Ministry of Health (New Zealand). https://www.tewhatuora.govt.nz/our-health-system/preventative-healthwellness/nutrition/eating-and-activity-guidelines#current-guidelines . Accessed 06 June 2023.

Dietary Guidelines for Chinese Residents. Chinese Center for Disease Control and Prevention. 2022. https://en.chinacdc.cn/health_topics/nutrition_health/202206/t20220616_259702.html . Accessed 19 Oct 2023.

Rehm CD, Peñalvo JL, Afshin A, Mozaffarian D. Dietary intake among US adults, 1999–2012. JAMA. 2016;315(23):2542–53. https://doi.org/10.1001/jama.2016.7491 .

Zhang Q, Chen X, Liu Z, Varma DS, Wan R, Wan Q, Zhao S. Dietary patterns in relation to general and central obesity among adults in Southwest China. Int J Environ Res Public Health. 2016;13(11):1080. https://doi.org/10.3390/ijerph13111080 .

Zhang R, Wang Z, Fei Y, Zhou B, Zheng S, Wang L, Huang L, Jiang S, Liu Z, Jiang J, Yu Y. The difference in nutrient intakes between Chinese and Mediterranean, Japanese and American diets. Nutrients. 2015;7(6):4661–88. https://doi.org/10.3390/nu7064661 .

Li M, Shi Z. Dietary pattern during 1991–2011 and its Association with Cardio Metabolic Risks in Chinese adults: the China Health and Nutrition Survey. Nutrients. 2017;9(11): 1218. https://doi.org/10.3390/nu9111218 .

Kim WK, Shin D, Song WO. Are dietary patterns Associated with Depression in U.S. adults? J Med Food. 2016;19(11):1074–84. https://doi.org/10.1089/jmf.2016.0043 .

Xu PP, Yang TT, Xu J, Li L, Cao W, Gan Q, Hu XQ, Pan H, Zhao WH, Zhang Q. Dairy Consumption and associations with Nutritional Status of Chinese Children and adolescents. Biomed Environ Sci. 2019;32(6):393–405. https://doi.org/10.3967/bes2019.054 .

Wei J, Wang J. Chinese residents’ knowledge about and behavior towards dairy products: a cross-sectional study. BMC Public Health. 2023;23(1):374. https://doi.org/10.1186/s12889-023-15254-1 .

Fuller F, Beghin J, Rozelle S. Consumption of dairy products in urban China: results from Beijing, Shangai and Guangzhou. Australian J Agricultural Resource Econ. 2007;51(4):459–74. https://doi.org/10.1111/j.1467-8489.2007.00379.x .

Article   Google Scholar  

Yuan M, Seale JL Jr, Wahl T, Bai J. The changing dietary patterns and health issues in China. China Agricultural Economic Review. 2019;11(1):143–59.

He Y, Zhai F, Wang Z, Hu Y. Status of dietary calcium intake of Chinese residents. Wei Sheng Yan Jiu. 2007;36(5):600–2.

CAS   PubMed   Google Scholar  

Feeney EL, Nugent AP, Mc Nulty B, Walton J, Flynn A, Gibney ER. An overview of the contribution of dairy and cheese intakes to nutrient intakes in the Irish diet: results from the National Adult Nutrition Survey. Br J Nutr. 2016;115(4):709–17.

Górska-Warsewicz H, Rejman K, Laskowski W, Czeczotko M. Milk and dairy products and their nutritional contribution to the average Polish diet. Nutrients. 2019;11(8): 1771.

Liu X, Zhao X, Xu L. Food sources of calcium and iron in the diet of Beijing elderly. Wei Sheng Yan Jiu. 2004;33(3):336–8.

Fuller F, Huang JK, Ma HY, Rozelle S. Got milk? The rapid rise of China’s dairy sector and its future prospects. Food Policy. 2006;31(3):201–15. https://doi.org/10.1016/j.foodpol.2006.03.002 .

Cheng LL, Yin CB, Chien HP. Demand for milk quantity and safety in urban China: evidence from Beijing and Harbin. Australian J Agricultural Resource Econ. 2015;59(2):275–87. https://doi.org/10.1111/1467-8489.12065 .

He YN, Yang XG, Xia J, Zhao LY, Yang YX. Consumption of meat and dairy products in China: a review. Proceed Nutr Soc. 2016;75(3):385–91. https://doi.org/10.1017/s0029665116000641 .

Tian X, Huang Y, Wang H. Deviation of Chinese Adults’ Diet from the Chinese Food Pagoda 2016 and Its Association with Adiposity. Nutrients. 2017;9(9): 995. https://doi.org/10.3390/nu9090995 .

Wang Q, Parsons R, Zhang G. China’s dairy markets: trends, disparities, and implications for trade. China Agricultural Econ Rev. 2010;2:356.

Cheng R, Wang Q, Wei L. Income growth, employment structure transition and the rise of modern markets: The impact of urbanization on residents’ consumption of dairy products in China. PLoS ONE. 2022;17(4):e0267006. https://doi.org/10.1371/journal.pone.0267006 .

Britten P, Marcoe K, Yamini S, Davis C. Development of food intake patterns for the MyPyramid Food Guidance System. J Nutr Educ Behav. 2006;38(6):78–S92.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71. https://doi.org/10.1136/bmj.n71 .

China Health and Nutrition Survey. https://www.cpc.unc.edu/projects/china . Accessed 11 Oct 2022.

Critical Appraisal Skills Programme. CASP (Checklist for Cohort Study) [online]. 2018. https://casp-uk.net/images/checklist/documents/CASP-Cohort-Study-Checklist/CASP-Cohort-Study-Checklist-2018_fillable_form.pdf . Accessed 11 Oct 2022.

Cheng M, Wang H, Wang Z, Du W, Ouyang Y, Zhang B. Relationship between dietary factors and the number of altered metabolic syndrome components in Chinese adults: a cross-sectional study using data from the China Health and Nutrition Survey. BMJ Open. 2017;7(5): e014911. https://doi.org/10.1136/bmjopen-2016-014911 .

Xu XY, Hall J, Byles J, Shi ZM. Do older Chinese people’s diets meet the Chinese Food Pagoda guidelines? Results from the China Health and Nutrition Survey 2009. Public Health Nutr. 2015;18(16):3020–30. https://doi.org/10.1017/s1368980015000129 .

Zhai F, Wang H, Du S, He Y, Wang Z, Ge K, Popkin BM. Lifespan nutrition and changing socio-economic conditions in China. Asia Pac J Clin Nutr. 2007;16(Suppl 1):374–82.

Shen X, Fang A, He J, Liu Z, Guo M, Gao R, Li K. Trends in dietary fat and fatty acid intakes and related food sources among Chinese adults: a longitudinal study from the China Health and Nutrition Survey (1997–2011). Public Health Nutr. 2017;20(16):2927–36. https://doi.org/10.1017/s1368980017001781 .

Batis C, Sotres-Alvarez D, Gordon-Larsen P, Mendez MA, Adair L, Popkin B. Longitudinal analysis of dietary patterns in Chinese adults from 1991 to 2009. Br J Nutr. 2014;111(8):1441–51. https://doi.org/10.1017/S0007114513003917 .

Ren YJ, Zhang YJ, Loy JP, Glauben T. Food consumption among income classes and its response to changes in income distribution in rural China. China Agricultural Econ Rev. 2018;10(3):406–24. https://doi.org/10.1108/caer-08-2014-0079 .

Song G, Li M, Fullana-i-Palmer P, Williamson D, Wang Y. Dietary changes to mitigate climate change and benefit public health in China. Sci Total Environ. 2017;577:289–98. https://doi.org/10.1016/j.scitotenv.2016.10.184 .

Tian X, Wu M, Zang J, Zhu Y, Wang H. Dietary diversity and adiposity in Chinese men and women: an analysis of four waves of cross-sectional survey data. Eur J Clin Nutr. 2017;71(4):506–11. https://doi.org/10.1038/ejcn.2016.212 .

Huang F, Wang H, Wang Z, Zhang J, Su C, Du W, Jiang H, Jia X, Ouyang Y, Wang Y, Li L, Zhang B. Knowledge, behavior and consumption types of milk and dairy products among the Chinese aged 60 and above in 15 provinces (autonomous regions and municipalities) in 2015. Wei Sheng Yan Jiu. 2019;48(1):9–15.

Wang Y, Jia X, Du W, Wang Z, Wang H, Zhang B. Dairy consumption characteristics among Chinese adult residents from 1989 to 2011. Wei Sheng Yan Jiu. 2017;46(3):409–33.

Na X, Xi Y, Qian S, Zhang J, Yang Y, Zhao A. Association between Dairy Product Intake and risk of fracture among adults: a Cohort Study from China Health and Nutrition Survey. Nutrients. 2022;14(8): 1632. https://doi.org/10.3390/nu14081632 .

Wang L, Ouyang Y, Jiang H, Zhang B, Wang H, Zhang J, Du W, Niu R, Wang Z. Secular trends in food intakes among the elderly aged 60 and older in nine provinces in China from 1991 to 2015. Wei Sheng Yan Jiu. 2022;51(1):24–31. https://doi.org/10.19813/j.cnki.weishengyanjiu.2022.01.005 .

Fu WG, Gandhi VP, Cao LJ, Liu HB, Zhou ZY. Rising consumption of Animal products in China and India: National and Global implications. Chin World Econ. 2012;20(3):88–106. https://doi.org/10.1111/j.1749-124X.2012.01289.x .

Li G, Sun GX, Williams PN, Nunes L, Zhu YG. Inorganic arsenic in Chinese food and its cancer risk. Environ Int. 2011;37(7):1219–25. https://doi.org/10.1016/j.envint.2011.05.007 .

Zhang XY, Dagevos H, He YN, van der Lans I, Zhai FY. Consumption and corpulence in China - A consumer segmentation study based on the food perspective. Food Policy. 2008;33(1):37–47. https://doi.org/10.1016/j.foodpol.2007.06.002 .

Liu Z, Pang S, Li Y, Man Q, Li L, Zhang J. Consumption status of dairy products in Chinese aged 60 and above in 2010–2012. Wei Sheng Yan Jiu. 2016;45(5):708–13.

Liu Z, Pang SJ, Man QQ, Wang JZ, Zhao WH, Zhang J. Prevalence of Undernutrition and related dietary factors among people aged 75 years or older in China during 2010–2012. Biomed Environ Sci: BES. 2018;31(6):425–37. https://doi.org/10.3967/bes2018.056 .

Yin P, Zhang M, Li YC, Chen XR, Wang LM, Jiang Y, Bao HL, Zhao WH. Survey of dietary intake of Chinese adults in 2010. Zhonghua Yu Fang Yi Xue Za Zhi. 2012;46(8):692–6.

He M, Guo Z, Lu Z, Wei S, Wang Z. High milk consumption is associated with carotid Atherosclerosis in middle and old-aged Chinese. Int J Cardiology Hypertens. 2020;5:100031. https://doi.org/10.1016/j.ijchy.2020.100031 .

Ba L, Zhao J, Li S. Milk consumption pattens among residents in beijing. Ann Nutr Metab. 2013;63:1763. https://doi.org/10.1159/000354245 .

Tang J, Huang Z, Pan XD. Exposure assessment of heavy metals (cd, hg, and pb) by the intake of local foods from Zhejiang, China. Environ Geochem Health. 2014;36(4):765–71. https://doi.org/10.1007/s10653-014-9594-7 .

Fu Z, Liu J, Liu H, Jin D. Dietary patterns of urban residents from 1982 to 2012 in Hunan. Zhong Nan Da Xue Xue Bao Yi Xue Ban. 2014;39(7):713–7. https://doi.org/10.11817/j.issn.1672-7347.2014.07.011 .

Zou Y, Zhang R, Zhou B, Huang L, Chen J, Gu F, Zhang H, Fang Y, Ding G. A comparison study on the prevalence of obesity and its associated factors among city, township and rural area adults in China. BMJ Open. 2015;5(7): e008417. https://doi.org/10.1136/bmjopen-2015-008417 .

Zhao X, Yuan L, Feng L, Xi Y, Yu H, Ma W, Zhang D, Xiao R. Association of dietary intake and lifestyle pattern with mild cognitive impairment in the elderly. J Nutr Health Aging. 2015;19(2):164–8. https://doi.org/10.1007/s12603-014-0524-2 .

Sun Y, Jiang C, Cheng KK, Zhang W, Leung GM, Lam TH, Schooling CM. Milk consumption and cardiovascular risk factors in older Chinese: the Guangzhou Biobank Cohort Study. PLoS ONE. 2014;9(1): e84813. https://doi.org/10.1371/journal.pone.0084813 .

Zong G, Sun Q, Yu D, Zhu J, Sun L, Ye X, Li H, Jin Q, Zheng H, Hu FB, Lin X. Dairy consumption, type 2 Diabetes, and changes in cardiometabolic traits: a prospective cohort study of middle-aged and older Chinese in beijing and Shanghai. Diabetes Care. 2014;37(1):56–63. https://doi.org/10.2337/dc13-0975 .

Zhang X, Fang YJ, Feng XL, Abulimiti A, Huang CY, Luo H, Zhang NQ, Chen YM, Zhang CX. Higher intakes of dietary vitamin D, calcium and dairy products are inversely associated with the risk of Colorectal cancer: a case-control study in China. Br J Nutr. 2020;123(6):699–711. https://doi.org/10.1017/S000711451900326X .

Boyapati SM, Shu XO, Jin F, Dai Q, Ruan Z, Gao YT, Zheng W. Dietary calcium intake and Breast cancer risk among Chinese women in Shanghai. Nutr Cancer. 2003;46(1):38–43. https://doi.org/10.1207/s15327914nc4601_05 .

Bao PP, Shu XO, Zheng Y, Cai H, Ruan ZX, Gu K, Su Y, Gao YT, Zheng W, Lu W. Fruit, vegetable, and animal food intake and Breast cancer risk by hormone receptor status. Nutr Cancer. 2012;64(6):806–19. https://doi.org/10.1080/01635581.2012.707277 .

Wang XJ, Jiang CQ, Zhang WS, Zhu F, Jin YL, Woo J, Cheng KK, Lam TH, Xu L. Milk consumption and risk of mortality from all-cause, Cardiovascular Disease and cancer in older people. Clin Nutr. 2020;39(11):3442–51. https://doi.org/10.1016/j.clnu.2020.03.003 .

Zhou C, Li M, Liu L, Zhao F, Cong W, Zhang F. Food consumption and dietary patterns of local adults living on the Tibetan Plateau: results from 14 countries along the Yarlung Tsangpo River. Nutrients. 2021;13(7): 2444. https://doi.org/10.3390/nu13072444 .

Song PK, Li C, Feng GY, Zhang J, Li L, Yang TT, Dong MN, Liu AL, Zhang Q, Liu YJ, Wang JL, Ge TX, Yu Q, Ding GQ. Health Status and Food Intake frequency of Elderly residents during COVID-19 Community Closure - Jiangxia District, Wuhan City, China, 2020. China Cdc Weekly. 2020;2(27):496–501. https://doi.org/10.46234/ccdcw2020.120 .

Shuai M, Zuo LSY, Miao Z, Gou W, Xu F, Jiang Z, Ling CW, Fu Y, Xiong F, Chen YM, Zheng JS. Multi-omics analyses reveal relationships among dairy consumption, gut microbiota and cardiometabolic health. EBioMedicine. 2021;66:103284. https://doi.org/10.1016/j.ebiom.2021.103284 .

Cheng Z, Shuai P, Qiao Q, Li T. Validity and reliability of a simplified food frequency questionnaire: a cross sectional study among physical health examination adults in southwest region of China. Nutr J. 2020;19(1):114. https://doi.org/10.1186/s12937-020-00630-z .

He P, Lu Y, Liang Y, Chen B, Wu M, Li S, He G, Jin T. Exposure assessment of dietary cadmium: findings from Shanghainese over 40 years, China. BMC Public Health. 2013;13: 590. https://doi.org/10.1186/1471-2458-13-590 .

Bai JF, Wahl TI, McCluskey JJ. Fluid milk consumption in urban Qingdao, China. Australian J Agricultural Resource Econ. 2008;52(2):133–47. https://doi.org/10.1111/j.1467-8489.2008.00401.x .

Qiao G, Guo T, Klein KK. Melamine in Chinese milk products and consumer confidence. Appetite. 2010;55(2):190–5. https://doi.org/10.1016/j.appet.2010.05.047 .

Yang GY, Lin XL, Fang AP, Zhu HL. Eating habits and lifestyles during the initial stage of the COVID-19 Lockdown in China: a cross-sectional study. Nutrients. 2021;13(3): 970. https://doi.org/10.3390/nu13030970 .

Wang X, Lei SM, Le S, Yang Y, Zhang B, Yao W, Gao Z, Cheng S. Bidirectional influence of the COVID-19 pandemic lockdowns on Health behaviors and Quality of Life among Chinese adults. Int J Environ Res Public Health. 2020;17(15): 5575. https://doi.org/10.3390/ijerph17155575 .

Feng RN, Du SS, Chen Y, Li Z, Zhang YF, Sun CH, Jiang YS. An internet-based food frequency questionnaire for a large Chinese population. Asia Pac J Clin Nutr. 2016;25(4):841–8. https://doi.org/10.6133/apjcn.092015.26 .

Song XY, Li R, Guo L, Guo PP, Zhang M, Feng RN. Association between dairy consumption and prevalence of obesity in adult population of northeast China: an internet-based cross-sectional study. Asia Pac J Clin Nutr. 2020;29(1):110–9. https://doi.org/10.6133/apjcn.202003_29(1).0015 .

Guo P, Zhu H, Pan H, Feng R, Chen Y, Wang Y, Wang X, Li Y, Yang L, Zhou B, Wang X, Zhao Y. Dose-response relationships between dairy intake and chronic metabolic Diseases in a Chinese population. J Diabetes. 2019;11(11):846–56. https://doi.org/10.1111/1753-0407.12921 .

Silanikove N, Leitner G, Merin U. The interrelationships between Lactose Intolerance and the modern dairy industry: global perspectives in Evolutional and historical backgrounds. Nutrients. 2015;7(9):7312–31. https://doi.org/10.3390/nu7095340 .

Górska-Warsewicz H, Rejman K, Laskowski W, Czeczotko M. Milk and dairy products and their nutritional contribution to the average Polish Diet. Nutrients. 2019;11(8): 1771. https://doi.org/10.3390/nu11081771 .

Fulgoni V 3, Nicholls J, Reed A, Buckley R, Kafer K, Huth P, DiRienzo D, Miller GD. Dairy consumption and related nutrient intake in African-American adults and children in the United States: continuing survey of food intakes by individuals 1994–1996, 1998, and the National Health and Nutrition Examination Survey 1999–2000. J Am Diet Assoc. 2007;107(2):256–64. https://doi.org/10.1016/j.jada.2006.11.007 .

Lee HJ, Cho JI, Lee HS, Kim CI, Cho E. Intakes of dairy products and calcium and obesity in Korean adults: Korean National Health and Nutrition Examination Surveys (KNHANES) 2007–2009. PLoS ONE. 2014;9(6):e99085. https://doi.org/10.1371/journal.pone.0099085 .

Parker CE, Vivian WJ, Oddy WH, Beilin LJ, Mori TA, O’Sullivan TA. Changes in dairy food and nutrient intakes in Australian adolescents. Nutrients. 2012;4(12):1794–811. https://doi.org/10.3390/nu4121794 .

Machado-Fragua MD, Struijk EA, Caballero FF, Ortolá R, Lana A, Banegas JR, Rodríguez-Artalejo F, Lopez-Garcia E. Dairy consumption and risk of falls in 2 European cohorts of older adults. Clin Nutr. 2020;39(10):3140–6. https://doi.org/10.1016/j.clnu.2020.01.025 .

Partearroyo T, Samaniego-Vaesken ML, Ruiz E, Aranceta-Bartrina J, Gil Á, González-Gross M, Ortega RM, Serra-Majem L, Varela-Moreiras G. Current food consumption amongst the Spanish ANIBES Study Population. Nutrients. 2019;11(11):2663. https://doi.org/10.3390/nu11112663 .

Beydoun MA, Gary TL, Caballero BH, Lawrence RS, Cheskin LJ, Wang Y. Ethnic differences in dairy and related nutrient consumption among US adults and their association with obesity, central obesity, and the metabolic syndrome. Am J Clin Nutr. 2008;87(6):1914–25. https://doi.org/10.1093/ajcn/87.6.1914 .

Ribeiro I, Gomes M, Figueiredo D, Lourenço J, Paúl C, Costa E. Dairy product intake in older adults across Europe based on the SHARE database. J Nutr Gerontol Geriatr. 2019;38(3):297–306. https://doi.org/10.1080/21551197.2019.1627972 .

Pellay H, Marmonier C, Samieri C, Feart C. Socio-demographic characteristics, Dietary, and nutritional intakes of French Elderly Community dwellers according to their dairy product consumption: data from the three-city cohort. Nutrients. 2020;12(11): 3418. https://doi.org/10.3390/nu12113418 .

Huang Y, Wang H, Tian X. Changing diet quality in China during 2004–2011. Int J Environ Res Public Health. 2017;14(1):13.

Zhao A, Szeto IMY, Wang Y, Li C, Pan M, Li T, Wang PY, Zhang YM. Knowledge, attitude, and practice (KAP) of dairy products in Chinese Urban Population and the effects on dairy intake quality. Nutrients. 2017;9(7):668. https://doi.org/10.3390/nu9070668 .

Article   PubMed Central   Google Scholar  

Fuller FH, Beghin JC. China’s growing market for dairy products. Iowa Ag Review. 2015;10(3):Article 5.

Google Scholar  

Kim D, Kim J. Dairy consumption is associated with a lower incidence of the metabolic syndrome in middle-aged and older Korean adults: the Korean Genome and Epidemiology Study (KoGES). Br J Nutr. 2017;117(1):148–60. https://doi.org/10.1017/s000711451600444x .

Feeney EL, Barron R, Dible V, Hamilton Z, Power Y, Tanner L, Flynn C, Bouchier P, Beresford T, Noronha N, Gibney ER. Dairy matrix effects: response to consumption of dairy fat differs when eaten within the cheese matrix-a randomized controlled trial. Am J Clin Nutr. 2018;108(4):667–74. https://doi.org/10.1093/ajcn/nqy146 .

Download references

Acknowledgements

Not applicable.

This work was supported by Food for Health Ireland and China Scholarship Council. The funding bodies had no role in the decision to publish. S.Y. is funded by Food for Health Ireland which is a research organisation that receives funding from Enterprise Ireland, grant number TC20180025, and from members of the Irish dairy industry, and funded by the China Scholarship Council.

Author information

Authors and affiliations.

Food for Health Ireland, University College Dublin, Dublin 4, Republic of Ireland

Shuhua Yang, Nupur Bhargava, Aileen O’Connor, Eileen R. Gibney & Emma L. Feeney

Institute of Food and Health, University College Dublin, Dublin 4, Republic of Ireland

School of Agriculture and Food Science, University College Dublin, Dublin 4, Republic of Ireland

Shuhua Yang, Eileen R. Gibney & Emma L. Feeney

You can also search for this author in PubMed   Google Scholar

Contributions

E.R.G., E.L.F. and S.Y. designed the study. S.Y. and N.B. carried out the literature search and screening, and S.Y., N.B., and E.R.G. reviewed articles for inclusion. S.Y. drafted the paper. E.R.G., E.L.F., N.B. and A.O’C. contributed to writing the paper.

Corresponding author

Correspondence to Emma L. Feeney .

Ethics declarations

Ethics approval and consent to participate.

In this present review, 40 papers reported data from national surveys, with existing ethical approval, or specifically reported ethical approval for the analysis presented. 2 papers reported to be conducted in a sub-sample of Household Income and Expenditure Survey (HIES), and as such would be covered by ethical approval within the original study, although this was not explicitly reported. 2 studies both appear to have conducted market research surveys, which did not seek ethical approval, but received permission from the retailer to administer questionnaires to customers. Finally, 3 studies did not report any details on ethical approval.

Consent for publication

Competing interests.

E.R.G. and E.L.F. and A.O’C. have previously received travel expenses and speaking honoraria from the National Dairy Council, UK. E.R.G. and E.L.F. have received research funding through the Food for Health Ireland project, funded by Enterprise Ireland, grant number TC20180025. The funders had no role in the analyses or interpretation of data; in the writing of the manuscript or in the decision to publish the findings. The other two authors(S.Y. and N.B.) do not have competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Yang, S., Bhargava, N., O’Connor, A. et al. Dairy consumption in adults in China: a systematic review. BMC Nutr 9 , 116 (2023). https://doi.org/10.1186/s40795-023-00781-2

Download citation

Received : 20 October 2022

Accepted : 12 October 2023

Published : 21 October 2023

DOI : https://doi.org/10.1186/s40795-023-00781-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Dairy consumption
• Systematic review

BMC Nutrition

ISSN: 2055-0928

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 23 October 2023

Behavioural economic interventions to reduce health care appointment non-attendance: a systematic review and meta-analysis

• Kalin Werner   ORCID: orcid.org/0000-0001-7759-5397 1 ,
• Sara Abdulrahman Alsuhaibani 2 , 3 ,
• Reem F. Alsukait 4 , 5 ,
• Reem Alshehri 2 ,
• Christopher H. Herbst 5 ,
• Mohammed Alhajji 2 , 6 &
• Tracy Kuo Lin 1  

BMC Health Services Research volume  23 , Article number:  1136 ( 2023 ) Cite this article

276 Accesses

Metrics details

Appointment non-attendance – often referred to as “missed appointments”, “patient no-show”, or “did not attend (DNA)” – causes volatility in health systems around the world. Of the different approaches that can be adopted to reduce patient non-attendance, behavioural economics-oriented mechanisms (i.e., psychological, cognitive, emotional, and social factors that may impact individual decisions) are reasoned to be better suited in such contexts – where the need is to persuade, nudge, and/ or incentivize patients to honour their scheduled appointment. The aim of this systematic literature review is to identify and summarize the published evidence on the use and effectiveness of behavioural economic interventions to reduce no-shows for health care appointments.

We systematically searched four databases (PubMed/Medline, Embase, Scopus, and Web of Science) for published and grey literature on behavioural economic strategies to reduce no-shows for health care appointments. Eligible studies met four criteria for inclusion; they were (1) available in English, Spanish, or French, (2) assessed behavioural economics interventions, (3) objectively measured a behavioural outcome (as opposed to attitudes or preferences), and (4) used a randomized and controlled or quasi-experimental study design.

Our initial search of the five databases identified 1,225 articles. After screening studies for inclusion criteria and assessing risk of bias, 61 studies were included in our final analysis. Data was extracted using a predefined 19-item extraction matrix. All studies assessed ambulatory or outpatient care services, although a variety of hospital departments or appointment types. The most common behaviour change intervention assessed was the use of reminders (n = 56). Results were mixed regarding the most effective methods of delivering reminders. There is significant evidence supporting the effectiveness of reminders (either by SMS, telephone, or mail) across various settings. However, there is a lack of evidence regarding alternative interventions and efforts to address other heuristics, leaving a majority of behavioural economic approaches unused and unassessed.

The studies in our review reflect a lack of diversity in intervention approaches but point to the effectiveness of reminder systems in reducing no-show rates across a variety of medical departments. We recommend future studies to test alternative behavioural economic interventions that have not been used, tested, and/or published before.

Peer Review reports

Introduction

Appointment non-attendance – often referred to as “missed appointment”, “patient no-show”, or “did not attend (DNA)” – causes volatility in health systems around the world. Empty and unfilled timeslots, which could otherwise be used if patients were to show up for appointments as scheduled, lead to unnecessary staffing expenses and revenue losses. Failure to attend scheduled appointments may also contribute to inefficient use of limited health care resources, worsening patient access and healthcare quality. For example, the National Health Service (NHS) in the United Kingdom has reported that patients who miss their general practitioner appointments alone cost the NHS around £216 million a year [ 1 ]. The cost and concern regarding patient no-show is so tremendous that introducing a fine for NHS patient no show became a salient issue during the 2022 Conservative party leadership bid [ 2 ].

Addressing the issue of non-attendance is essential to improving access and safeguarding limited health care resources. Furthermore, it may serve to reduce disparity in healthcare amongst minorities and patients with major mental illness and medically complex care whom have an increased likelihood of missed appointments. [ 3 ] Many studies have identified predictors of non-attendance [ 4 , 5 ], and many of these predictors characterize the issues patients are likely to experience, and as a consequence, are more likely to miss an appointment, because of their sociodemographic characteristics. As such, it is critical to focus on ways that may mitigate issues related to non-attendance.

The problem of non-attendance can be attributed to numerous reasons including physical barriers to access (e.g., lack of affordable transportation [ 6 ], absence of childcare [ 7 ]), opportunity cost (e.g., the time required to seek care), and patient forgetfulness [ 8 ]. Moreover, behavioural science indicates that often patients do not behave in the way we would expect, and that behavioural factors such as limited attention, cognitive overload, and avoidance can impede timely care seeking and influence motivation to honour appointments. For example, in some circumstances feelings of fatalism and fear of negative outcomes have been found to act as a barrier to patients attendance of health screenings [ 9 ]. By understanding the psychological, emotional, cognitive, social factors that may influence patients’ decisions, behavioural insights can be applied to health system planning and guide policy design around appointment attendance.

Behavioural economic informed interventions are defined as an intervention designed to change behaviour within a decision context by counteracting psychological and cognitive biases or leveraging them for better decision making [ 10 , 11 ]. Individuals may satisfice and choose sub-optimal options that may be against their own best interest. Behavioural economic insights can contribute to developing policies which encourage or guide behaviour without limiting free choice [ 12 , 13 ]. Of the different approaches to reduce patient non-attendance, it can be reasoned that behavioural economics insights are best suited to understand how choice problems are optimized or solved to motivate patients to attend their scheduled appointment. Strategies to circumvent these barriers could modify choice architecture by directly removing the physical barriers (e.g., provide free transportation) and making it easier for patients to attend their appointments. Concomitantly, interventions may leverage behavioural economics theory and mechanisms to encourage individuals towards the desired behaviour (i.e., honour appointments).

Ways to encourage, persuade, and/or nudge patients to honour their scheduled appointments include providing resources to circumvent barriers to access (e.g., arranging transportation to healthcare clinics), providing information by reminding patients about their appointment (e.g., text message reminders), and financial incentives (e.g., a gift card for attending scheduled appointment). There is some evidence to suggest that even subtle encouragement that minimizes attentional biases, such as modified Short Message Service (SMS) reminders with details of the cost of missed appointments, can have meaningful impacts on behaviour [ 14 ].

Existing systematic reviews have indicated that the range of interventions proposed to reduce non-attendance all have a modest effect, but fail to summarize the key behavioural mechanism that impacts patient decision-making [ 15 , 16 ]. In particular, when examining the evidence on the expected effect of the use of economic incentives, defined by a material gain or loss, remains sparse and mixed [ 17 , 18 ]. The aim of this systematic literature review is to focus on and summarize the published evidence on the use and effectiveness of behavioural economics-related interventions to reduce no-shows for health care appointments; these studies may focus on one behavioural economic interventions, combine an behavioural economic intervention with a none behavioural economic intervention, or utilize an intervention to change behaviour by leveraging rationale from behavioural economics. The findings will contribute to evidence-based policymaking regarding interventions to reduce non-attendance and inform the development of future interventions in the healthcare sector.

Study design

We conducted a review following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting standards and registered with PROSPERO (CRD42022320844) [ 19 ]. We systematically searched four databases (PubMed/Medline, Embase, Scopus, and Web of Science) for published and grey literature on behavioural economic strategies to reduce no-shows for health care appointments. The authors used a combination of MeSH and text word searches to develop search strings to cover the following concepts: (1) patients scheduling and appointments, (2) no-show patients, and (3) behavioural economics. The complete search strategy is available in Supplementary 1, and an example of terms used in our search is provided in Box 1.

Eligibility criteria

Eligible studies met four criteria for inclusion; they were (1) available in English, Spanish, or French, (2) assessed behavioural economics interventions, (3) objectively measured a behavioural outcome (as opposed to attitudes or preferences), and (4) used a randomized and controlled or quasi-experimental study design to enable causal inference. Studies which were non-randomized controlled study designs needed to control for relevant patient and care setting characteristics to be considered in our review. Since patient non-attendance is not a novel issue, our search did not limit based on date, and we aimed to capture all potential literature on the topic. Conference abstracts, posters, or protocols were excluded from the review. Although the systematic reviews did not fall within our inclusion criteria, we searched the references lists of topical reviews for any additional relevant studies. Duplicate removal, voting consensus, and extraction was conducted using Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org ). For relevance assessment, two of three reviewers used the inclusion criteria and independently assessed each study for relevance, first by title and abstract and subsequently by screening available full texts. We followed definitions of key terms in determining the eligibility of studies (Table  1 ). We did not consider switching care services from in-person to telehealth virtual care appointments, as behavioural economic interventions. However, we included studies which integrated the use of telehealth services, such as online appointment management services, to existing in-person appointment services as a means to reduce opportunity costs for patients. Reviewers checked all within-publication references to identify additional sources. As a desk-based review, no ethical approval was sought.

Risk of bias assessment

A single reviewer assessed each study for risk of bias using the Mixed Methods Appraisal Tool (MMAT) [ 21 ]. The tool uses seven criteria based on study design for items ranging from clear research question, appropriate randomization, representativeness of target population, and confounders being accounted for in design and analysis. Scores of this assessment can be found in Appendix 1. Studies that achieved lower than 70% (4 out of 7) of the MMAT list of the appropriate study design were considered at high risk of bias and poor quality and, therefore, excluded from our review.

Data extraction

Studies that met our inclusion criteria were then divided amongst the team of three reviewers for data extraction, using a predefined 19-item extraction matrix. The following variables were extracted; the aim of the study, the study setting/country, study design, description of population and total n, targeted hospital department or appointment type, intervention, time frame and frequency of intervention, mechanism of intervention, financial or non-financial intervention, comparators, outcomes measured, results, effect-size and confidence interval (either reported as the odds ratio or co-efficient). Summary of the attributes of included studies is presented in Appendix 2. Two reviewers discussed the variables of each study extracted using the matrix for final consensus.

Data analysis

Study outcomes were first summarised in a narrative synthesis. Where it was appropriate to combine studies, such as the use of SMS texts as reminders, meta-analyses were conducted.

Only studies for which reported odds ratios, 95% confidence intervals, and study population size were included in the meta-analysis. We used a random-effects model, assuming that study effect sizes differ, to estimate the mean distribution of affects. This approach allows for a more equal weighting than fixed-effects models [ 22 ]. Heterogeneity was examined using Cochran’s Q test and Higgin’s I2 test. Weighted effect sizes for each study were calculated using inverse-variance weight and plotted using Microsoft Excel (2022).

Our initial search of the five databases identified 1,225 articles. Duplicates were removed, after which 981 titles and abstracts were screened by reviewers. We identified 723 studies for removal. The remaining 258 articles then underwent full-text review. Of these studies, 198 studies were excluded – 71 were published as abstracts only, 79 used a study design that did not meet our inclusion criterion (not randomized, controlled or quasi-experimental), 14 studies measured the irrelevant outcomes (not an objectively measured behavioural outcome). The full text was unavailable for 21 studies. Lastly, 12 studies were excluded due to poor quality (falling below the 70% threshold) as identified by the risk of bias assessment. The final analysis included 61 studies. Figure  1 presents a summary of our results.

PRISMA Flow Diagram of Screening Process

Results are first reported by descriptive study characteristics. Using details and phrases in included papers, studies are then categorized by the mechanisms through which the interventions seek to modify patient behaviour; reminders, incentives/disincentives or other. This section then summarizes the comparative efficiency and characteristics of each delivery mode.

Study characteristics

Full study characteristics of included articles are reported in Appendix 2. A majority of the studies were conducted in and used data from the United States (n = 28). Seven studies used data from the United Kingdom and five from Australia. Three studies were conducted in Switzerland, and another three were conducted in Malaysia. Two studies used data from Saudi Arabia. Two studies originated from Scotland specifically (not covering other regions in the United Kingdom). One study was conducted in and focused on Hong Kong specifically. Lastly, there was a single study from each of the following countries: Brazil, Cameroon, Canada, China, Denmark, Israel, South Korea, Nigeria, Pakistan, and the Netherlands.

The most commonly used study designs were randomised control study design (n = 41) and non-randomized experimental design (n = 15). The remaining studies used quasi-experimental approaches, controlling for variables in their analysis. Two studies utilised a cross-sectional study design, and one study used each of the following designs: cohort study, case-control study, and retrospective observational study.

Most studies included in their study populations anyone attending clinics (n = 43). However, ten studies defined their population as adult outpatient or primary care clinic patients, and seven studies focused on paediatric populations and their caretakers.

All studies assessed ambulatory or outpatient care services, although a variety of hospital departments or appointment types. Studies further focused on the following appointment types; primary care (n = 16), mental health (n = 8), gastrointestinal including patients scheduled for colonoscopies and endoscopies (n = 5), dental (n = 3), radiology/ diagnostic imaging (n = 2), addiction (n = 2) and one of each of the following, respiratory, physical therapy, radiation, emergency department, neurological exam, pain centre, eye clinic and paediatric HIV appointments.

Almost all studies in our review assessed non-financial interventions (n = 58). One study evaluated the use of financial penalties [ 23 ], and another used financial rewards to influence patient behaviour [ 24 ].

Behaviour change interventions categories

We categorized the interventions by the mechanisms through which the interventions seek to modify patient behaviour – whether it is by addressing the heuristics or mental shortcuts that lead patients to make irrational or suboptimal choices. A summary of study intervention and effect size, organized by change intervention category is provided in Table  2 . The most common behaviour change intervention assessed was the use of reminders (n = 56). Other categories included the use of incentives and disincentives and language and cultural congruency.

Delivery mode of reminders

Reminders seek to steer patients’ attention towards particular decisions to create behaviour change. Heterogeneity in reminders was explored by categorising the variation between the mode of delivery and timing of reminders.

Studies assessed reminders delivered through three main channels short message system (SMS), text reminders or electronic reminders (n = 26), telephone (n = 13), or physical mail (n = 5). Thirteen studies assessed combinations of interventions, including phone and SMS (n = 8), telephone and mail (n = 4), or all three means of delivery mode (n = 1). One study delivered reminders via an online portal system [ 25 ] and another used clinic signage and appointment reminder cards [ 26 ].

The timing of reminders ranged from two hours to two weeks before a scheduled appointment. Most studies (n = 16) used reminders between one to three days before appointments; 15 studies focused employed short-term reminders (less than one week). Other common time ranges included one week (n = 6) and two weeks (n = 2). Nine studies varied the timing of reminders delivered in their interventions between one day to two weeks. There were eight studies where timing details were either not applicable to the intervention or not made available in the study.

Overall, studies report mixed results on the effect of interventions involving reminders. Mailed reminders were found to increase kept appointments in all five studies [ 27 , 28 , 29 , 30 , 31 ]. A large number of studies (n = 15) found that using SMS reminders significantly reduced non-attendance rates. Simultaneously, four other studies using SMS reminders indicated that SMS reminders did not lead to reductions in non-attendance rates [ 32 , 33 , 34 , 35 ]. In another study, SMS reminders increased the number of unable to attend rates [ 36 ]. The effect size for studies that included odds ratios related to the effect of SMS reminders is reported in Fig.  2 .

Interventions that relied on telephone calls to remind patients of their appointments yielded inconsistent results. One study found that telephone reminders may decrease no-show rates while simultaneously increasing patient cancellation rates [ 37 ]. Concomitantly, other studies found limited (null) effects and small coefficients and odds ratios of telephone reminder interventions on attendance [ 38 , 39 ].

Effect of SMS/reminders on No-show rates

Figure  2 depicts the direction and magnitude of the overall effects on no-show rates across individual studies. An odds ratio above 1 shows that improvement in attendance for the SMS reminded group was greater than the non-reminded group. Using a random-effects meta-analysis model, our pooled summary effect of OR 1.21 (CI 0.41-2.00) indicates a positive effect of SMS reminders. Results of the Cochran’s Q test indicate no significant heterogeneity amongst the studies (Q = 7.5, p = 0.941) further confirmed by a the complementary Higgins I2 test (0%). The plot indicates that most of the reporting odds ratios pointed towards significant improvements in attendance for patients receiving SMS reminders over those who received none.

Characteristics of reminder delivery mode interventions

A subgroup of these studies compared different ways to deliver reminders (e.g., SMS text compared to telephone, or automated telephone calls compared to calls provided by clinical staff) and reminder efficiency (n = 13).

Results were mixed regarding the most effective methods of delivering reminders. Although no-show rates did not vary significantly between specific reminders delivery methods (calls, letters, or receiving both), any type of contact was found to decrease no-show rates [ 40 ]. Patients receiving telephone calls were more likely to keep their appointments than those who received postcards [ 41 ]. Similar results were found when comparing phone calls to patients receiving SMS reminders [ 42 , 43 , 44 , 45 ]. Combining multiple methods of reminders, such as both text and calls, were found to be most effective under specific circumstances [ 46 ]. Hallsworth et al. found that reminder messages are more effective when the messages note the specific cost of a missed appointment to make the incurred costs of missed appointments more salient to the patients opportunity cost calculation [ 14 ].

Additionally, three studies compared the characteristics of telephone call reminders, testing the difference generated by the use of human-initiated calls versus automated calls. Results were mixed, where interactive voice response (IVR) system calls were as effective as real-life nurse phone calls [ 47 ] in some instances while clinic staff reminders were more effective in lowering no-show rates compared to automated reminder systems [ 48 ]. Additional studies found no meaningful change when switching to automated messaging from traditional human-initiated calls [ 49 ].

Incentives / disincentives

Only two studies presented incentives or disincentives for patient behaviour [ 23 , 24 ]. The change mechanism was most commonly associated with financial rewards or penalties for patients. Results of the studies indicate that incentives may be more successful that fines. One study found fining patients DKK250 (€34) for non-attendance did not appear to reduce non-attendance [ 23 ]. In another study, providing patients with small incentives to attend designated appointments, such as $15 gift cards to Target or CVS, was associated with improved appointment attendance [ 24 ]. The maximum pay out per patient was limited to $45 and patients in the study were 94% more likely to attend their appointments (OR 1.94 CI 1.16–3.24).

Addressing opportunity costs

Opportunity costs, or the loss of potential benefits from other options when one option is chosen, also contribute to decisions patients make about attending appointments. Two studies assessed interventions, which sought to minimize opportunity costs for appointment attendance, including language and cultural congruency and transportation. Andreae 2017 et al. found that contacting patients using human reminder calls in patients’ preferred language before their appointment improved overall attendance rates [ 50 ]. Chaiyachati et al. combined patient reminders with an offer of free rideshare-based transportation services [ 51 ]. The authors found that offering transportation to patients may improve the convenience and reduce opportunity cost of attending appointments; however, ridesharing uptake was low and did not impact missed care appointments.

This systematic review described and summarized the published evidence on behavioural economic interventions to reduce patient non-attendance. We highlighted studies that point to the effectiveness of using behavioural economic interventions, such as reminders and financial incentives. In particular, our review identified a large body of literature related to the use of appointment reminders either via mail, telephone or SMS to improve patient attendance – all of which serves to nudge patients into attending their scheduled appointment. We found, mainly, that similar interventions and research have been repeated with minimal change over the past three decades. Namely, the studies focused on issuing reminders as interventions to reduce patient non-attendance. There has been minimal inclusion of additional mechanisms such as ways to remove barriers to care or rewards and/or penalties to incentivize attendance.

This section summarizes the strength of previous studies and underlines future research needs to add to our understanding of the behavioural economic mechanism and interventions that may reduce patient non-attendance in healthcare settings. Specifically, we address (1) the need to expand on the evaluation of the characteristics and mechanisms of the interventions implemented, (2) the sparse use of behavioural economic-based interventions and the necessity to understand behavioural issues related to non-attendance, and (3) the contemporaneous effect of interventions. We conclude our discussion with policy recommendations that can be derived from current findings.

Detailed evaluation of characteristics and mechanisms of interventions. Reminders bring the appointment to the forefront of each patient’s thought process to circumvent attentional biases. This approach addresses the patients’ limitations in memory and attention, which may lead them to act against their self-interest of keeping appointment times. Of the studies which assessed the effectiveness of how reminders are delivered, ten reported mixed results or minimal changes. Subgroup analyses in one study found that changes in attendance rates varied between different consultation types and were significantly for general and smoking cessation consultations but insignificant in HIV clinics and dietician consultations [ 52 ]. Similarly, mixed results on outcomes were found in the case of reminders for electrodiagnostic examinations which lacked significance in attendance rate compared to the significant changes observed within only needle electromyography appointment attendance [ 39 ].

These findings underscore the criticality of analysing the characteristics as well as substantive content of reminder messages, preferably in conjuncture. We identified three studies that evaluated how the varying characteristics (e.g., automated calls or human-initiated calls) of reminders in the same mode (e.g., telephone calls) may impact the effectiveness of the intervention [ 47 , 48 , 49 ]. The results are mixed, with one study showing IVR calls were as effective as real-life nurse phone calls [ 47 ], one study indicating that reminder calls initiated by clinic staff reminders were more effective in lowering no-show rates [ 48 ], and the third finding substantively null results when switching to automated messaging from traditional human-initiated calls [ 49 ]. The mixed results may be due to an unobserved variable – the content of the messages. This finding suggests the content of the message may be the key to understanding the discrepancy in the above outlined effectiveness of reminders. This rationale is substantiated by one study findings that focused on how the framing of reminder messages impact the effectiveness of those messages [ 14 , 53 ]. Senderey et al. tested eleven separate framings of SMS reminders, to drive different motivational narratives on appointment attendance. Five types of messages produced statistically significant effect of reducing no-show rates, with emotional guilt and specific cost message frames created the greatest difference in no-show rates [ 53 ]. Furthermore, reminders in patients preferred language were particularly effective amongst Hispanic patients, pointing to the success of the cultural congruence [ 50 ].

Behavioural economic-based interventions. Our review identified significant evidence supporting the effectiveness of reminders (either by SMS, telephone, or mail) across various settings. However, there is a lack of evidence regarding alternative interventions and efforts to address other heuristics, leaving a majority of behavioural economic approaches unused and unassessed. For example, in one study – excluded in our review due to the high risk of bias – patient choice and agency were employed in interventions that allowed patients to select their initial appointment dates and times [ 54 ]. To determine how to leverage behavioural economics in ameliorating patient non-attendance, it is crucial to first discuss behavioural economic issues in appointment attendance.

High opportunity costs, such as costly or inconvenient transportation, or the need for a translator, can contribute to increased DNA rates. Removing or mitigating opportunity costs can be seen as a way to reduce barriers to care, increasing the likelihood that patients honour their scheduled appointment. Patients may be more likely to miss appointments if consequences for non-attendance are low, or obligation to attend is not in the forefront. The broad body of evidence points to the value of financial penalties, if they are executed correctly [ 18 , 55 , 56 ]. Other approaches such as loss incentives (penalties), or gain incentives (rewards) are theoretically promising and should be explored in future research. The lack of richness to the variety of available behavioural economic approaches found in our review could be due to the relative novelty of behavioural economics in public policymaking. Popularized for use in public policy during the early 2000’s, and furthered by the widespread recognition of the importance of the “nudge theory” introduced by Sunstein and Reich in 2017 [ 57 ], enough time may not have passed to allow for a significant body of robust randomized controlled trials to test these theories.

Contemporaneous effects of interventions. Our review identified six studies which assessed mail-based reminders; all but one was published prior to 2012. We reason that the ubiquitous nature of smartphone use and the estimated 67.1% penetration rate of the present day [ 58 ] may make physically mailed reminders a less cost-effective strategy when compared to SMS or phone calls. While we did not set a time frame for study inclusion, as we reason that the issue of non-attendance and the intervention to reduce non-attendance is a timeless one, we want to highlight that time-period and context may make one effective strategy more or less cost-effective when compared to other strategies. However, despite the demonstrated effectiveness of physically mailed reminders, in the current global context, we would recommend a careful evaluation of the strategy and the healthcare context prior to adopting sending reminders through the channel of physical mails.

Future Studies and Policy Recommendations. Many studies in our review were conducted in public settings where payments are not required for patient’s visits. Evidence points to individuals being motivated by losses more than gains [ 59 ]. Using this knowledge to craft loss incentives, such as has been proven successful in other areas of public health concern, including increasing physical activity amongst obese adults [ 60 ]. Another approach is no-show fees to incentivize patient attendance. We recognize that the broader evidence indicates that penalties may be problematic and exacerbate disparities in healthcare [ 61 , 62 , 63 ]; however, we reason that this approach may be beneficial for patients from all background as well as cost-effective at a societal level when combined with other interventions such as a reminder system. Although the body of evidence is limited in our review, there are some indications that positive financial incentives could have a stronger impact that negative ones [ 23 , 24 ].

Patients are prone to present bias in which the benefits of the care received from attending an appointment, particularly for chronic or primary care visits, occurs in the distant future. The benefits of the visit may offer long-term benefits, however this might come into direct conflict with the immediate costs of attending the appointment, such as missed working hours or transportation costs. Lowering opportunity costs for patients may be another effective approach that warrants further evaluation. The opportunity costs of adults seeking medical care through ambulatory services have been estimated at $43, which was substantial and exceeds the average patient’s out of pocket payment [ 64 ]. Although there was minimal evidence available in our review in regards to interventions addressing opportunity costs, broader literature points to the value of minimizing opportunity costs to improve health care delivery. For example, Lee et al. 2020 found that it is possible that the study setting of a safety-net based primary care hospital contributed to the strength of effectiveness of the incentive based intervention [ 24 ]. The small financial benefit may have a particularly strong effect on lowering opportunity costs for patients most at risk. Furthermore, these interventions could not only increase health system efficiency but also result in high patient satisfaction.

The bulk of our evidence assessed the use of reminders to address barriers to access. Nevertheless, existing public health literature has robust evidence supporting the pattern that mechanisms that can reduce the barriers to care results in increased healthcare utilization [ 65 , 66 , 67 ]. One potential intervention is leveraging heuristics to incentivize patients to circumvent these barriers and access care. For example, offering a one-time fee waiver if patients were to sign up for an online portal that can help patients schedule telehealth care when patients cannot make it to in-person care. Another intervention may be offering incentives for patients to sign up for a ride-share service, making it easier for patients to attend appointments while incurring a lower transportation cost. Future studies must endeavour to better understand the heuristics that lead patients to engage in behaviours that minimize the difficulties of attending an appointment.

Lastly, four studies included in our review discussed the cost or cost-effectiveness of these interventions [ 27 , 68 , 69 , 70 ]. This is a good start; however, cost and budget impact assessments will be a critical part of any decision-maker’s choice to implement the interventions assessed in this review. Therefore, further research into the cost implications of any behavioural economic interventions, especially in comparison to the potential losses faced by high no-show rates, should be prioritized in any future research.

Limitations

Our review has several important limitations. Firstly, some studies utilized interventions and quality study design that met the inclusion criteria, but did not report outcomes in a manner that allowed the pooling of results in a meta-analysis, were consequently excluded from this review. As we aim to include comparable results, many studies were excluded in the risk of bias assessment step due to a lack of study design that enables causal inference; however, many of these studies included unique interventions that warrant future evaluation. Because our search terms included “behavioural economics”, some studies that included intervention(s) that are behavioural economics-oriented and incentivize desired behaviour may have been excluded if the authors did not categorize their intervention as one that derives from behavioural economics. Lastly, we are cognizant that publication bias may favour studies with positive results, leaving out many interventions to reduce patient non-attendance; nevertheless, our review captures ten studies with null results [ 23 , 32 , 33 , 34 , 35 , 36 , 39 , 49 , 51 , 68 ].

Our review identified 61 studies on the use of behavioural economic interventions to reduce no-show rates. The included studies reflect a lack of diversity in intervention approaches but point to the effectiveness of reminder systems in reducing no-show rates across a variety of medical departments. We recommend future studies to test additional behavioural economic interventions that have not been used, tested, and/or published before. And, when examining frequently tested interventions, such as reminders, one should focus on the substantive aspect of the reminder message (e.g., framing of the message) and the characteristics of these messages (e.g., automated or human-initiated). Decision-makers will want to consider current findings with caution and ensure to evaluate the healthcare context before implementing effective interventions outlined by this systematic review.

Data Availability

All data generated or analyzed during this study are included in this published article and its supplementary information file.

Missed GP. appointments costing NHS millions. https://www.england.nhs.uk/2019/01/missed-gp-appointments-costing-nhs-millions/ .

Schofield K. Rishi Sunak Says People Should Be Charged £10 If They Miss A GP Or Hospital Appointment. Huffington Post. 2022.

Shimotsu S, Roehrl A, McCarty M, Vickery K, Guzman-Corrales L, Linzer M, et al. Increased likelihood of missed appointments (no shows) for Racial/Ethnic minorities in a Safety Net Health System. J Prim Care Community Health. 2016;7:38–40.

Article   PubMed   Google Scholar  

Dantas LF, Fleck JL, Cyrino Oliveira FL, Hamacher S. No-shows in appointment scheduling – a systematic literature review. Health Policy (New York). 2018;122:412–21.

Article   Google Scholar  

Carreras-García D, Delgado-Gómez D, Llorente-Fernández F, Arribas-Gil A. Patient No-Show Prediction: A Systematic Literature Review. 2020. https://doi.org/10.3390/e22060675 .

Bhise V, Modi V, Kalavar A, Espadas D, Hanser L, Gould M, et al. Patient-reported attributions for missed colonoscopy appointments in two large Healthcare Systems. Dig Dis Sci. 2016;61:1853–61.

Kumthekar A, Johnson B. Improvement of appointment compliance in an underserved lupus clinic. BMC Health Serv Res. 2018;18:610.

Article   PubMed   PubMed Central   Google Scholar  

Martin C, Perfect T, Mantle G. Non-attendance in primary care: the views of patients and practices on its causes, impact and solutions. Fam Pract. 2005;22:638–43.

Harte E, MacLure C, Martin A, Saunders CL, Walter FM, Mant J, et al. Reasons why people do not attend NHS health checks: a systematic review and qualitative synthesis. Br J Gen Pract. 2018;68:e28–35.

Münscher R, Vetter M, Scheuerle T. A review and taxonomy of Choice Architecture techniques. J Behav Decis Mak. 2016;29:511–24.

Hansen PG, Jespersen AM. Nudge and the manipulation of choice: a Framework for the responsible use of the Nudge Approach to Behaviour Change in Public Policy. Eur J Risk Regul. 2013;4:3–28.

Matjasko JL, Cawley JH, Baker-Goering MM, Yokum DV. Applying behavioral economics to Public Health Policy: illustrative examples and promising directions. Am J Prev Med. 2016;50(5 Suppl 1):13–9.

Madrian BC. Applying insights from behavioral economics to Policy Design. Annu Rev Econom. 2014;6:663–88.

Hallsworth M, Berry D, Sanders M, Sallis A, King D, Vlaev I et al. Stating appointment costs in SMS reminders reduces missed Hospital appointments : findings from two Randomised Controlled Trials. 2015;:1–14.

Stubbs ND, Sanders S, Jones DB, Geraci SA, Stephenson PL. Methods to reduce outpatient non-attendance. Am J Med Sci. 2012;344:211–9.

Crable EL, Biancarelli DL, Aurora M, Drainoni M, Walkey AJ. Interventions to increase appointment attendance in safety net health centers : a systematic review and meta-analysis. August. 2021;2020:965–75.

Google Scholar  

Gneezy URI, Rustichini A, Glaeser E, Levine D, Nyhus E, Palfrey T et al. A fine is a price. 2000;XXIX January.

Bech M. The economics of non-attendance and the expected effect of charging a fine on non-attendees. 2005;74:181–91.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA Statement for reporting systematic reviews and Meta-analyses of studies that evaluate Health Care Interventions: explanation and elaboration. PLoS Med. 2009;6:e1000100.

Kheirkhah P, Feng Q, Travis LM, Tavakoli-Tabasi S, Sharafkhaneh A. Prevalence, predictors and economic consequences of no-shows. BMC Health Serv Res. 2016;16:1–6.

Hong Q, Fabregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, et al. The mixed methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. 2018;34:285–91.

Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:964–7.

Blæhr EE, Væggemose U, Søgaard R. Effectiveness and cost-effectiveness of fining non-attendance at public hospitals: a randomised controlled trial from danish outpatient clinics. BMJ Open. 2018;8.

Lee KS, Quintiliani L, Heinz A, Johnson NL, Xuan Z, Truong V, et al. A financial incentive program to improve appointment attendance at a safety-net hospital-based primary care hepatitis C treatment program. PLoS ONE. 2020;15:1–10.

CAS   Google Scholar  

Chung S, Martinez MC, Frosch DL, Jones VG, Chan AS. Patient-centric scheduling with the implementation of health information technology to improve the patient experience and access to care: retrospective case-control analysis. J Med Internet Res. 2020;22:1–11.

Groden P, Capellini A, Levine E, Wajnberg A, Duenas M, Sow S, et al. The success of behavioral economics in improving patient retention within an intensive primary care practice. BMC Fam Pract. 2021;22:253.

Quattlebaum TG, Darden PM, Sperry JB. Effectiveness appointment of computer-generated reminders. Pediatrics. 1991;88:801–5.

Article   CAS   PubMed   Google Scholar  

Rusius CW. Improving out-patient attendance using postal appointment reminders. Psychiatr Bull. 1995;19:291–2.

Can S, Macfarlane T, O’Brien KD. The use of postal reminders to reduce non-attendance at an orthodontic clinic: a randomised controlled trial. Br Dent J. 2003;195:199–201.

Krishna M, Amarjothi S. Reminder letters to imprve rate of attendance at community mental health centre. Br J Med Pract. 2012;5:1–3.

Mikhaeil JS, Celo E, Shanahan J, Harvey B, Sipos B, Law MP. Attend: a Two-Pronged Trial to eliminate no shows in Diagnostic Imaging at a community-based hospital. J Med Imaging Radiat Sci. 2019;50:36–42.

Fairhurst K, Sheikh A. Texting appointment reminders to repeated non-attenders in primary care: Randomised controlled study. Qual Saf Heal Care. 2008;17:373–6.

Article   CAS   Google Scholar  

Narring F, Perron NJ, Dao MD, Righini NC, Humair JP, Broers B, et al. Text-messaging to reduce missed appointment in a youth clinic: a randomised controlled trial. J Epidemiol Community Health. 2013;67:888–91.

Clough BA, Casey LM. Using SMS reminders in psychology clinics: a cautionary tale. Behav Cogn Psychother. 2014;42:257–68.

Mahmud N, Asch DA, Sung J, Reitz C, Coniglio MS, McDonald C, et al. Effect of text messaging on Bowel Preparation and Appointment Attendance for Outpatient Colonoscopy: a Randomized Clinical Trial. JAMA Netw open. 2021;4:e2034553.

Stormon N, Sexton C, Chen C, Hsu E, Chen P-C, McGowan K. SMS reminders to improve outpatient attendance for public dental services: a retrospective study. Heal Soc Care. 2021;00:1–9.

Hashim MJ, Franks P, Fiscella K. Effectiveness of telephone reminders in improving rate of appointments kept at an outpatient clinic: a randomized controlled trial. J Am Board Fam Pract. 2001;14:193–6.

CAS   PubMed   Google Scholar  

Ruggeri K, Folke T, Benzerga A, Verra S, Büttner C, Steinbeck V, et al. Nudging New York: adaptive models and the limits of behavioral interventions to reduce no-shows and health inequalities. BMC Health Serv Res. 2020;20:1–11.

Kwon SB, Jung S, Kang SY, Hong SS, Hwang SH. The effect of telephone call reminders on electrodiagnostic laboratory attendance in Korea. HealthMED. 2012;6:2850–5.

Kourany RF, Garber J, Tornusciolo G. Improving First Appointment Attendance Rates in Child Psychiatry Outpatient Clinics. J Am Acad Child Adolesc Psychiatry. 1990;29:657–60.

Koren ME, Bartel JC, Corliss J. Interventions to improve patient appointments in an ambulatory care facility. Journale Ambul Care Manag. 1994;17:76–80.

Lance RC, Pazin-Filho A, Neves FF. Comparison between short text messages and phone calls to reduce no-show rates in outpatient medical appointments: a randomized trial. J Ambul Care Manage. 2021;44:314–20.

Liew SM, Tong SF, Lee VKM, Ng CJ, Leong KC, Teng CL. Text messaging reminders to reduce non-attendance in chronic disease follow-up: a clinical trial. Br J Gen Pract. 2009;59:916–20.

Leong KC, Chen WS, Leong KW, Mastura I, Mimi O, Sheikh MA, et al. The use of text messaging to improve attendance in primary care: a randomized controlled trial. Fam Pract. 2006;23:699–705.

Nelson TM, Berg JH, Bell JF, Leggott PJ, Seminario AL. Assessing the effectiveness of text messages as appointment reminders in a pediatric dental setting. J Am Dent Assoc. 2011;142:397–405.

Bigna JJR, Noubiap JJN, Kouanfack C, Plottel CS, Koulla-Shiro S. Effect of mobile phone reminders on follow-up medical care of children exposed to or infected with HIV in Cameroon (MORE CARE): a multicentre, single-blind, factorial, randomised controlled trial. Lancet Infect Dis. 2014;14:600–8.

Griffin JM, Hulbert EM, Vernon SW, Nelson D, Hagel EM, Nugent S, et al. Improving endoscopy completion: effectiveness of an interactive voice response system. Am J Manag Care. 2011;17:199–208.

PubMed   Google Scholar  

Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of Outpatient appointment reminder Systems in reducing No-Show Rates. Am J Med. 2010;123:542–8.

Roseland ME, Shankar PR, Houck G, Davenport MS. Targeting missed Care Opportunities using Modern Communication Methods: a Quality Improvement Initiative to Improve Access to CT and MRI appointments. Acad Radiol. 2022;29:395–401.

Andreae MH, Nair S, Gabry JS, Goodrich B, Hall C, Shaparin N. A pragmatic trial to improve adherence with scheduled appointments in an inner-city Pain Clinic by human phone calls in the patient’s Preferred Language. J Clin Anesth. 2017;42:77–83.

Chaiyachati KH, Hubbard RA, Yeager A, Mugo B, Lopez S, Asch E, et al. Association of rideshare-based transportation services and missed primary care appointments: a clinical trial. JAMA Intern Med. 2018;178:383–9.

Junod Perron N, Dominicé Dao M, Kossovsky MP, Miserez V, Chuard C, Calmy A, et al. Reduction of missed appointments at an urban primary care clinic: a randomised controlled study. BMC Fam Pract. 2010;11:79.

Article   PubMed Central   Google Scholar  

Senderey AB, Kornitzer T, Lawrence G, Zysman H, Hallak Y, Ariely D, et al. It’s how you say it : systematic A / B testing of digital messaging cut hospital no-show rates. PLoS ONE. 2020;15:1–13.

Jenkins PE. Reducing Non-Attendance Rates for Assessment at an eating Disorders Service: a Quality Improvement Initiative. Community Ment Health J. 2017;53:878–82.

Mäntyjärvi M. No-show patients in an ophthalmological out‐patient department. Acta Ophthalmol. 1994;72:284–9.

Lesaca T. Assessing the influence of a no-show fee on patient compliance at a CMHC. Adm Policy Ment Health. 1995;22:629–31.

Sunstein C, Reisch L. The Economics of Nudge. Crit Concepts Econ. 2017;4.

Simon Kemp. Digital 2021 October Global Statshot Report.

Kahneman D, Tversky A, Tversky ’ A. Prospect Theory: an analysis of decision under risk. Source Econom. 1979;47:263–92.

Patel MS, Asch DA, Rosin R, Small DS, Bellamy SL, Heuer J, et al. Framing Financial Incentives to increase physical activity among overweight and obese adults: a Randomized, Controlled Trial. Ann Intern Med. 2016;164:385–94.

Vlaev I, King D, Darzi A, Dolan P. Changing health behaviors using financial incentives: a review from behavioral economics. BMC Public Health. 2019;19:1–9.

Upadhyay PA, Maddison AD, Spiegel JS. Why inequitable and burdensome court- issued fines and fees are a health issue — and what health and policy leaders can do about it. October; 2021.

Loewenstein G, Asch DA, Volpp KG. Behavioral economics holds potential to deliver better results for patients, insurers, and employers. Health Aff. 2013;32:1244–50.

Ray KN, Chari AV, Engberg J, Bertolet M, Mehrotra A. Opportunity costs of Ambulatory Medical Care in the United States. Am J fo Manag Care. 2015;21.

Borno HT, Lin TK, Zhang S, Skafel A, Lalanne A, Dornsife D, et al. Accelerating cancer clinical trial recruitment through a financial reimbursement program integrated with patient navigation: an interrupted time series analysis. J Cancer Policy. 2021;30:100305.

Allen EM, Call KT, Beebe TJ, McAlpine DD, Johnson PJ. Barriers to Care and Healthcare utilization among the publicly insured. Med Care. 2017;55:207–14.

Stajduhar K, Mollison A, Giesbrecht M, McNeil R, Pauly B, Reimer-Kirkham S et al. Just too busy living in the moment and surviving: barriers to accessing health care for structurally vulnerable populations at end-of-life. BMC Palliat Care. 2019;18.

Perron NJ, Dao MD, Righini NC, Humair JP, Broers B, Narring F et al. Text-messaging versus telephone reminders to reduce missed appointments in an academic primary care clinic: a randomized controlled trial. BMC Health Serv Res. 2013;13.

Sims H, Sanghara H, Hayes D, Wandiembe S, Finch M, Jakobsen H, et al. Text message reminders of appointments: a pilot intervention at four community mental health clinics in London. Psychiatr Serv. 2012;63:161–8.

Roberts N, Meade K, Partridge M. The effect of telephone reminders on attendance in respiratory outpatient clinics. J Heal Serv Res Policy. 2007;12:69–72.

Download references

Acknowledgements

The authors would like to thank the World Bank peer reviewers, Jungkyu Rhys Lim, Abigail Dalton, and Fatme Koek whose inputs and feedback helped improve the quality of the paper. The team is also grateful for the overall support provided by Mohammed Alabdulaali, Assistant Ministry of the Ministry of Health; Rekha Menon, World Bank Practice Manager of HNP in MENA, and Issam Abousleiman, World Bank Regional Director for the GCC countries.

This work was supported by the Ministry of Health, Kingdom of Saudi Arabia and World Bank. Financing for the analysis was provided by the Saudi Health Council and the Health, Nutrition and Population Reimbursable Advisory Services Program between the World Bank and the Ministry of Finance in Saudi Arabia.

Author information

Authors and affiliations.

Institute for Health & Aging, Department of Social and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA

Kalin Werner & Tracy Kuo Lin

Nudge Unit, Ministry of Health, Riyadh, KSA, Saudi Arabia

Sara Abdulrahman Alsuhaibani, Reem Alshehri & Mohammed Alhajji

Department of Health Sciences, College of Health and Rehabilitation Sciences, Princess Nourah bint Abdulrahman University, Riyadh, KSA, Saudi Arabia

Sara Abdulrahman Alsuhaibani

Community Health Sciences, College of Applied Medical Sciences, King Saud University, Riyadh, KSA, Saudi Arabia

Reem F. Alsukait

Health, Nutrition and Population Global Practice, The World Bank, Washington, D.C, USA

Reem F. Alsukait & Christopher H. Herbst

College of Medicine, Alfaisal University, Riyadh, KSA, Saudi Arabia

Mohammed Alhajji

You can also search for this author in PubMed   Google Scholar

Contributions

CH, MA, TL and KW contributed to the conception and design of the work. KW,TL and SAA conducted data collection and analysis. KW and TL performed data interpretation, data visualization and drafted the article. RFA lead overall project administration. CH, MA, RA, SAA, RFA provided critical revisions of the article. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kalin Werner .

Ethics declarations

Ethics approval and consent to participate.

Not applicable. Ethical approval for this type of study is not required by our institute.

Consent for publication

Not applicable.

Competing interests

The authors declare no conflicts of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Werner, K., Alsuhaibani, S.A., Alsukait, R.F. et al. Behavioural economic interventions to reduce health care appointment non-attendance: a systematic review and meta-analysis. BMC Health Serv Res 23 , 1136 (2023). https://doi.org/10.1186/s12913-023-10059-9

Download citation

Received : 31 August 2022

Accepted : 24 September 2023

Published : 23 October 2023

DOI : https://doi.org/10.1186/s12913-023-10059-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Behavioural economics
• No-show rates
• Attendance rates
• Systematic reviews

BMC Health Services Research

ISSN: 1472-6963

• Submission enquiries: [email protected]
• General enquiries: [email protected]

SYSTEMATIC REVIEW article

This article is part of the research topic.

Experts in Neurodegeneration: Heritability in Neurodegenerative Diseases

Exploring the Relationship between IGHMBP2 Gene Mutations and Spinal Muscular Atrophy with Respiratory Distress Type 1 (SMARD1) and Charcot-Marie-Tooth Disease Type 2S (CMT2S): A Systematic Review

• 1 Third Affiliated Hospital of Zhengzhou University, China

The final, formatted version of the article will be published soon.

Background:IGHMBP2 is a crucial gene for the development and maintenance of the nervous system, especially in the survival of motor neurons. Mutations in this gene have been associated with spinal muscular atrophy with respiratory distress type 1 (SMARD1) and Charcot-Marie-Tooth disease type 2S (CMT2S). Methods:We conducted a systematic literature search using the PubMed database to identify studies published up to April 1st, 2023, that investigated the association between IGHMBP2 mutations and SMARD1 or CMT2S. We compared the nontruncating mutations and truncating mutations of the IGHMBP2 gene and selected highfrequency mutations of the IGHMBP2 gene. Results: We identified 52 articles that investigated the association between IGHMBP2 mutations and SMARD1/CMT2S. We found 6 hotspot mutations of the IGHMBP2 gene. The truncating mutations in trans were all associated with SMARD1.This study provides evidence that the complete LOF mechanism of the IGHMBP2 gene defect may be an important cause of SMARD1.

Keywords: IGHMBP2 gene, Spinal muscular atrophy with respiratory distress type 1 (SMARD1, Charcot-Marie-Tooth disease (CMT2S), mutations, clinical diagnosis

Received: 03 Jul 2023; Accepted: 03 Nov 2023.

Copyright: © 2023 Tian, Xing, Shi and Yuan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Yuan Tian, Third Affiliated Hospital of Zhengzhou University, Zhengzhou, China Prof. Enwu Yuan, Third Affiliated Hospital of Zhengzhou University, Zhengzhou, China

People also looked at