systematic review about psychology

Psychology: Systematic Reviews and Literature Reviews

• Find Articles
• Find Journals
• Dissertations
• Videos & More
• Systematic Reviews and Literature Reviews

Guidelines for Systematic Reviews

• PRISMA Checklist The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) has developed a 27 checklist items pertaining to the content of a systematic review and meta-analysis, which include the title, abstract, methods, results, discussion and funding.
• NIH Guidelines This website provides guidelines developed by the Institute of Medicine (IOM). more... less... In 2011, the Institute of Medicine (IOM) defined a systematic evidence review as "a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may include a quantitative synthesis (meta-analysis), depending on the available data."

Register your Systematic Review

Prospero Registry

"PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome." (Website- About)

Conducting a Literature Review

• The Literature Review (25 minutes Video - opens in a new window)) This in-depth video lecture explains how to write a Literature Review, and examines which elements are required in one. This video was created the Student Learning Centre at Massey University, Auckland.

Systematic Reviews - What to Consider

Before you Begin: 

• Did you know, it takes an average of 18 months to complete a systematic review?
• Did you know that it's recommended to have at least 3 people on a systematic review team? 
• Did you know that a systematic review answers a very specific type of research question? Is your question a good fit for a systematic review? 

Contact your librarian, Emily Hart , to learn more about systematic reviews. 

Types of Reviews

Comparison of Review Types from Cornell University Libraries

" A systematic review is a review of the literature that is conducted in a methodical manner based on a pre-specified protocol and with the aim of synthesizing the retrieved information often by means of a meta-analysis."

" A literature review is a descriptive and/or analytic summary of the existing material relating to some topic or area of study."

(Sage Research Methods Online)

Systematic Review vs. Literature Review - What's the Difference?

• << Previous: Videos & More
• Last Updated: Aug 8, 2023 3:49 PM
• URL: https://researchguides.library.syr.edu/psychology
• Print This Page

• Psychology and Psychiatry
• PsycINFO (Ovid) Tutorial
• Point of Care Resources
• Tests and Measures

Systematic Literature Review

• Referencing and Citing
• Library Drop-in Sessions

 Outlined below is a systematic approach to conducting a literature review using the PICO(T) principle.

Follow the steps to:

• formulate an answerable question/issue
• refine concepts and keywords
• develop search strategies
• critically appraise literature
• complete the Research Planner

Step 1: Question Type

Step 2: pico(t) principle.

The PICO(T) principle is a framework in which to formulate a question or issue of interest. Questions should be answerable and refined, so to identify each of the components in the PICO formula. This will allow for keywords to be easily defined and more effective search results returned.

P  Population/Problem

I   Intervention

C  Comparison/Control

O  Outcome

(T) Timeframe

An answerable question would ask: In P how does I compared with C affect the O ?

Step 3: Prepare for Searching

Identify the Search Terms

• Consider what the main concepts of the question are
• Identify keywords and synonyms

 Combine your search terms using Boolean Operators

AND    Look for articles that include all identified keywords

OR      Look for articles that include any identified keywords

NOT    Exclude articles that contain this specific keyword

Step 4: Effective Searching

MeSH - Medical Subject Headings Is a controlled vocabulary and is used for the purpose of indexing journals and books. They are pre-determined concepts/terms which describe content in discipline specific, indexing databases PubMed, Medline and Cinahl databases include MeSH Headings or Cinhal Subject Headings, which can also serve as a Thesaurus. Searching the databases using MeSH terms allows for terms to be searched as a Major Concept (the search will return only records for which the subject heading is a major point of the article) and also provides the ability to Explode (expands the subject heading to retrieve all references indexed to that term, plus any narrower terms that are connected)

Discipline specific Databases

Medline:  One of the most acknowledged biomedicine databases, Medline's coverage includes basic research in the biomedical and clinical sciences as well as major health related subjects including nursing and allied health. Its records are indexed using MeSH.  PubMed is the publicly accessible version of Medline.

Cochrane Library:   P rovides access to a collection of databases that contain high-quality, independent evidence to inform health care decision-making. These databases include: Cochrane Reviews, Database of Abstracts of Reviews of Effects - DARE (Other Reviews), Clinical Trials and NHS Economic Evaluation Database.

PubMed: PubMed Clinical Queries allows you to find critically appraised information indexed in MEDLINE. PubMed uses a broad definition of systematic reviews. It includes systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, and guidelines.

Scopus:  Offers peer-reviewed journals in the scientific, technical, medical and social science fields. An excellent source for locating Authors' works.

Step 5: Critical Appraisal

Critical appraisal is applying a systematic approach to analyse and identify unbiased/biased and valid literature. A literature review should not be a list summarising each article retrieved. Instead, it should appear as organised sections into themes or concepts, synthesised and evaluated, according to the research question.

Set your criteria to assist in critically appraising the literature e.g.,

• Does the literature directly relate to the question?
• What are the strengths and limitations of the literature and how does it contribute to the study?

Read more about Critical Appraisal under the Evidence Based Nursing tab  

Use CASP (Critical Appraisal Skills Program) to analyse and critique the literature

Step 6: Research Planner

• Research Planner Use the Planner to record your search strategies etc

Use the Research Planner to assist in developing a recordable framework for the Literature Review

Tools to Manage References

Students are required to use APA Style referencing

       Recite - APA 

EndNote is software that allows you to record, organise and use references and PDF files. It assists in producing bibliographies and will generate citations and bibliographies in specific styles (e.g. APA 6th) into word documents. Read more about EndNote

• << Previous: Evidence Based Psychology
• Next: Referencing and Citing >>
• Last Updated: Nov 3, 2023 2:56 PM
• URL: https://unimelb.libguides.com/psychology

• Academic Resources
• Grey Literature: Resources
• Conducting a Systematic Review
• "How to" videos: searching
• Reference Management Software: Zotero This link opens in a new window
• Further Support

Systematic Reviews: Introduction

• What is a systematic review?
• How does it differ from a systematic literature review?

A systematic review is a review of research literature (often limited to randomised controlled trials) that is searched for in a systematic or exact manner, and only includes studies that have met strict inclusion criteria. The methodology of the review often follows a internationally recognized systematic review standard or guideline. 

PRISMA definition

"A systematic review is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review." (PRISMA, 2015)

A systematic literature search is a literature review which demonstrates that you have compiled a list of appropriate search terms and includes the structure of your search history, which provides the evidence on which your assignment is based.  You may be told that you need to conduct a systematic review when in fact you just need to perform a literature search in a systematic manner. 

This is a less rigorous process than a systematic review. A systematic review usually covers a wider scope; you would be expected to look at all the available research in the area in question, systematically searching multiple academic databases.

If you are unsure about the differences between a systematic review and a literature review take a look at this guide:  What’s in a Name? The difference between a Systematic Review and a Literature Review and Why it Matters .

Systematic reviews: your research question

Your research question

• Inclusion and exclusion criteria
• PICO framework
• PICO: example

Where possible, select a research topic that you have some basic knowledge in, or are familiar with in.  If this is not possible, seek guidance from your supervisor on the most relevant terminology for your chosen topic.  This will make it easier to devise your research question, ensuring you are using the most appropriate and relevant phrasing.This is especially important when you come to build your search strategy, as the wording of your research question research question will frame your initial search. 

The search terms, phrases, words and subject headings you use in your systematic search play a vital role in your systematic review, because they determine the number and range of results you will get from each database. I f your question is too wide-reaching so will your search, and you will be overwhelmed with  an unmanageable number of search results. You also have to think very carefully about the limitations of your search - such as whether or not you will have a specific date range or limit to a particular study design.  Remember, the clue is in the name - be  systematic  in your approach. 

Inclusion and exclusion criteria 

Before you start to build your search strategy you will need to decide on your review's inclusion and exclusion criteria. This will have a bearing on your search strategy, and potentially determine which studies your systematic search captures. Systematic reviews are measured against a set of specific criteria outlined at the start.  Your inclusion and exclusion criteria dictates which studies will be included in your systematic review (they meet all aspects of your inclusion criteria) and those that will be excluded (does not match your inclusion criteria fully, or meets the exclusion criteria).  You need to think very carefully about the limitations of your research - will you limit to a specific date range or to a particular study design for example.  Remember, the clue is in the name - be  systematic  in your approach. 

Research question:

"For UK children under 5 years of age with a suspected diagnosis of ADHD, what are the recommended clinical assessments used for diagnosis?" 

Inclusion criteria:

• children 0-5 years old 
• suspected ADHD
• limited to clinical assessments specifically, for diagnosis
• UK population studies only

Exclusion criteria:

• any member of the population over the age of 5 
• animal studies
• pre-determined diagnosis of ADHD 
• assessments not defined as "clinical" 
• studies of populations outside of the UK

PICO Framework

In order to structure your search strategy  it can be helpful to use a structured framework.They are designed to help you formulate your research question, search strategy, and inclusion and exclusion criteria. The PICO framework is a popular framework for physiological and health sciences systematic reviews.

PICO stands for:

Population / Patient

Intervention 

You do not have to structure your research to fit into all of the above - it is not prescriptive, but merely a framework to guide you in building your search strategy. Your research question may not have a defined comparative intervention, for example, so you may only use PIO for your framework. 

PICO example

Research question: What impact does Cognitive Behavioural Therapy (CBT) versus pharmaceutical therapy have as treatments for diagnosed anxiety in children  in reducing their anxiety?

Systematic Reviews: the search

• Your search strategy
• Where to search
• Search techniques
• PRESS checklist
• Sample strategy

Your search strategy - where to start

The search terms, phrases, words and subject headings you use in your search play a vital role in your systematic review, because they determine the number and range of results you will get from each database. 

Start with your research question to pick out the  keywords  you will use as the basis of your search. A common practice in systematic review searches is to refer back to the framework you used to structure your research question, such as the PICO framework . This framework gives a clear picture of not only the terms you want to use in your search, but also how you will combine or connect them together in the database.

Using PICO framework for your search 

Research question: Is Cognitive Behavioural Therapy (CBT) more effective than pharmaceutical therapy as a treatment for diagnosed anxiety in teenagers aged 13-18 in managing symptoms of anxiety?

Framework  

Search terms identified: 

• teen/teenager
• 13-18 year olds
• anxiety 
• CBT/cognitive behavioural the rapy
• pharmaceutical/medication
• Improved   management of symptoms

Where to search - Databases

We have grouped the key databases and online resources for psychology in the Key Resources tab . Your dissertation supervisor may also suggest or advise you on the various databases suitable to your search. Some of the most commonly searched databases include: PsycINFO, Web of Science, Cochrane, Scopus, and PubMed. The links for these can be found in the Key Resources tab. 

Where to search - Grey Literature

Depending on your research, you may also need to search grey literature sources in addition to academic database collections. Have a look at the grey literature tab  on the left hand side more information about what defines grey literature and the sources you can search. 

Search techniques 

Consider your keywords carefully 

It's often helpful to look at the abstracts and reference lists of studies or papers you have already engaged with on your topic for a sense of what terminology and specific words/phrases are most common. 

Once you have made a comprehensive list of all the keywords you want to include in your search you need to combine them using what are known as Boolean operators or search connectors. 

Boolean search operators 

In order to combine multiple facets of your search together in a database you will need to use the search operators: AND, OR, NOT. Using operators is common in systematic reviews, and improves the efficiency of your search resulting in more relevant results. Watch the Boolean searching videos to learn more about how to use operators.

Proximity operators 

Using proximity operators in your search allows you to  search for two or more words that occur within a certain number of words from each other. Proximity operators can be helpful when you don't want to limit yourself to searching an exact phrase, for example "diagnosed anxiety", but where you still want to refine your search to a degree. Each database will have its own method of proximity searching, but generally your search looks something like: diagnosed NEAR/3 anxiety - NEAR signifying near to, and 3 signifying the number of words that can appear between "diagnosed" and "anxiety". To ensure proximity searching is available in the database you are searching, have a look for their "search tips" or "search help" page. 

PRESS (Peer Review of Electronic Search Strategies)

The PRESS checklist is a comprehensive evidence-based checklist that does just that. It covers six key aspects of search strategies: 

• the translation of the research question into a search strategy
• the use of any Boolean or proximity operators
• the keywords used 
• spelling, syntax, and commands
• database filters / limits intended for use
• database Subject headings intended for use not essential, and may not apply to your strategy)

Each aspect above has a set of questions to check against your search strategy. The full checklist can be downloaded here . 

Research question

Is Cognitive Behavioural Therapy (CBT) more effective than pharmaceutical therapy as a treatment for diagnosed anxiety in teenagers aged 13-18 in managing symptoms of anxiety?

Step 1: Framework

Step 2: Identify search terms

Step 3: Build search strategy with Boolean operators / commands / proximity operators

Systematic Reviews: guidelines and protocols

• Standards and guidelines
• Cochrane handbook
• Choosing a standard/guideline

Standards and guidelines 

These can serve as guidelines for the structure of your review, from what headings/sections to include, (such as methods, search strategy, etc) to best practice protocols for search strategies, reporting, etc. 

​ The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review. 

PRISMA standards

PRISM A  (Preferred Reporting Items for Systematic Reviews and  Meta-Analyses )  is an evidence-based protocol for reporting on systematic reviews and meta-analyses. The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review. The PRISMA checklist makes this easy, providing a checklist of criteria to be adhered to for each of the sections of your review, for example: introduction, methods, results, etc.

Useful PRISMA links: 

• PRISMA Checklist  (checklist of sections to include in your systematic review, eg: methods, results, etc.) 
• PRISMA flow chart   (charts the number of records identified, included and excluded, and the reasons for exclusions.)
• Citing PRISMA

Cochrane Handbook 

Cochrane Handbook for Systematic Reviews of Interventions  

Choosing a systematic review standard/guideline

The School of Psychology recommends its students follow the PRISMA reporting guidance when conducting a systematic review, but it is also helpful to read through the Cochrane Handbook of Systematic Review, particularly if your research topic spans both medical and human sciences.

• << Previous: Grey Literature: Resources
• Next: "How to" videos: searching >>
• Last Updated: Oct 12, 2023 11:19 AM
• URL: https://guides.lib.sussex.ac.uk/psychology

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 13 July 2021

Systematic review and meta-analysis of depression, anxiety, and suicidal ideation among Ph.D. students

• Emily N. Satinsky 1 ,
• Tomoki Kimura 2 ,
• Mathew V. Kiang 3 , 4 ,
• Rediet Abebe 5 , 6 ,
• Scott Cunningham 7 ,
• Hedwig Lee 8 ,
• Xiaofei Lin 9 ,
• Cindy H. Liu 10 , 11 ,
• Igor Rudan 12 ,
• Srijan Sen 13 ,
• Mark Tomlinson 14 , 15 ,
• Miranda Yaver 16 &
• Alexander C. Tsai 1 , 11 , 17  

Scientific Reports volume  11 , Article number:  14370 ( 2021 ) Cite this article

83k Accesses

53 Citations

809 Altmetric

Metrics details

• Epidemiology
• Health policy
• Quality of life

University administrators and mental health clinicians have raised concerns about depression and anxiety among Ph.D. students, yet no study has systematically synthesized the available evidence in this area. After searching the literature for studies reporting on depression, anxiety, and/or suicidal ideation among Ph.D. students, we included 32 articles. Among 16 studies reporting the prevalence of clinically significant symptoms of depression across 23,469 Ph.D. students, the pooled estimate of the proportion of students with depression was 0.24 (95% confidence interval [CI], 0.18–0.31; I 2  = 98.75%). In a meta-analysis of the nine studies reporting the prevalence of clinically significant symptoms of anxiety across 15,626 students, the estimated proportion of students with anxiety was 0.17 (95% CI, 0.12–0.23; I 2  = 98.05%). We conclude that depression and anxiety are highly prevalent among Ph.D. students. Data limitations precluded our ability to obtain a pooled estimate of suicidal ideation prevalence. Programs that systematically monitor and promote the mental health of Ph.D. students are urgently needed.

Introduction

Mental health problems among graduate students in doctoral degree programs have received increasing attention 1 , 2 , 3 , 4 . Ph.D. students (and students completing equivalent degrees, such as the Sc.D.) face training periods of unpredictable duration, financial insecurity and food insecurity, competitive markets for tenure-track positions, and unsparing publishing and funding models 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 —all of which may have greater adverse impacts on students from marginalized and underrepresented populations 13 , 14 , 15 . Ph.D. students’ mental health problems may negatively affect their physical health 16 , interpersonal relationships 17 , academic output, and work performance 18 , 19 , and may also contribute to program attrition 20 , 21 , 22 . As many as 30 to 50% of Ph.D. students drop out of their programs, depending on the country and discipline 23 , 24 , 25 , 26 , 27 . Further, while mental health problems among Ph.D. students raise concerns for the wellbeing of the individuals themselves and their personal networks, they also have broader repercussions for their institutions and academia as a whole 22 .

Despite the potential public health significance of this problem, most evidence syntheses on student mental health have focused on undergraduate students 28 , 29 or graduate students in professional degree programs (e.g., medical students) 30 . In non-systematic summaries, estimates of the prevalence of clinically significant depressive symptoms among Ph.D. students vary considerably 31 , 32 , 33 . Reliable estimates of depression and other mental health problems among Ph.D. students are needed to inform preventive, screening, or treatment efforts. To address this gap in the literature, we conducted a systematic review and meta-analysis to explore patterns of depression, anxiety, and suicidal ideation among Ph.D. students.

Flowchart of included articles.

The evidence search yielded 886 articles, of which 286 were excluded as duplicates (Fig.  1 ). An additional nine articles were identified through reference lists or grey literature reports published on university websites. Following a title/abstract review and subsequent full-text review, 520 additional articles were excluded.

Of the 89 remaining articles, 74 were unclear about their definition of graduate students or grouped Ph.D. and non-Ph.D. students without disaggregating the estimates by degree level. We obtained contact information for the authors of most of these articles (69 [93%]), requesting additional data. Three authors clarified that their study samples only included Ph.D. students 34 , 35 , 36 . Fourteen authors confirmed that their study samples included both Ph.D. and non-Ph.D. students but provided us with data on the subsample of Ph.D. students 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 . Where authors clarified that the sample was limited to graduate students in non-doctoral degree programs, did not provide additional data on the subsample of Ph.D. students, or did not reply to our information requests, we excluded the studies due to insufficient information (Supplementary Table S1 ).

Ultimately, 32 articles describing the findings of 29 unique studies were identified and included in the review 16 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 (Table 1 ). Overall, 26 studies measured depression, 19 studies measured anxiety, and six studies measured suicidal ideation. Three pairs of articles reported data on the same sample of Ph.D. students 33 , 38 , 45 , 51 , 53 , 56 and were therefore grouped in Table 1 and reported as three studies. Publication dates ranged from 1979 to 2019, but most articles (22/32 [69%]) were published after 2015. Most studies were conducted in the United States (20/29 [69%]), with additional studies conducted in Australia, Belgium, China, Iran, Mexico, and South Korea. Two studies were conducted in cross-national settings representing 48 additional countries. None were conducted in sub-Saharan Africa or South America. Most studies included students completing their degrees in a mix of disciplines (17/29 [59%]), while 12 studies were limited to students in a specific field (e.g., biomedicine, education). The median sample size was 172 students (interquartile range [IQR], 68–654; range, 6–6405). Seven studies focused on mental health outcomes in demographic subgroups, including ethnic or racialized minority students 37 , 41 , 43 , international students 47 , 50 , and sexual and gender minority students 42 , 54 .

In all, 16 studies reported the prevalence of depression among a total of 23,469 Ph.D. students (Fig.  2 ; range, 10–47%). Of these, the most widely used depression scales were the PHQ-9 (9 studies) and variants of the Center for Epidemiologic Studies-Depression scale (CES-D, 4 studies) 63 , and all studies assessed clinically significant symptoms of depression over the past one to two weeks. Three of these studies reported findings based on data from different survey years of the same parent study (the Healthy Minds Study) 40 , 42 , 43 , but due to overlap in the survey years reported across articles, these data were pooled. Most of these studies were based on data collected through online surveys (13/16 [81%]). Ten studies (63%) used random or systematic sampling, four studies (25%) used convenience sampling, and two studies (13%) used multiple sampling techniques.

Pooled estimate of the proportion of Ph.D. students with clinically significant symptoms of depression.

The estimated proportion of Ph.D. students assessed as having clinically significant symptoms of depression was 0.24 (95% confidence interval [CI], 0.18–0.31; 95% predictive interval [PI], 0.04–0.54), with significant evidence of between-study heterogeneity (I 2  = 98.75%). A subgroup analysis restricted to the twelve studies conducted in the United States yielded similar findings (pooled estimate [ES] = 0.23; 95% CI, 0.15–0.32; 95% PI, 0.01–0.60), with no appreciable difference in heterogeneity (I 2  = 98.91%). A subgroup analysis restricted to the studies that used the PHQ-9 to assess depression yielded a slightly lower prevalence estimate and a slight reduction in heterogeneity (ES = 0.18; 95% CI, 0.14–0.22; 95% PI, 0.07–0.34; I 2  = 90.59%).

Nine studies reported the prevalence of clinically significant symptoms of anxiety among a total of 15,626 Ph.D. students (Fig.  3 ; range 4–49%). Of these, the most widely used anxiety scale was the 7-item Generalized Anxiety Disorder scale (GAD-7, 5 studies) 64 . Data from three of the Healthy Minds Study articles were pooled into two estimates, because the scale used to measure anxiety changed midway through the parent study (i.e., the Patient Health Questionnaire-Generalized Anxiety Disorder [PHQ-GAD] scale was used from 2007 to 2012 and then switched to the GAD-7 in 2013 40 ). Most studies (8/9 [89%]) assessed clinically significant symptoms of anxiety over the past two to four weeks, with the one remaining study measuring anxiety over the past year. Again, most of these studies were based on data collected through online surveys (7/9 [78%]). Five studies (56%) used random or systematic sampling, two studies (22%) used convenience sampling, and two studies (22%) used multiple sampling techniques.

Pooled estimate of the proportion of Ph.D. students with clinically significant symptoms of anxiety.

The estimated proportion of Ph.D. students assessed as having anxiety was 0.17 (95% CI, 0.12–0.23; 95% PI, 0.02–0.41), with significant evidence of between-study heterogeneity (I 2  = 98.05%). The subgroup analysis restricted to the five studies conducted in the United States yielded a slightly lower proportion of students assessed as having anxiety (ES = 0.14; 95% CI, 0.08–0.20; 95% PI, 0.00–0.43), with no appreciable difference in heterogeneity (I 2  = 98.54%).

Six studies reported the prevalence of suicidal ideation (range, 2–12%), but the recall windows varied greatly (e.g., ideation within the past 2 weeks vs. past year), precluding pooled estimation.

Additional stratified pooled estimates could not be obtained. One study of Ph.D. students across 54 countries found that phase of study was a significant moderator of mental health, with students in the comprehensive examination and dissertation phases more likely to experience distress compared with students primarily engaged in coursework 59 . Other studies identified a higher prevalence of mental ill-health among women 54 ; lesbian, gay, bisexual, transgender, and queer (LGBTQ) students 42 , 54 , 60 ; and students with multiple intersecting identities 54 .

Several studies identified correlates of mental health problems including: project- and supervisor-related issues, stress about productivity, and self-doubt 53 , 62 ; uncertain career prospects, poor living conditions, financial stressors, lack of sleep, feeling devalued, social isolation, and advisor relationships 61 ; financial challenges 38 ; difficulties with work-life balance 58 ; and feelings of isolation and loneliness 52 . Despite these challenges, help-seeking appeared to be limited, with only about one-quarter of Ph.D. students reporting mental health problems also reporting that they were receiving treatment 40 , 52 .

Risk of bias

Twenty-one of 32 articles were assessed as having low risk of bias (Supplementary Table S2 ). Five articles received one point for all five categories on the risk of bias assessment (lowest risk of bias), and one article received no points (highest risk). The mean risk of bias score was 3.22 (standard deviation, 1.34; median, 4; IQR, 2–4). Restricting the estimation sample to 12 studies assessed as having low risk of bias, the estimated proportion of Ph.D. students with depression was 0.25 (95% CI, 0.18–0.33; 95% PI, 0.04–0.57; I 2  = 99.11%), nearly identical to the primary estimate, with no reduction in heterogeneity. The estimated proportion of Ph.D. students with anxiety, among the 7 studies assessed as having low risk of bias, was 0.12 (95% CI, 0.07–0.17; 95% PI, 0.01–0.34; I 2  = 98.17%), again with no appreciable reduction in heterogeneity.

In our meta-analysis of 16 studies representing 23,469 Ph.D. students, we estimated that the pooled prevalence of clinically significant symptoms of depression was 24%. This estimate is consistent with estimated prevalence rates in other high-stress biomedical trainee populations, including medical students (27%) 30 , resident physicians (29%) 65 , and postdoctoral research fellows (29%) 66 . In the sample of nine studies representing 15,626 Ph.D. students, we estimated that the pooled prevalence of clinically significant symptoms of anxiety was 17%. While validated screening instruments tend to over-identify cases of depression (relative to structured clinical interviews) by approximately a factor of two 67 , 68 , our findings nonetheless point to a major public health problem among Ph.D. students. Available data suggest that the prevalence of depressive and anxiety disorders in the general population ranges from 5 to 7% worldwide 69 , 70 . In contrast, prevalence estimates of major depressive disorder among young adults have ranged from 13% (for young adults between the ages of 18 and 29 years in the 2012–2013 National Epidemiologic Survey on Alcohol and Related Conditions III 71 ) to 15% (for young adults between the ages of 18 and 25 in the 2019 U.S. National Survey on Drug Use and Health 72 ). Likewise, the prevalence of generalized anxiety disorder was estimated at 4% among young adults between the ages of 18 and 29 in the 2001–03 U.S. National Comorbidity Survey Replication 73 . Thus, even accounting for potential upward bias inherent in these studies’ use of screening instruments, our estimates suggest that the rates of recent clinically significant symptoms of depression and anxiety are greater among Ph.D. students compared with young adults in the general population.

Further underscoring the importance of this public health issue, Ph.D. students face unique stressors and uncertainties that may put them at increased risk for mental health and substance use problems. Students grapple with competing responsibilities, including coursework, teaching, and research, while also managing interpersonal relationships, social isolation, caregiving, and financial insecurity 3 , 10 . Increasing enrollment in doctoral degree programs has not been matched with a commensurate increase in tenure-track academic job opportunities, intensifying competition and pressure to find employment post-graduation 5 . Advisor-student power relations rarely offer options for recourse if and when such relationships become strained, particularly in the setting of sexual harassment, unwanted sexual attention, sexual coercion, and rape 74 , 75 , 76 , 77 , 78 . All of these stressors may be magnified—and compounded by stressors unrelated to graduate school—for subgroups of students who are underrepresented in doctoral degree programs and among whom mental health problems are either more prevalent and/or undertreated compared with the general population, including Black, indigenous, and other people of color 13 , 79 , 80 ; women 81 , 82 ; first-generation students 14 , 15 ; people who identify as LGBTQ 83 , 84 , 85 ; people with disabilities; and people with multiple intersecting identities.

Structural- and individual-level interventions will be needed to reduce the burden of mental ill-health among Ph.D. students worldwide 31 , 86 . Despite the high prevalence of mental health and substance use problems 87 , Ph.D. students demonstrate low rates of help-seeking 40 , 52 , 88 . Common barriers to help-seeking include fears of harming one’s academic career, financial insecurity, lack of time, and lack of awareness 89 , 90 , 91 , as well as health care systems-related barriers, including insufficient numbers of culturally competent counseling staff, limited access to psychological services beyond time-limited psychotherapies, and lack of programs that address the specific needs either of Ph.D. students in general 92 or of Ph.D. students belonging to marginalized groups 93 , 94 . Structural interventions focused solely on enhancing student resilience might include programs aimed at reducing stigma, fostering social cohesion, and reducing social isolation, while changing norms around help-seeking behavior 95 , 96 . However, structural interventions focused on changing stressogenic aspects of the graduate student environment itself are also needed 97 , beyond any enhancements to Ph.D. student resilience, including: undercutting power differentials between graduate students and individual faculty advisors, e.g., by diffusing power among multiple faculty advisors; eliminating racist, sexist, and other discriminatory behaviors by faculty advisors 74 , 75 , 98 ; valuing mentorship and other aspects of “invisible work” that are often disproportionately borne by women faculty and faculty of color 99 , 100 ; and training faculty members to emphasize the dignity of, and adequately prepare Ph.D. students for, non-academic careers 101 , 102 .

Our findings should be interpreted with several limitations in mind. First, the pooled estimates are characterized by a high degree of heterogeneity, similar to meta-analyses of depression prevalence in other populations 30 , 65 , 103 , 104 , 105 . Second, we were only able to aggregate depression prevalence across 16 studies and anxiety prevalence across nine studies (the majority of which were conducted in the U.S.) – far fewer than the 183 studies included in a meta-analysis of depression prevalence among medical students 30 and the 54 studies included in a meta-analysis of resident physicians 65 . These differences underscore the need for more rigorous study in this critical area. Many articles were either excluded from the review or from the meta-analyses for not meeting inclusion criteria or not reporting relevant statistics. Future research in this area should ensure the systematic collection of high-quality, clinically relevant data from a comprehensive set of institutions, across disciplines and countries, and disaggregated by graduate student type. As part of conducting research and addressing student mental health and wellbeing, university deans, provosts, and chancellors should partner with national survey and program institutions (e.g., Graduate Student Experience in the Research University [gradSERU] 106 , the American College Health Association National College Health Assessment [ACHA-NCHA], and HealthyMinds). Furthermore, federal agencies that oversee health and higher education should provide resources for these efforts, and accreditation agencies should require monitoring of mental health and programmatic responses to stressors among Ph.D. students.

Third, heterogeneity in reporting precluded a meta-analysis of the suicidality outcomes among the few studies that reported such data. While reducing the burden of mental health problems among graduate students is an important public health aim in itself, more research into understanding non-suicidal self-injurious behavior, suicide attempts, and completed suicide among Ph.D. students is warranted. Fourth, it is possible that the grey literature reports included in our meta-analysis are more likely to be undertaken at research-intensive institutions 52 , 60 , 61 . However, the direction of bias is unpredictable: mental health problems among Ph.D. students in research-intensive environments may be more prevalent due to detection bias, but such institutions may also have more resources devoted to preventive, screening, or treatment efforts 92 . Fifth, inclusion in this meta-analysis and systematic review was limited to those based on community samples. Inclusion of clinic-based samples, or of studies conducted before or after specific milestones (e.g., the qualifying examination or dissertation prospectus defense), likely would have yielded even higher pooled prevalence estimates of mental health problems. And finally, few studies provided disaggregated data according to sociodemographic factors, stage of training (e.g., first year, pre-prospectus defense, all-but-dissertation), or discipline of study. These factors might be investigated further for differences in mental health outcomes.

Clinically significant symptoms of depression and anxiety are pervasive among graduate students in doctoral degree programs, but these are understudied relative to other trainee populations. Structural and clinical interventions to systematically monitor and promote the mental health and wellbeing of Ph.D. students are urgently needed.

This systematic review and meta-analysis follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach (Supplementary Table S3 ) 107 . This study was based on data collected from publicly available bibliometric databases and did not require ethical approval from our institutional review boards.

Eligibility criteria

Studies were included if they provided data on either: (a) the number or proportion of Ph.D. students with clinically significant symptoms of depression or anxiety, ascertained using a validated scale; or (b) the mean depression or anxiety symptom severity score and its standard deviation among Ph.D. students. Suicidal ideation was examined as a secondary outcome.

We excluded studies that focused on graduate students in non-doctoral degree programs (e.g., Master of Public Health) or professional degree programs (e.g., Doctor of Medicine, Juris Doctor) because more is known about mental health problems in these populations 30 , 108 , 109 , 110 and because Ph.D. students face unique uncertainties. To minimize the potential for upward bias in our pooled prevalence estimates, we excluded studies that recruited students from campus counseling centers or other clinic-based settings. Studies that measured affective states, or state anxiety, before or after specific events (e.g., terrorist attacks, qualifying examinations) were also excluded.

If articles described the study sample in general terms (i.e., without clarifying the degree level of the participants), we contacted the authors by email for clarification. Similarly, if articles pooled results across graduate students in doctoral and non-doctoral degree programs (e.g., reporting a single estimate for a mixed sample of graduate students), we contacted the authors by email to request disaggregated data on the subsample of Ph.D. students. If authors did not reply after two contact attempts spaced over 2 months, or were unable to provide these data, we excluded these studies from further consideration.

Search strategy and data extraction

PubMed, Embase, PsycINFO, ERIC, and Business Source Complete were searched from inception of each database to November 5, 2019. The search strategy included terms related to mental health symptoms (e.g., depression, anxiety, suicide), the study population (e.g., graduate, doctoral), and measurement category (e.g., depression, Columbia-Suicide Severity Rating Scale) (Supplementary Table S4 ). In addition, we searched the reference lists and the grey literature.

After duplicates were removed, we screened the remaining titles and abstracts, followed by a full-text review. We excluded articles following the eligibility criteria listed above (i.e., those that were not focused on Ph.D. students; those that did not assess depression and/or anxiety using a validated screening tool; those that did not report relevant statistics of depression and/or anxiety; and those that recruited students from clinic-based settings). Reasons for exclusion were tracked at each stage. Following selection of included articles, two members of the research team extracted data and conducted risk of bias assessments. Discrepancies were discussed with a third member of the research team. Key extraction variables included: study design, geographic region, sample size, response rate, demographic characteristics of the sample, screening instrument(s) used for assessment, mean depression or anxiety symptom severity score (and its standard deviation), and the number (or proportion) of students experiencing clinically significant symptoms of depression or anxiety.

Risk of bias assessment

Following prior work 30 , 65 , the Newcastle–Ottawa Scale 111 was adapted and used to assess risk of bias in the included studies. Each study was assessed across 5 categories: sample representativeness, sample size, non-respondents, ascertainment of outcomes, and quality of descriptive statistics reporting (Supplementary Information S5 ). Studies were judged as having either low risk of bias (≥ 3 points) or high risk of bias (< 3 points).

Analysis and synthesis

Before pooling the estimated prevalence rates across studies, we first transformed the proportions using a variance-stabilizing double arcsine transformation 112 . We then computed pooled estimates of prevalence using a random effects model 113 . Study specific confidence intervals were estimated using the score method 114 , 115 . We estimated between-study heterogeneity using the I 2 statistic 116 . In an attempt to reduce the extent of heterogeneity, we re-estimated pooled prevalence restricting the analysis to studies conducted in the United States and to studies in which depression assessment was based on the 9-item Patient Health Questionnaire (PHQ-9) 117 . All analyses were conducted using Stata (version 16; StataCorp LP, College Station, Tex.). Where heterogeneity limited our ability to summarize the findings using meta-analysis, we synthesized the data using narrative review.

Woolston, C. Why mental health matters. Nature 557 , 129–131 (2018).

Article   ADS   CAS   Google Scholar  

Woolston, C. A love-hurt relationship. Nature 550 , 549–552 (2017).

Article   Google Scholar  

Woolston, C. PhD poll reveals fear and joy, contentment and anguish. Nature 575 , 403–406 (2019).

Article   ADS   CAS   PubMed   Google Scholar  

Byrom, N. COVID-19 and the research community: The challenges of lockdown for early-career researchers. Elife 9 , e59634 (2020).

Article   PubMed   PubMed Central   Google Scholar  

Alberts, B., Kirschner, M. W., Tilghman, S. & Varmus, H. Rescuing US biomedical research from its systemic flaws. Proc. Natl. Acad. Sci. USA 111 , 5773–5777 (2014).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

McDowell, G. S. et al. Shaping the future of research: A perspective from junior scientists. F1000Res 3 , 291 (2014).

Article   PubMed   Google Scholar  

Petersen, A. M., Riccaboni, M., Stanley, H. E. & Pammoli, F. Persistence and uncertainty in the academic career. Proc. Natl. Acad. Sci. USA 109 , 5213–5218 (2012).

Leshner, A. I. Rethinking graduate education. Science 349 , 349 (2015).

National Academies of Sciences Engineering and Medicine. Graduate STEM Education for the 21st Century (National Academies Press, 2018).

Google Scholar  

Charles, S. T., Karnaze, M. M. & Leslie, F. M. Positive factors related to graduate student mental health. J. Am. Coll. Health https://doi.org/10.1080/07448481.2020.1841207 (2021).

Riddle, E. S., Niles, M. T. & Nickerson, A. Prevalence and factors associated with food insecurity across an entire campus population. PLoS ONE 15 , e0237637 (2020).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Soldavini, J., Berner, M. & Da Silva, J. Rates of and characteristics associated with food insecurity differ among undergraduate and graduate students at a large public university in the Southeast United States. Prev. Med. Rep. 14 , 100836 (2019).

Clark, U. S. & Hurd, Y. L. Addressing racism and disparities in the biomedical sciences. Nat. Hum. Behav. 4 , 774–777 (2020).

Gardner, S. K. The challenges of first-generation doctoral students. New Dir. High. Educ. 2013 , 43–54 (2013).

Seay, S. E., Lifton, D. E., Wuensch, K. L., Bradshaw, L. K. & McDowelle, J. O. First-generation graduate students and attrition risks. J. Contin. High. Educ. 56 , 11–25 (2008).

Rummell, C. M. An exploratory study of psychology graduate student workload, health, and program satisfaction. Prof. Psychol. Res. Pract. 46 , 391–399 (2015).

Salzer, M. S. A comparative study of campus experiences of college students with mental illnesses versus a general college sample. J. Am. Coll. Health 60 , 1–7 (2012).

Hysenbegasi, A., Hass, S. & Rowland, C. The impact of depression on the academic productivity of university students. J. Ment. Health Policy Econ. 8 , 145–151 (2005).

PubMed   Google Scholar  

Harvey, S. et al. Depression and work performance: An ecological study using web-based screening. Occup. Med. (Lond.) 61 , 209–211 (2011).

Article   CAS   Google Scholar  

Eisenberg, D., Golberstein, E. & Hunt, J. B. Mental health and academic success in college. BE J. Econ. Anal. Policy 9 , 40 (2009).

Lovitts, B. E. Who is responsible for graduate student attrition--the individual or the institution? Toward an explanation of the high and persistent rate of attrition. In:  Annual Meeting of the American Education Research Association (New York, 1996). Available at: https://eric.ed.gov/?id=ED399878.

Gardner, S. K. Student and faculty attributions of attrition in high and low-completing doctoral programs in the United States. High. Educ. 58 , 97–112 (2009).

Lovitts, B. E. Leaving the Ivory Tower: The Causes and Consequences of Departure from Doctoral Study (Rowman & Littlefield Publishers, 2001).

Rigler Jr, K. L., Bowlin, L. K., Sweat, K., Watts, S. & Throne, R. Agency, socialization, and support: a critical review of doctoral student attrition. In:  Proceedings of the Third International Conference on Doctoral Education: Organizational Leadership and Impact , University of Central Florida, Orlando, (2017).

Golde, C. M. The role of the department and discipline in doctoral student attrition: Lessons from four departments. J. High. Educ. 76 , 669–700 (2005).

Council of Graduate Schools. PhD Completion and Attrition: Analysis of Baseline Program Data from the PhD Completion Project (Council of Graduate Schools, 2008).

National Research Council. A Data-Based Assessment of Research-Doctorate Programs in the United States (The National Academies Press, 2011).

Akhtar, P. et al. Prevalence of depression among university students in low and middle income countries (LMICs): A systematic review and meta-analysis. J. Affect. Disord. 274 , 911–919 (2020).

Mortier, P. et al. The prevalence of suicidal thoughts and behaviours among college students: A meta-analysis. Psychol. Med. 48 , 554–565 (2018).

Article   CAS   PubMed   Google Scholar  

Rotenstein, L. S. et al. Prevalence of depression, depressive symptoms, and suicidal ideation among medical students: A systematic review and meta-analysis. JAMA 316 , 2214–2236 (2016).

Tsai, J. W. & Muindi, F. Towards sustaining a culture of mental health and wellness for trainees in the biosciences. Nat. Biotechnol. 34 , 353–355 (2016).

Levecque, K., Anseel, F., De Beuckelaer, A., Van der Heyden, J. & Gisle, L. Work organization and mental health problems in PhD students. Res. Policy 46 , 868–879 (2017).

Nagy, G. A. et al. Burnout and mental health problems in biomedical doctoral students. CBE Life Sci. Educ. 18 , 1–14 (2019).

Garcia-Williams, A., Moffitt, L. & Kaslow, N. J. Mental health and suicidal behavior among graduate students. Acad. Psychiatry 28 , 554–560 (2014).

Sheldon, K. M. Emotionality differences between artists and scientists. J. Res. Pers. 28 , 481–491 (1994).

Lightstone, S. N., Swencionis, C. & Cohen, H. W. The effect of bioterrorism messages on anxiety levels. Int. Q. Community Health Educ. 24 , 111–122 (2006).

Clark, C. R., Mercer, S. H., Zeigler-Hill, V. & Dufrene, B. A. Barriers to the success of ethnic minority students in school psychology graduate programs. School Psych. Rev. 41 , 176–192 (2012).

Eisenberg, D., Gollust, S. E., Golberstein, E. & Hefner, J. L. Prevalence and correlates of depression, anxiety, and suicidality among university students. Am. J. Orthopsychiatry 77 , 534–542 (2007).

Farrer, L. M., Gulliver, A., Bennett, K., Fassnacht, D. B. & Griffiths, K. M. Demographic and psychosocial predictors of major depression and generalised anxiety disorder in Australian university students. BMC Psychiatry 16 , 241 (2016).

Lipson, S. K., Zhou, S., Wagner, B. III., Beck, K. & Eisenberg, D. Major differences: Variations in undergraduate and graduate student mental health and treatment utilization across academic disciplines. J. Coll. Stud. Psychother. 30 , 23–41 (2016).

Lilly, F. R. W. et al. The influence of racial microaggressions and social rank on risk for depression among minority graduate and professional students. Coll. Stud. J. 52 , 86–104 (2018).

Lipson, S. K., Raifman, J., Abelson, S. & Reisner, S. L. Gender minority mental health in the U.S.: Results of a national survey on college campuses. Am. J. Prev. Med. 57 , 293–301 (2019).

Lipson, S. K., Kern, A., Eisenberg, D. & Breland-Noble, A. M. Mental health disparities among college students of color. J. Adolesc. Health 63 , 348–356 (2018).

Baker, A. J. L. & Chambers, J. Adult recall of childhood exposure to parental conflict: Unpacking the black box of parental alienation. J. Divorce Remarriage 52 , 55–76 (2011).

Golberstein, E., Eisenberg, D. & Gollust, S. E. Perceived stigma and mental health care seeking. Psychiatr. Serv. 59 , 392–399 (2008).

Hindman, R. K., Glass, C. R., Arnkoff, D. B. & Maron, D. D. A comparison of formal and informal mindfulness programs for stress reduction in university students. Mindfulness 6 , 873–884 (2015).

Hirai, R., Frazier, P. & Syed, M. Psychological and sociocultural adjustment of first-year international students: Trajectories and predictors. J. Couns. Psychol. 62 , 438–452 (2015).

Lee, J. S. & Jeong, B. Having mentors and campus social networks moderates the impact of worries and video gaming on depressive symptoms: A moderated mediation analysis. BMC Public Health 14 , 1–12 (2014).

Corral-Frias, N. S., Velardez Soto, S. N., Frias-Armenta, M., Corona-Espinosa, A. & Watson, D. Concurrent validity and reliability of two short forms of the mood and anxiety symptom questionnaire in a student sample from Northwest Mexico. J. Psychopathol. Behav. Assess. 41 , 304–316 (2019).

Meghani, D. T. & Harvey, E. A. Asian Indian international students’ trajectories of depression, acculturation, and enculturation. Asian Am. J. Psychol. 7 , 1–14 (2016).

Barry, K. M., Woods, M., Martin, A., Stirling, C. & Warnecke, E. A randomized controlled trial of the effects of mindfulness practice on doctoral candidate psychological status. J. Am. Coll. Health 67 , 299–307 (2019).

Bolotnyy, V., Basilico, M. & Barreira, P. Graduate student mental health: lessons from American economics departments. J. Econ. Lit. (in press).

Barry, K. M., Woods, M., Warnecke, E., Stirling, C. & Martin, A. Psychological health of doctoral candidates, study-related challenges and perceived performance. High. Educ. Res. Dev. 37 , 468–483 (2018).

Boyle, K. M. & McKinzie, A. E. The prevalence and psychological cost of interpersonal violence in graduate and law school. J. Interpers. Violence   36 , 6319-6350 (2021).

Heinrich, D. L. The causal influence of anxiety on academic achievement for students of differing intellectual ability. Appl. Psychol. Meas. 3 , 351–359 (1979).

Hish, A. J. et al. Applying the stress process model to stress-burnout and stress-depression relationships in biomedical doctoral students: A cross-sectional pilot study. CBE Life Sci. Educ. 18 , 1–11 (2019).

Jamshidi, F. et al. A cross-sectional study of psychiatric disorders in medical sciences students. Mater. Sociomed. 29 , 188–191 (2017).

Liu, C. et al. Prevalence and associated factors of depression and anxiety among doctoral students: The mediating effect of mentoring relationships on the association between research self-efficacy and depression/anxiety. Psychol. Res. Behav. Manag. 12 , 195–208 (2019).

Sverdlik, A. & Hall, N. C. Not just a phase: Exploring the role of program stage on well-being and motivation in doctoral students. J. Adult Contin. Educ. 26 , 1–28 (2019).

University of California Office of the President. The University of California Graduate student Well-Being Survey Report (University of California, 2017).

The Graduate Assembly. Graduate Student Happiness & Well-Being Report (University of California at Berkeley, 2014).

Richardson, C. M., Trusty, W. T. & George, K. A. Trainee wellness: Self-critical perfectionism, self-compassion, depression, and burnout among doctoral trainees in psychology. Couns. Psychol. Q. 33 , 187-198 (2020).

Radloff, L. S. The CES-D Scale: A self-report depression scale for research in the general population. Appl. Psychol. Meas. 1 , 385–401 (1977).

Spitzer, R. L., Kroenke, K., Williams, J. B. W. & Lowe, B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch. Intern. Med. 166 , 1092–1097 (2006).

Mata, D. A. et al. Prevalence of depression and depressive symptoms among residents physicians: A systematic review and meta-analysis. JAMA 314 , 2373–2383 (2015).

Gloria, C. T. & Steinhardt, M. A. Flourishing, languishing, and depressed postdoctoral fellows: Differences in stress, anxiety, and depressive symptoms. J. Postdoct. Aff. 3 , 1–9 (2013).

Levis, B. et al. Patient Health Questionnaire-9 scores do not accurately estimate depression prevalence: Individual participant data meta-analysis. J. Clin. Epidemiol. 122 , 115-128.e111 (2020).

Tsai, A. C. Reliability and validity of depression assessment among persons with HIV in sub-Saharan Africa: Systematic review and meta-analysis. J. Acquir. Immune Defic. Syndr. 66 , 503–511 (2014).

Baxter, A. J., Scott, K. M., Vos, T. & Whiteford, H. A. Global prevalence of anxiety disorders: A systematic review and meta-regression. Psychol. Med. 43 , 897–910 (2013).

Ferrari, A. et al. Global variation in the prevalence and incidence of major depressive disorder: A systematic review of the epidemiological literature. Psychol. Med. 43 , 471–481 (2013).

Hasin, D. S. et al. Epidemiology of adult DSM-5 major depressive disorder and its specifiers in the United States. JAMA Psychiatry   75 , 336–346 (2018).

US Substance Abuse and Mental Health Services Administration. Key Substance Use and Mental Health Indicators in the United States: Results from the 2019 National Survey on Drug Use and Health (Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration, 2020).

Kessler, R. C. et al. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch. Gen. Psychiatry 62 , 593–602 (2005).

Working Group report to the Advisory Committee to the NIH Director. Changing the Culture to End Sexual Harassment (U. S. National Institutes of Health, 2019).

National Academies of Sciences Engineering and Medicine. Sexual Harassment of Women: Climate, Culture, and Consequences in Academic Sciences, Engineering, and Medicine (The National Academies Press, 2018).

Wadman, M. A hidden history. Science 360 , 480–485 (2018).

Hockfield, S., Magley, V. & Yoshino, K. Report of the External Review Committee to Review Sexual Harassment at Harvard University (External Review Committee to Review Sexual Harassment at Harvard University, 2021).

Bartlett, T. & Gluckman, N. She left Harvard. He got to stay. Chronicle High. Educ. 64 , A14 (2021). Available at: https://www.chronicle.com/article/she-left-harvard-he-got-to-stay/.

Tseng, M. et al. Strategies and support for Black, Indigenous, and people of colour in ecology and evolutionary biology. Nat. Ecol. Evol. 4 , 1288–1290 (2020).

Williams, D. R. et al. Prevalence and distribution of major depressive disorder in African Americans, Caribbean blacks, and non-Hispanic whites: Results from the National Survey of American Life. Arch. Gen. Psychiatry   64 , 305–315 (2007).

Wu, A. H. Gender bias in rumors among professionals: An identity-based interpretation. Rev. Econ. Stat. 102 , 867–880 (2020).

Kessler, R. C. Epidemiology of women and depression. J. Affect. Disord. 74 , 5–13 (2003).

Mattheis, A., Cruz-Ramirez De Arellano, D. & Yoder, J. B. A model of queer STEM identity in the workplace. J. Homosex 67 , 1839–1863 (2020).

Semlyen, J., King, M., Varney, J. & Hagger-Johnson, G. Sexual orientation and symptoms of common mental disorder or low wellbeing: Combined meta-analysis of 12 UK population health surveys. BMC Psychiatry 16 , 1–19 (2016).

Lark, J. S. & Croteau, J. M. Lesbian, gay, and bisexual doctoral students’ mentoring relationships with faculty in counseling psychology: A qualitative study. Couns. Psychol. 26 , 754–776 (1998).

Jaremka, L. M. et al. Common academic experiences no one talks about: Repeated rejection, imposter syndrome, and burnout. Perspect Psychol Sci 15 , 519–543 (2020).

Allen, H. K. et al. Substance use and mental health problems among graduate students: Individual and program-level correlates. J. Am. Coll. Health https://doi.org/10.1080/07448481.2020.1725020 (2020).

Turner, A. & Berry, T. Counseling center contributions to student retention and graduation: A longitudinal assessment. J. Coll. Stud. Dev. 41 , 627–636 (2000).

Dyrbye, L. N., Thomas, M. R. & Shanafelt, T. D. Medical student distress: Causes, consequences, and proposed solutions. Mayo Clin. Proc. 80 , 1613–1622 (2005).

Tija, J., Givens, J. L. & Shea, J. A. Factors associated with undertreatment of medical student depression. J. Am. Coll. Health 53 , 219–224 (2005).

Dearing, R., Maddux, J. & Tangney, J. Predictors of psychological help seeking in clinical and counseling psychology graduate students. Prof. Psychol. Res. Pract. 36 , 323–329 (2005).

Langin, K. Amid concerns about grad student mental health, one university takes a novel approach. Science https://doi.org/10.1126/science.caredit.aay7113 (2019).

Guillory, D. Combating anti-blackness in the AI community. arXiv , arXiv:2006.16879 (2020).

Galán, C. A. et al. A call to action for an antiracist clinical science. J. Clin. Child Adolesc. Psychol   50 , 12-57 (2021).

Wyman, P. A. et al. Effect of the Wingman-Connect upstream suicide prevention program for air force personnel in training: A cluster randomized clinical trial. JAMA Netw Open 3 , e2022532 (2020).

Knox, K. L. et al. The US Air Force Suicide Prevention Program: Implications for public health policy. Am. J. Public Health 100 , 2457–2463 (2010).

Inclusive Climate Subcommittee of the Government Department Climate Change Committee. Government Department Climate Change: Final Report and Recommendations (Government Department, Harvard University, 2019).

Inclusive Climate Subcommittee of the Government Department Climate Change Committee. Government Department Climate Survey Report (Government Department, Harvard University, 2019).

Magoqwana, B., Maqabuka, Q. & Tshoaedi, M. “Forced to care” at the neoliberal university: Invisible labour as academic labour performed by Black women academics in the South African university. S. Afr. Rev. Sociol. 50 , 6–21 (2019).

Jones, H. A., Perrin, P. B., Heller, M. B., Hailu, S. & Barnett, C. Black psychology graduate students’ lives matter: Using informal mentoring to create an inclusive climate amidst national race-related events. Prof. Psychol. Res. Pract. 49 , 75–82 (2018).

Mathur, A., Meyers, F. J., Chalkley, R., O’Brien, T. C. & Fuhrmann, C. N. Transforming training to reflect the workforce. Sci. Transl. Med. 7 , 285 (2015).

Scharff, V. Advice: Prepare your Ph.D.s for diverse career paths. Chronicle High. Educ. 65 , 30 (2018).

Beattie, T. S., Smilenova, B., Krishnaratne, S. & Mazzuca, A. Mental health problems among female sex workers in low- and middle-income countries: A systematic review and meta-analysis. PLoS Med. 17 , e1003297 (2020).

Ismail, Z. et al. Prevalence of depression in patients with mild cognitive impairment: A systematic review and meta-analysis. JAMA Psychiatry   74 , 58–67 (2017).

Lim, G. Y. et al. Prevalence of depression in the community from 30 countries between 1994 and 2014. Sci. Rep. 8 , 1–10 (2018).

Article   ADS   Google Scholar  

Jones-White, D. R., Soria, K. M., Tower, E. K. B. & Horner, O. G. Factors associated with anxiety and depression among U.S. doctoral students: Evidence from the gradSERU survey. J. Am. Coll. Health https://doi.org/10.1080/07448481.2020.1865975 (2021).

Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann. Intern. Med. 151 , 264–269 (2009).

Helmers, K. F., Danoff, D., Steinert, Y., Leyton, M. & Young, S. N. Stress and depressed mood in medical students, law students, and graduate students at McGill University. Acad. Med. 72 , 708–714 (1997).

Rabkow, N. et al. Facing the truth: A report on the mental health situation of German law students. Int. J. Law Psychiatry 71 , 101599 (2020).

Bergin, A. & Pakenham, K. Law student stress: Relationships between academic demands, social isolation, career pressure, study/life imbalance and adjustment outcomes in law students. Psychiatr. Psychol. Law 22 , 388–406 (2015).

Stang, A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur. J. Epidemiol. 25 , 603–605 (2010).

Freeman, M. F. & Tukey, J. W. Transformations related to the angular and the square root. Ann. Math. Stat. 21 , 607–611 (1950).

Article   MathSciNet   MATH   Google Scholar  

DerSimonian, R. & Laird, N. Meta-analysis in clinical trials. Control Clin. Trials 7 , 177–188 (1986).

Wilson, E. B. Probable inference, the law of succession, and statistical inference. J. Am. Stat. Assoc. 22 , 209–212 (1927).

Newcombe, R. G. Two-sided confidence intervals for the single proportion: Comparison of seven methods. Stat. Med. 17 , 857–872 (1998).

Higgins, J. P. T. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21 , 1539–1558 (2002).

Kroenke, K., Spitzer, R. L. & Williams, J. B. W. The PHQ-9: Validity of a brief depression severity measure. J. Gen. Intern. Med. 16 , 606–613 (2001).

Download references

Acknowledgements

We thank the following investigators for generously sharing their time and/or data: Gordon J. G. Asmundson, Ph.D., Amy J. L. Baker, Ph.D., Hillel W. Cohen, Dr.P.H., Alcir L. Dafre, Ph.D., Deborah Danoff, M.D., Daniel Eisenberg, Ph.D., Lou Farrer, Ph.D., Christy B. Fraenza, Ph.D., Patricia A. Frazier, Ph.D., Nadia Corral-Frías, Ph.D., Hanga Galfalvy, Ph.D., Edward E. Goldenberg, Ph.D., Robert K. Hindman, Ph.D., Jürgen Hoyer, Ph.D., Ayako Isato, Ph.D., Azharul Islam, Ph.D., Shanna E. Smith Jaggars, Ph.D., Bumseok Jeong, M.D., Ph.D., Ju R. Joeng, Nadine J. Kaslow, Ph.D., Rukhsana Kausar, Ph.D., Flavius R. W. Lilly, Ph.D., Sarah K. Lipson, Ph.D., Frances Meeten, D.Phil., D.Clin.Psy., Dhara T. Meghani, Ph.D., Sterett H. Mercer, Ph.D., Masaki Mori, Ph.D., Arif Musa, M.D., Shizar Nahidi, M.D., Ph.D., Arthur M. Nezu, Ph.D., D.H.L., Angelo Picardi, M.D., Nicole E. Rossi, Ph.D., Denise M. Saint Arnault, Ph.D., Sagar Sharma, Ph.D., Bryony Sheaves, D.Clin.Psy., Kennon M. Sheldon, Ph.D., Daniel Shepherd, Ph.D., Keisuke Takano, Ph.D., Sara Tement, Ph.D., Sherri Turner, Ph.D., Shawn O. Utsey, Ph.D., Ron Valle, Ph.D., Caleb Wang, B.S., Pengju Wang, Katsuyuki Yamasaki, Ph.D.

A.C.T. acknowledges funding from the Sullivan Family Foundation. This paper does not reflect an official statement or opinion from the County of San Mateo.  

Author information

Authors and affiliations.

Center for Global Health, Massachusetts General Hospital, Boston, MA, USA

Emily N. Satinsky & Alexander C. Tsai

San Mateo County Behavioral Health and Recovery Services, San Mateo, CA, USA

Tomoki Kimura

Department of Epidemiology and Population Health, Stanford University, Palo Alto, CA, USA

Mathew V. Kiang

Center for Population Health Sciences, Stanford University School of Medicine, Palo Alto, CA, USA

Harvard Society of Fellows, Harvard University, Cambridge, MA, USA

Rediet Abebe

Department of Electrical Engineering and Computer Science, University of California Berkeley, Berkeley, CA, USA

Department of Economics, Hankamer School of Business, Baylor University, Waco, TX, USA

Scott Cunningham

Department of Sociology, Washington University in St. Louis, St. Louis, MO, USA

Department of Microbiology, Immunology, and Molecular Genetics, Institute for Quantitative and Computational Biosciences, University of California Los Angeles, Los Angeles, CA, USA

Xiaofei Lin

Departments of Newborn Medicine and Psychiatry, Brigham and Women’s Hospital, Boston, MA, USA

Cindy H. Liu

Harvard Medical School, Boston, MA, USA

Cindy H. Liu & Alexander C. Tsai

Centre for Global Health, Edinburgh Medical School, Usher Institute, University of Edinburgh, Edinburgh, Scotland, UK

Department of Psychiatry, University of Michigan, Ann Arbor, MI, USA

Department of Global Health, Institute for Life Course Health Research, Stellenbosch University, Cape Town, South Africa

Mark Tomlinson

School of Nursing and Midwifery, Queens University, Belfast, UK

Fielding School of Public Health, Los Angeles Area Health Services Research Training Program, University of California Los Angeles, Los Angeles, CA, USA

Miranda Yaver

Mongan Institute, Massachusetts General Hospital, Boston, MA, USA

Alexander C. Tsai

You can also search for this author in PubMed   Google Scholar

Contributions

A.C.T. conceptualized the study and provided supervision. T.K. conducted the search. E.N.S. contacted authors for additional information not reported in published articles. E.N.S. and T.K. extracted data and performed the quality assessment appraisal. E.N.S. and A.C.T. conducted the statistical analysis and drafted the manuscript. T.K., M.V.K., R.A., S.C., H.L., X.L., C.H.L., I.R., S.S., M.T. and M.Y. contributed to the interpretation of the results. All authors provided critical feedback on drafts and approved the final manuscript.

Corresponding authors

Correspondence to Emily N. Satinsky or Alexander C. Tsai .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary information., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and Permissions

About this article

Cite this article.

Satinsky, E.N., Kimura, T., Kiang, M.V. et al. Systematic review and meta-analysis of depression, anxiety, and suicidal ideation among Ph.D. students. Sci Rep 11 , 14370 (2021). https://doi.org/10.1038/s41598-021-93687-7

Download citation

Received : 31 March 2021

Accepted : 24 June 2021

Published : 13 July 2021

DOI : https://doi.org/10.1038/s41598-021-93687-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

A single-center assessment of mental health and well-being in a biomedical sciences graduate program.

• Sarah K. Jachim
• Bradley S. Bowles
• Autumn J. Schulze

Nature Biotechnology (2023)

How to improve academic well-being: an analysis of the leveraging factors based on the Italian case

• Alice Tontodimamma
• Emiliano del Gobbo
• Antonio Aquino

Quality & Quantity (2023)

Mental Health Problems Among Graduate Students in Turkey: a Cross-Sectional Study

• Cafer Kılıç
• Faika Şanal Karahan

International Journal for the Advancement of Counselling (2023)

Suicidal affective risk among female college students: the impact of life satisfaction

• Dawei Huang
• Xianbin Wang

Current Psychology (2023)

A study in University of Ruhuna for investigating prevalence, risk factors and remedies for psychiatric illnesses among students

• Patikiri Arachchige Don Shehan Nilm Wijesekara

Scientific Reports (2022)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

University Libraries      University of Nevada, Reno

• Skill Guides
• Subject Guides

Systematic, Scoping, and Other Literature Reviews: Overview

• Project Planning

What Is a Systematic Review?

Regular literature reviews are simply summaries of the literature on a particular topic. A systematic review, however, is a comprehensive literature review conducted to answer a specific research question. Authors of a systematic review aim to find, code, appraise, and synthesize all of the previous research on their question in an unbiased and well-documented manner. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) outline the minimum amount of information that needs to be reported at the conclusion of a systematic review project. 

Other types of what are known as "evidence syntheses," such as scoping, rapid, and integrative reviews, have varying methodologies. While systematic reviews originated with and continue to be a popular publication type in medicine and other health sciences fields, more and more researchers in other disciplines are choosing to conduct evidence syntheses. 

This guide will walk you through the major steps of a systematic review and point you to key resources including Covidence, a systematic review project management tool. For help with systematic reviews and other major literature review projects, please send us an email at  [email protected] .

Getting Help with Reviews

Organization such as the Institute of Medicine recommend that you consult a librarian when conducting a systematic review. Librarians at the University of Nevada, Reno can help you:

• Understand best practices for conducting systematic reviews and other evidence syntheses in your discipline
• Choose and formulate a research question
• Decide which review type (e.g., systematic, scoping, rapid, etc.) is the best fit for your project
• Determine what to include and where to register a systematic review protocol
• Select search terms and develop a search strategy
• Identify databases and platforms to search
• Find the full text of articles and other sources
• Become familiar with free citation management (e.g., EndNote, Zotero)
• Get access to you and help using Covidence, a systematic review project management tool

Doing a Systematic Review

• Plan - This is the project planning stage. You and your team will need to develop a good research question, determine the type of review you will conduct (systematic, scoping, rapid, etc.), and establish the inclusion and exclusion criteria (e.g., you're only going to look at studies that use a certain methodology). All of this information needs to be included in your protocol. You'll also need to ensure that the project is viable - has someone already done a systematic review on this topic? Do some searches and check the various protocol registries to find out. 
• Identify - Next, a comprehensive search of the literature is undertaken to ensure all studies that meet the predetermined criteria are identified. Each research question is different, so the number and types of databases you'll search - as well as other online publication venues - will vary. Some standards and guidelines specify that certain databases (e.g., MEDLINE, EMBASE) should be searched regardless. Your subject librarian can help you select appropriate databases to search and develop search strings for each of those databases.  
• Evaluate - In this step, retrieved articles are screened and sorted using the predetermined inclusion and exclusion criteria. The risk of bias for each included study is also assessed around this time. It's best if you import search results into a citation management tool (see below) to clean up the citations and remove any duplicates. You can then use a tool like Rayyan (see below) to screen the results. You should begin by screening titles and abstracts only, and then you'll examine the full text of any remaining articles. Each study should be reviewed by a minimum of two people on the project team. 
• Collect - Each included study is coded and the quantitative or qualitative data contained in these studies is then synthesized. You'll have to either find or develop a coding strategy or form that meets your needs. 
• Explain - The synthesized results are articulated and contextualized. What do the results mean? How have they answered your research question?
• Summarize - The final report provides a complete description of the methods and results in a clear, transparent fashion. 

Adapted from

Types of reviews, systematic review.

These types of studies employ a systematic method to analyze and synthesize the results of numerous studies. "Systematic" in this case means following a strict set of steps - as outlined by entities like PRISMA and the Institute of Medicine - so as to make the review more reproducible and less biased. Consistent, thorough documentation is also key. Reviews of this type are not meant to be conducted by an individual but rather a (small) team of researchers. Systematic reviews are widely used in the health sciences, often to find a generalized conclusion from multiple evidence-based studies. 

Meta-Analysis

A systematic method that uses statistics to analyze the data from numerous studies. The researchers combine the data from studies with similar data types and analyze them as a single, expanded dataset. Meta-analyses are a type of systematic review.

Scoping Review

A scoping review employs the systematic review methodology to explore a broader topic or question rather than a specific and answerable one, as is generally the case with a systematic review. Authors of these types of reviews seek to collect and categorize the existing literature so as to identify any gaps.

Rapid Review

Rapid reviews are systematic reviews conducted under a time constraint. Researchers make use of workarounds to complete the review quickly (e.g., only looking at English-language publications), which can lead to a less thorough and more biased review. 

Narrative Review

A traditional literature review that summarizes and synthesizes the findings of numerous original research articles. The purpose and scope of narrative literature reviews vary widely and do not follow a set protocol. Most literature reviews are narrative reviews. 

Umbrella Review

Umbrella reviews are, essentially, systematic reviews of systematic reviews. These compile evidence from multiple review studies into one usable document. 

Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal , vol. 26, no. 2, 2009, pp. 91-108. doi: 10.1111/j.1471-1842.2009.00848.x .

• Next: Project Planning >>

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Advanced Search
• Journal List
• Indian J Dermatol
• v.59(2); Mar-Apr 2014

Understanding and Evaluating Systematic Reviews and Meta-analyses

Michael bigby.

From the Department of Dermatology, Harvard Medical School, Beth Israel Deaconess Medical Center, Boston, MA 02215, USA

A systematic review is a summary of existing evidence that answers a specific clinical question, contains a thorough, unbiased search of the relevant literature, explicit criteria for assessing studies and structured presentation of the results. A systematic review that incorporates quantitative pooling of similar studies to produce an overall summary of treatment effects is a meta-analysis. A systematic review should have clear, focused clinical objectives containing four elements expressed through the acronym PICO (Patient, group of patients, or problem, an Intervention, a Comparison intervention and specific Outcomes). Explicit and thorough search of the literature is a pre-requisite of any good systematic review. Reviews should have pre-defined explicit criteria for what studies would be included and the analysis should include only those studies that fit the inclusion criteria. The quality (risk of bias) of the primary studies should be critically appraised. Particularly the role of publication and language bias should be acknowledged and addressed by the review, whenever possible. Structured reporting of the results with quantitative pooling of the data must be attempted, whenever appropriate. The review should include interpretation of the data, including implications for clinical practice and further research. Overall, the current quality of reporting of systematic reviews remains highly variable.

Introduction

A systematic review is a summary of existing evidence that answers a specific clinical question, contains a thorough, unbiased search of the relevant literature, explicit criteria for assessing studies and structured presentation of the results. A systematic review can be distinguished from a narrative review because it will have explicitly stated objectives (the focused clinical question), materials (the relevant medical literature) and methods (the way in which studies are assessed and summarized).[ 1 , 2 ] A systematic review that incorporates quantitative pooling of similar studies to produce an overall summary of treatment effects is a meta-analysis.[ 1 , 2 ] Meta-analysis may allow recognition of important treatment effects by combining the results of small trials that individually might lack the power to consistently demonstrate differences among treatments.[ 1 ]

With over 200 speciality dermatology journals being published, the amount of data published just in the dermatologic literature exceeds our ability to read it.[ 3 ] Therefore, keeping up with the literature by reading journals is an impossible task. Systematic reviews provide a solution to handle information overload for practicing physicians.

Criteria for reporting systematic reviews have been developed by a consensus panel first published as Quality of Reporting of Meta-analyses (QUOROM) and later refined as Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).[ 4 , 5 ] This detailed, 27-item checklist contains items that should be included and reported in high quality systematic reviews and meta-analyses. The methods for understanding and appraising systematic reviews and meta-analyses presented in this paper are a subset of the PRISMA criteria.

The items that are the essential features of a systematic review include having clear objectives, explicit criteria for study selection, an assessment of the quality of included studies, criteria for which studies can be combined, appropriate analysis and presentation of results and practical conclusions that are based on the evidence evaluated [ Table 1 ]. Meta-analysis is only appropriate if the included studies are conceptually similar. Meta-analyses should only be conducted after a systematic review.[ 1 , 6 ]

Criteria for evaluating a systematic review or the meta-analysis

A Systematic Review Should Have Clear, Focused Clinical Objectives

A focused clinical question for a systematic review should contain the same four elements used to formulate well-built clinical questions for individual studies, namely a Patient, group of patients, or problem, an Intervention, a Comparison intervention and specific Outcomes.[ 7 ] These features can be remembered by the acronym PICO. The interventions and comparison interventions should be adequately described so that what was done can be reproduced in future studies and in practice. For diseases with established effective treatments, comparisons of new treatments or regimens to established treatments provide the most useful information. The outcomes reported should be those that are most relevant to physicians and patients.[ 1 ]

Explicit and Thorough Search of the Literature

A key question to ask of a systematic review is: “Is it unlikely that important, relevant studies were missed?” A sound systematic review can be performed only if most or all of the available data are examined. An explicit and thorough search of the literature should be performed. It should include searching several electronic bibliographic databases including the Cochrane Controlled Trials Registry, which is part of the Cochrane Library, Medline, Embase and Literatura Latino Americana em Ciências da Saúde. Bibliographies of retrieved studies, review articles and textbooks should be examined for studies fitting inclusion criteria. There should be no language restrictions. Additional sources of data include scrutiny of citation lists in retrieved articles, hand-searching for conference reports, prospective trial registers (e.g., clinical trials.gov for the USA and clinical trialsregister.eu for the European union) and contacting key researchers, authors and drug companies.[ 1 , 8 ]

Reviews should have Pre-defined Explicit Criteria for what Studies would be Included and the Analysis should Include Only those Studies that Fit the Inclusion Criteria

The overwhelming majority of systematic reviews involve therapy. Randomized, controlled clinical trials should therefore be used for systematic reviews of therapy if they are available, because they are generally less susceptible to selection and information bias in comparison with other study designs.[ 1 , 9 ]

Systematic reviews of diagnostic studies and harmful effects of interventions are increasingly being performed and published. Ideally, diagnostic studies included in systematic reviews should be cohort studies of representative populations. The studies should include a criterion (gold) standard test used to establish a diagnosis that is applied uniformly and blinded to the results of the test(s) being studied.[ 1 , 9 ]

Randomized controlled trials can be included in systematic reviews of studies of adverse effects of interventions if the events are common. For rare adverse effects, case-control studies, post-marketing surveillance studies and case reports are more appropriate.[ 1 , 9 ]

The Quality (Risk of Bias) of the Primary Studies should be Critically Appraised

The risk of bias of included therapeutic trials is assessed using the criteria that are used to evaluate individual randomized controlled clinical trials. The quality criteria commonly used include concealed, random allocation; groups similar in terms of known prognostic factors; equal treatment of groups; blinding of patients, researchers and analyzers of the data to treatment allocation and accounting for all patients entered into the trial when analyzing the results (intention-to-treat design).[ 1 ] Absence of these items has been demonstrated to increase the risk of bias of systematic reviews and to exaggerate the treatment effects in individual studies.[ 10 ]

Structured Reporting of the Results with Quantitative Pooling of the Data, if Appropriate

Systematic reviews that contain studies that have results that are similar in magnitude and direction provide results that are most likely to be true and useful. It may be impossible to draw firm conclusions from systematic reviews in which studies have results of widely different magnitude and direction.[ 1 , 9 ]

Meta-analysis should only be performed to synthesize results from different trials if the trials have conceptual homogeneity.[ 1 , 6 , 9 ] The trials must involve similar patient populations, have used similar treatments and have measured results in a similar fashion at a similar point in time.

Once conceptual homogeneity is established and the decision to combine results is made, there are two main statistical methods by which results are combined: random-effects models (e.g., DerSimonian and Laird) and fixed-effects models (e.g., Peto or Mantel-Haenszel).[ 11 ] Random-effects models assume that the results of the different studies may come from different populations with varying responses to treatment. Fixed-effects models assume that each trial represents a random sample of a single population with a single response to treatment [ Figure 1 ]. In general, random-effects models are more conservative (i.e., random-effects models are less likely to show statistically significant results than fixed-effects models). When the combined studies have statistical homogeneity (i.e., when the studies are reasonably similar in direction, magnitude and variability), random-effects and fixed-effects models give similar results.

Fixed-effects models (a) assume that each trial represents a random sample (colored curves) of a single population with a single response to treatment. Random-effects models (b) assume that the different trials’ results (colored curves) may come from different populations with varying responses to treatment.

The point estimates and confidence intervals of the individual trials and the synthesis of all trials in meta-analysis are typically displayed graphically in a forest plot [ Figure 2 ].[ 12 ] Results are most commonly expressed as the odds ratio (OR) of the treatment effect (i.e., the odds of achieving a good outcome in the treated group divided by the odds of achieving a good result in the control group) but can be expressed as risk differences (i.e., difference in response rate) or relative risk (probability of achieving a good outcome in the treated group divided by the probability in the control group). An OR of 1 (null) indicates no difference between treatment and control and is usually represented by a vertical line passing through 1 on the x-axis. An OR of greater or less than 1 implies that the treatment is superior or inferior to the control respectively.

Annotated results of a meta-analysis of six studies, using random effects models reported as odd ratios using MIX version 1.7 (Bax L, Yu LM, Ikeda N, Tsuruta H, Moons KGM. Development and validation of MIX: comprehensive free software for meta-analysis of causal research data. BMC Med Res Methodol http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1626481/ ). The central graph is a typical Forest Plot

The point estimate of individual trials is indicated by a square whose size is proportional to the size of the trial (i.e., number of patients analyzed). The precision of the trial is represented by the 95% confidence interval that appears in Forest Plots as the brackets surrounding point estimate. If the 95% confidence interval (brackets) does not cross null (OR of 1), then the individual trial is statistically significant at the P = 0.05 level.[ 12 ] The summary value for all trials is shown graphically as a parallelogram whose size is proportional to the total number of patients analyzed from all trials. The lateral tips of the parallelogram represent the 95% confidence interval and if they do not cross null (OR of 1), then the summary value of the meta-analysis is statistically significant at the P = 0.05 level. ORs can be converted to risk differences and numbers needed to treat (NNTs) if the event rate in the control group is known [ Table 2 ].[ 13 , 14 ]

Deriving numbers needed to treat from a treatment's odds ratio and the observed or expected event rates of untreated groups or individuals

The difference in response rate and its reciprocal, the NNT, are the most easily understood measures of the magnitude of the treatment effect.[ 1 , 9 ] The NNT represents the number of patients one would need to treat in order to achieve one additional cure. Whereas the interpretation of NNT might be straightforward within one trial, interpretation of NNT requires some caution within a systematic review, as this statistic is highly sensitive to baseline event rates.[ 1 ]

For example, if a treatment A is 30% more effective than treatment B for clearing psoriasis and 50% of people on treatment B are cleared with therapy, then 65% will clear with treatment A. These results correspond to a rate difference of 15% (65-50) and an NNT of 7 (1/0.15). This difference sounds quite worthwhile clinically. However if the baseline clearance rate for treatment B in another trial or setting is only 30%, the rate difference will be only 9% and the NNT now becomes 11 and if the baseline clearance rate is 10%, then the NNT for treatment A will be 33, which is perhaps less worthwhile.[ 1 ]

Therefore, NNT summary measures within a systematic review should be interpreted with caution because “control” or baseline event rates usually differ considerably between studies.[ 1 , 15 ] Instead, a range of NNTs for a range of plausible control event rates that occur in different clinical settings should be given, along with their 95% confidence intervals.[ 1 , 16 ]

The data used in a meta-analysis can be tested for statistical heterogeneity. Methods to tests for statistical heterogeneity include the χ 2 and I.[ 2 , 11 , 17 ] Tests for statistical heterogeneity are typically of low power and hence detecting statistical homogeneity does not mean clinical homogeneity. When there is evidence of heterogeneity, reasons for heterogeneity between studies – such as different disease subgroups, intervention dosage, or study quality – should be sought.[ 11 , 17 ] Detecting the source of heterogeneity generally requires sub-group analysis, which is only possible when data from many or large trials are available.[ 1 , 9 ]

In some systematic reviews in which a large number of trials have been performed, it is possible to evaluate whether certain subgroups (e.g. children versus adults) are more likely to benefit than others. Subgroup analysis is rarely possible in dermatology, because few trials are available. Subgroup analyses should always be pre-specified in a systematic review protocol in order to avoid spurious post hoc claims.[ 1 , 9 ]

The Importance of Publication Bias

Publication bias is the tendency that studies that show positive effects are more likely to be published and are easier to find.[ 1 , 18 ] It results from allowing factors other than the quality of the study to influence its acceptability for publication. Factors such as the sample size, the direction and statistical significance of findings, or the investigators’ perception of whether the findings are “interesting,” are related to the likelihood of publication.[ 1 , 19 , 20 ] Negative studies with small sample size are less likely to be published.[ 1 , 19 , 20 ] Studies published are often dominated by the pharmaceutical company sponsored trials of new, expensive treatments often compared with the placebo.

For many diseases, the studies published are dominated by drug company-sponsored trials of new, expensive treatments. Such studies are almost always “positive.”[ 1 , 21 , 22 ] This bias in publication can result in data-driven systematic reviews that draw more attention to those medicines. Systematic reviews that have been sponsored directly or indirectly by industry are also prone to bias through over-inclusion of unpublished “positive” studies that are kept “on file” by that company and by not including or not finishing registered trials whose results are negative.[ 1 , 23 ] The creation of study registers (e.g. http://clinicaltrials.gov ) and advance publication of research designs have been proposed as ways to prevent publication bias.[ 1 , 24 , 25 ] Many dermatology journals now require all their published trials to have been registered beforehand, but this policy is not well policed.[ 1 ]

Language bias is the tendency for studies that are “positive” to be published in an English-language journal and be more quickly found than inconclusive or negative studies.[ 1 , 26 ] A thorough systematic review should therefore not restrict itself to journals published in English.[ 1 ]

Publication bias can be detected by using a simple graphic test (funnel plot), by calculating the fail-safe N, Begg's rank correlation method, Egger regression method and others.[ 1 , 9 , 11 , 27 , 28 ] These techniques are of limited value when less than 10 randomized controlled trials are included. Testing for publication bias is often not possible in systematic reviews of skin diseases, due to the limited number and sizes of trials.[ 1 , 9 ]

Question-driven systematic reviews answer the clinical questions of most concern to practitioners. In many cases, studies that are of most relevance to doctors and patients have not been done in the field of dermatology, due to inadequate sources of independent funding.[ 1 , 9 ]

The Quality of Reporting of Systematic Reviews

The quality of reporting of systematic reviews is highly variable.[ 1 ] One cross-sectional study of 300 systematic reviews published in Medline showed that over 90% were reported in specialty journals. Funding sources were not reported in 40% of reviews. Only two-thirds reported the range of years that the literature was searched for trials. Around a third of reviews failed to provide a quality assessment of the included studies and only half of the reviews included the term “systematic review” or “meta-analysis” in the title.[ 1 , 29 ]

The Review should Include Interpretation of the Data, Including Implications for Clinical Practice and Further Research

The conclusions in the discussion section of a systematic review should closely reflect the data that have been presented within that review. Clinical recommendations can be made when conclusive evidence is found, analyzed and presented. The authors should make it clear which of the treatment recommendations are based on the review data and which reflect their own judgments.[ 1 , 9 ]

Many reviews in dermatology, however, find little evidence to address the questions posed. The review may still be of value even if it lacks conclusive evidence, especially if the question addressed is an important one.[ 1 , 30 ] For example, the systematic review may provide the authors with the opportunity to call for primary research in an area and to make recommendations on study design and outcomes that might help future researchers.[ 1 , 31 ]

Source of Support: Nil

Conflict of Interest: Nil.

Penn State University Libraries

Systematic reviews in education and psychology: an introductory guide.

• Systematic Review Essentials (Home)
• Sample Workflow for Education and Psychology Systematic Reviews
• Databases for Education and Psychology Systematic Reviews
• Software Supporting Systematic Reviews
• Other Review Types

Ellysa Stern Cahoy, Education Librarian and Co-Director, Reference Services

Welcome to Systematic Reviews!

Please note: this guide reuses and builds upon content from a prior library guide on systematic reviews in health care by Penn State librarians Christina Wissinger and Kathleen Phillips. Here, we have adapted their material to assist Education faculty and students.

A systematic review is a comprehensive analysis of all known evidence on a given subject. In the words of Siddaway, Wood, and Hedges (2019) , systematic reviews are "methodical, comprehensive, transparent, and replicable." Sometimes conducted for publication in scholarly venues, they are much more rigorous than the literature searches that students usually do when writing course papers. In Education and Psychology, systematic reviews typically include:

• A clearly defined research question. The research question is often developed after performing preliminary searches on the subject, ensuring that it is viable and that no other systematic reviews exist on the topic.  
• Evidence of a rigorous search process. Systematic searching demands a carefully planned search strategy, described in the methodology section of the review.. This often includes the databases used; the keywords and thesaurus terms searched; and any limits placed on the search.   
• Inclusion and exclusion criteria. Not all evidence found through a search will be relevant to the research question. Clearly defined criteria must be used to decide which studies should be included in analysis.  
• Critical appraisal of all included studies.

To Learn More About Systematic Reviews in Education and Psychology

• The Art and Science of Quality Literature Reviews Offers important guidance on conducting systematic reviews.
• The Concept of a Systematic Review Infographics explaining what systematic reviews are and how they work.
• High Quality Meta-Analysis in a Systematic Review Explains meta-analysis, an important facet of many systematic reviews.
• How to Do a Systematic Review Dispels many common misunderstandings about systematic reviews.
• Systematic Reviews in Educational Research A book length overview of conducting systematic reviews of educational research.
• What Are Systematic Reviews? A video that explains why systematic reviews are important, how they are done, and how interventions are compared.
• SAGE Research Methods Core This link opens in a new window Definitions, book chapters, and other explanations about research methods, including systematic reviews. more... less... SAGE Research Methods is a research methods tool created to help researchers, faculty and students with their research projects. SAGE Research Methods links over 100,000 pages of SAGE's renowned book, journal and reference content with truly advanced search and discovery tools. Researchers can explore methods concepts to help them design research projects, understand particular methods or identify a new method, conduct their research, and write up their findings. Since SAGE Research Methods focuses on methodology rather than disciplines, it can be used across the social sciences, health sciences, and more. SAGE Research Methods contains content from more than 640 books, dictionaries, encyclopedias, and handbooks, the entire Little Green Book, and Little Blue Book series, two Major Works collating a selection of journal articles, and newly commissioned videos. Our access is to: SRM Core Update 2020-2025; SRM Cases (includes updates through 2025); SRM Cases 2.
• Credo Information Literacy Tutorials Refreshers on choosing a research topic, primary vs. secondary literature, evaluating sources, and other general skills.

Kogut, A., Foster, M., Ramirez, D., & Xiao, D. (2019). Critical Appraisal of Mathematics Education Systematic Review Search Methods: Implications for Social Sciences Librarians. College & Research Libraries , 80 (7), 973–995. https://doi.org/10.5860/crl.80.7.973

• Next: Sample Workflow for Education and Psychology Systematic Reviews >>
• Last Updated: Sep 8, 2023 12:07 PM
• URL: https://guides.libraries.psu.edu/edpsyreviews

For database and search translation advice, please see the following Library Research Guides: 

• Psychology and Counselling 
• Criminology & Forensic Psychology

This section contains handbooks, reporting standards, protocol development, search construction and critical appraisal tools to give you background information on all stages of the review. 

Handbooks 

Handbooks are comprehensive guides to conducting a systematic review from protocol development to reporting the results. 

Reporting standards

Reporting standards aim to improve the standards of reporting in evidence synthesis. Utilising reporting standards will ensure all the relevant methodological information is included for the peer-review process.

Protocol development & registration

A review protocol defines the scope of a systematic review including the research question/s, populations, settings and outcomes. The search strategy for retrieving papers is also outlined in the protocol. Register your systematic review protocol to ensure your topic is not duplicated unknowingly by another researcher.  

Search construction & analysis

Learn how to develop a database search that retrieves all the published literature on a research topic. 

Critical appraisal 

Critical appraisal tools are frameworks used to evaluate the quality of studies included in your review. Decide how you will critically appraise the studies before you begin your review, and use them to inform the inclusion and exclusion criteria.

Method books, chapters & papers

A selection of books, book chapters and research papers on systematic review methods. 

• << Previous: Law
• Next: Social work >>
• Built environment
• Criminology
• Education (TESOL)
• Health sciences and medicine
• Social work

Contact your librarian

Faculty Librarian Society & Design Transformation Colab +61 7 5595 3347 [email protected]

Copyright © Bond University

• Library Search
• Electronic Resources A-Z
• Opening hours
• Main Library
• Law Library
• Multimedia Learning Centre
• Ask a Librarian
• Library leadership

In the spirit of reconciliation, Bond University acknowledges the Kombumerri people, the traditional Owners and Custodians of the land on which the university now stands. We pay respect to Elders past, present and emerging. Read more

CRICOS Provider Code 00017B | TEQSA Provider ID PRV12072

• Last Updated: Aug 16, 2023 1:28 PM

• Open supplemental data
• Reference Manager
• Simple TEXT file

People also looked at

Systematic review article, charting the psychological literacy landscape: systematic review highlighting psychology education.

• 1 School of Psychology, University of New South Wales, Sydney, NSW, Australia
• 2 School of Psychological Sciences, University of Tasmania, Hobart, TAS, Australia

The aim of this systematic review is to provide, for the first time, a broad overview of the scope and nature of the current English language concept of ‘psychological literacy' (PL) as evidenced in the literature, primarily since 2010. Covidence systematic review methodology yielded 112 papers that were scored on 23 specific criteria/questions relevant to addressing predetermined research questions. PL has been conceptualized as both a general capability (e.g., ethical application of psychological knowledge) and a group of capabilities (discipline-specific knowledge and research methods, and more generic capabilities such as communication). Variability in the constellation of group capabilities has led to challenges in operationalization and thus measurement, signaling the need for international consensus and improved measurement. We propose a model for how PL is related to psychologically literate citizenship and global citizenship. Key papers explore PL as an integrative concept in psychology. All papers were relevant to psychology education, with most prevalent being undergraduate level, and least prevalent being graduate level. There were numerous papers providing practical PL teaching and assessment strategies. PL as a pedagogical approach has been a necessary and richly diverse focus. Finally, a revisioning of PL within the context of psychology education, as well as recommendations for further research and development, are suggested.

Introduction

The aim of this systematic review is to provide a broad overview of the scope and nature of the current English language concept of “psychological literacy” (PL) as evidenced in the literature, primarily since 2010. We explore questions about the internationality of the term, the conceptual development of the term and its relationship to the notion of citizenship, its penetration into different levels of education, the existence of relevant teaching and assessment strategies, its measurement, the impact on pedagogy, and the potential for psychological literacy to be an integrative concept in psychology.

For this paper, we take McGovern et al. (2010) as the starting point for the modern English language understanding of PL. Almost all of the papers reviewed here refer to this paper. We also include two relevant “precursor” papers 1 ( Boneau, 1990 ; O'Hara, 2007 ) to McGovern et al. (2010) , along with an educational report by Cranney (2008) which referred to the upcoming McGovern et al. (2010) chapter (first drafted in 2008). As summarized by Morris et al. (2021) , McGovern et al. conceptualized PL as “encapsulating nine graduate capabilities that UG psychology major students should acquire, including discipline knowledge and its application to personal, professional, and societal contexts; developing scientific, critical, and creative ways of thinking; and behaving in an ethical and diversity-respectful manner” (p.3). Morris et al. (2021) refer to this kind of conceptualization as a “group” definition because the concept is a list of different capabilities which together make up the concept of PL, and such definitions are tied to broad psychology education outcomes.

In contrast to this “group” category of PL definition is the “general” category definition of PL ( Morris et al., 2021 ). Perhaps the most commonly cited example is that of Cranney et al. (2012a) , who defined PL as “the general capacity to adaptively and intentionally apply psychology to meet personal, professional and societal needs” (p. iii). Note that this definition is not tied to formal education. The existence of these two different approaches (group vs . general) to defining PL has caused some consternation amongst psychological scientists and educators, especially when different group definitions include different capabilities; consequently, the construct validity of PL has been questioned ( Newell et al., 2020 , 2021 ). Given the methodology of the current review, we cannot empirically address this issue; instead, we report on the frequency and nature of both group and general definitions of PL, as well as on the prevalence of papers attempting to measure PL based on these definitions, and then discuss theoretical and practical implications.

Another conceptual discrepancy within the recent PL literature relates to the distinction between the term PL and McGovern et al. (2010) “concept of a psychologically literate citizen – someone who responds to the call for ethical commitment and social responsibility” (p.10). Although “acting ethically” and “fostering respect for diversity” are two of the nine capabilities that McGovern et al. list as encompassing PL, the distinction is that the psychologically literate citizen has a values-driven intention to apply their PL to “the common good”. In this review we identify the papers that use the terms “psychologically literate citizen” (PLC), “psychologically literate global citizen” (PLGC), and “global citizen” (GC), and then discuss these distinctions further.

In considering the landscape of PL, this review considers how international, and how collaboratively international, is work on PL, taking into account the limitation that only English language papers are included in this review. We then identify what have been the main types of publication forms (e.g., peer-reviewed journal articles, chapters, reports), and consider whether this pattern has changed over the years.

Given that McGovern et al. (2010) creation of the term was within the context of psychology education, it would not be surprising that much of the literature is situated within that context, but in this review we test that assumption. Then we review the distribution of papers across different levels of education (i.e., pre-tertiary, undergraduate, graduate, interprofessional, public education), and the implications of this distribution for theory and practice. Given the argument that PL should be the outcome of psychology education ( Halpern, 2010 ), we also determine the prevalence of papers that provide educators with practical teaching and assessment strategies, and the implications of those findings.

A key consideration in this review is the translational impact of PL, particularly in terms of the teaching of PL and the associated pedagogy. The original McGovern et al. (2010) paper raised these issues, which Morris et al. (2021) termed “PL as a pedagogical philosophy”, whereby one inherent aspect is a commitment to PL as an outcome of psychology education. Other aspects (also referred to by McGovern et al., 2010 ) include (a) the use of evidence-based (or evidence-informed) teaching, and (b) the role-modeling of PL in other ways. As such, it was important to identify the prevalence of papers on this topic and the associated implications for theory and practice.

Finally, McGovern et al. (2010) spoke of the potential for the term PL to unite psychology education, research and practice:

“The unifying concept of psychological literacy has great potential to capture the imagination of diverse stakeholders invested in transforming higher education. With common understandings and rigorously defined standards, there still remains a healthy diversity in undergraduate programs. There was a commonly espoused principle in all that we read, with a bright spotlight shining on the universal demands for citizens' wellbeing and health care, and a need for greater commitment to those populations who have still-limited participation in the fruits of economic growth. Psychologists need to be leaders in these areas, especially because we can contribute integrated scientific and practitioner strengths” (p.19).

Interestingly, this quote somewhat transgresses their distinction between PL and PLC. Nevertheless, it raises the question of whether other PL/PLC papers also speak of this possibility, and the implications of adopting such an approach.

The research questions addressed in this study are grouped under three headings.

PL General Landscape:

• How international, and how collaboratively international, is the PL literature?

• What is the nature of the publication type (e.g., peer-reviewed journal vs. other types) for the PL literature, and has this changed over the years?

PL Conceptual Landscape:

• Which papers are considered to have made a significant contribution to the development of the concept of psychological literacy?

• What is the prevalence of papers that conceptualize PL as a general capacity compared to a group of capacities?

• What is the prevalence of papers that refer to the terms psychologically literate citizen, psychologically literate global citizen, or global citizen?

• What is the prevalence of papers that address the measurement of PL?

• How prevalent are papers that consider PL as a unifying concept in psychology?

PL Education Landscape:

• What is the prevalence of PL papers that have relevance to education, and what is the distribution across different levels/domains of education?

• How prevalent are papers that provide educators with practical teaching and assessment strategies for PL?

• How prevalent are papers that address PL as a pedagogical philosophy, with its various aspects?

Where appropriate, we (a) give some guidance to readers by briefly describing example papers related to these research questions, (b) identify gaps in the literature, and (c) expand on theoretical and practical implications.

The Cochrane Handbook of Systematic Reviews of Interventions ( Higgins et al., 2011 ) was followed in order to achieve transparency and consistency in the reporting of results. Selected electronic databases (Web of Science, PsycINFO, Scopus, MEDLINE) were searched for the term “psychological literacy” (see Supplementary material for syntax used and number of papers retrieved). Each database was searched from inception to the date of the first search (18/3/2021). Additional papers from various sources were also identified (up to 31/7/2021) through citation searching of included papers, and through correspondence with authors of included papers (see Supplementary material for additional details of this process). A final search (using the same databases as the original) was undertaken on 20/03/22. All identified papers were entered into the Covidence platform for initial screening and full-text review. A number of exclusion criteria (see Figure 1 ) were defined prior to abstract screening and full text review. Retrieved papers were distributed and screened by co-authors JC, SM, KN and CC against inclusion and exclusion criteria (see Figure 1 for details of initial and full-text screening stages and paper outcomes). All conflicts in both screening stages were discussed and resolved among authors.

Figure 1 . Flow diagram depicting initial screening and full-text review outcomes. *Information was considered redundant if it was in another paper by the same author that was already included in the review.

The data extraction process involved 23 questions in the domains of identifying details of the paper and author/s, conceptualization of PL, relevant level of psychology education (e.g., pre-tertiary) and contribution to the body of knowledge relating to PL (see Supplementary material ). Two co-authors (JC, SM) independently reviewed and completed data extraction of all papers for all questions, and one co-author (KN) independently reviewed Q7 (“Within the context of when it was written, how much does this paper contribute to our conceptual understanding of PL?”). Despite the lack of explicit criteria for this rating, there was minimal disagreement amongst the raters, which was consensually resolved (see Supplementary material ). Throughout the data extraction process, conflicts were resolved via discussion between the co-authors until a justified consensus was reached (see the Supplementary material for an example of conflict resolution processes). Chi-square tests of independence were conducted on included papers to examine the quantity and type of papers published over time.

Results and discussion

Pl general landscape.

Table 1 presents the assigned number for each of the reviewed papers, arranged by year of publication.

Table 1 . Review numbers for each paper by year (Q.2).

How international, and how collaboratively international, is work on PL?

Table 2 presents the national affiliation of the first author, including an indication of co-author multi-nationality. The first five nations mentioned are native English language speaking, which is not surprising, given that one of the inclusion criteria was that the paper had to be written in English. In the most recent period of 2020 to 2022, the constitution is somewhat similar to the total, with: 6 Australia, 4 USA, 4 UK, and one each from Canada, Germany, and Slovakia. There were 13 (11.6%) multinational co-authored papers, whereby the first co-author was affiliated with Australia (5), USA (4), UK (3), and Slovakia (1). Again, there is bias toward native English language countries, but approximately 12% multinational co-authorship is a reasonable proportion given that the concept of PL is relatively new.

Table 2 . Paper number as a function of national identification of first author.

What is the nature of the publication type (e.g., peer-reviewed journal vs. other types) for the PL literature, and has this changed over the years?

Table 3 presents type of publication and indicates that overall there were 47 chapters, 41 journal articles, 10 peer-reviewed reports, 6 Others, 4 books, 3 conference proceedings, and one edited book. Figure 2 presents the total number of papers across four time periods: 2010-2012; 2013-2015; 2016-2018; and 2019-2021. The greatest number of papers appears to be in the first time period, however a Chi Square analysis revealed no significant differences in numbers of papers across time periods. Because peer-reviewed journal articles are often considered the highest value publication type, these data are also summarized in Figure 2 . The number of this type of paper appears to have increased since the first time-period, however a Chi Square analysis revealed no significant differences.

Table 3 . Paper number as a function of paper/publication type (Q.6).

Figure 2 . Total number of papers, and number of journal articles, as a function of time period.

In summary, publication output on PL has been sustained; however growth in quantity, quality (e.g., journal articles), and representation across diverse landscapes (e.g., language) is required if this concept is to have global impact into the future.

PL conceptual landscape

Which papers are considered to have made a significant contribution to the development of the concept of psychological literacy.

The three raters scored on a 5-point scale (with higher scores indicating greater contributions) their answer to Q7 (“Within the context of when it was written, how much does this paper contribute to our conceptual understanding of PL?”). There were six papers receiving a rating of 5, and four papers receiving a rating of 4 (see Supplementary material for complete listings). Some of these papers will be discussed in the interim summary below.

What is the prevalence of papers that conceptualize PL as a general capacity compared to a group of capacities?

Table 4 indicates that approximately half the papers mentioned PL as a general capability, whereas there appear to be more papers that mention PL as a group of capabilities. Note that this is not an either-or situation; 45 papers (40.9 %) mentioned both conceptualizations. For PL as a general capability, many cite Cranney et al. (2012a) ; however there is some variability, occasionally with specific reference to education. For example, Amsel et al. (2014) described PL as “the integrated set of disciplinary attitudes, knowledge, values, beliefs, and skills which can be acquired through training and adaptively used to solve real world life and community problems” (p.1). The variety of ways in which PL has been defined as a group of capabilities is illustrated by Newell et al. (2021) .

Table 4 . Paper number as a function of type of psychological literacy (PL) definition mentioned, whether “citizen” was mentioned, whether relevant to measurement, and whether mentioned PL as a unifying concept.

What is the prevalence of papers that refer to the terms psychologically literate citizen, psychologically literate global citizen, or global citizen?

Table 4 indicates that over half of the papers mentioned at least one of these “citizen” terms. Thus, it can be concluded that PL is relatively strongly associated with this term.

Interim discussion

In terms of the ratings (1 [minimal] to 5 [significant]) of contribution to the conceptual understanding of PL (Q.7), amongst the 5-rated papers, McGovern et al. (2010) was identified in the Introduction as a key conceptual paper to which the majority of subsequent PL papers refer. The paper refers to PL as the group of nine graduate capabilities that all undergraduate psychology major students should acquire: “having a well-defined vocabulary and basic knowledge of the critical subject matter of psychology; valuing the intellectual challenges required to use scientific thinking and the disciplined analysis of information to evaluate alternative courses of action; taking a creative and amiable skeptic approach to problem solving; applying psychological principles to personal, social, and organizational issues in work, relationships, and the broader community; acting ethically; being competent in using and evaluating information and technology; communicating effectively in different modes and with many different audiences; recognizing, understanding, and fostering respect for diversity; and being insightful and reflective about one's own and others' behavior and mental processes” (p. 11).

As noted in the Introduction, McGovern et al. (2010) also makes the first reference to the term PLC, which is framed as building upon but distinct from PL. Cranney and Dunn's (2011a) edited book included the remit to authors to refer to the McGovern et al. (2010) chapter. In the first chapter, Cranney and Dunn (2011b) refer to McGovern et al.'s group of graduate capabilities, but also define PL in a general way as “psychological knowledge that is used adaptively” (p.8). These authors also argue that “psychologically literate citizens use their psychological literacy to solve problems in an ethical and socially responsible manner in a way that directly benefits their communities” (p.10). Murdoch (2016) defines PL in a general way as the “ethical application of psychology knowledge and skills” (p.189), but he also defines PL as a ‘meta-literacy' whereby there are a group of psychology-specific graduate capabilities (“Specific psychological knowledge in the core areas of psychology; Psychology-specific knowledge and skill in the generic literacies (e.g., appropriate search terms in information literacy, and double-blind methodologies); The ability to apply this knowledge and skill to personal, occupational and societal issues using the connected literacies and skills”; p.191), as well as a number of generic capabilities (e.g., multicultural literacy, scientific literacy, critical thinking). He views McGovern et al.'s concept of PLC favorably, but in need of further development, particularly in terms of including non-Western perspectives.

In summary, the “ group” conceptualizations are closely tied to consideration of what psychology major students should acquire during their program, including the generic capabilities (although as argued by Murdoch, 2016 ; there are psychology-specific aspects of these). Cranney et al. (2022a , b ) have recently proposed, for international discussion, a number of “core” PL capabilities (knowledge, including skills; research methods; application to personal, professional, and community domains) and generic (but informed by psychological science) capabilities such as communication and cultural responsivity.

In contrast, it could be argued that the “ general” definition of PL is not necessarily tied to a particular level of formal education, and indeed, could be acquired informally, as long as there is understanding of theory and research that underlies that particular aspect of PL [see Cranney and Morris (2021) , for a framework for understanding this notion 2 ]. It could be argued (as did one of our peer-reviewers) that the term “psychology” directly names the formal discipline and profession of psychology, whereas “psychological literacy” should be acquired by all of our graduates, but also may be acquired through informal education, to the benefit of the general public, as elaborated in the last paragraph of Section What is the prevalence of PL papers that have relevance to education, and what is the distribution across different levels/domains of education?. The reviewer offers “digital literacy” as an analogy: through their formal education and training, computer and digital scientists are digitally literate, however we also expect that any person using a digital device has some level of digital literacy. Others may argue that some theoretical knowledge underlying skill is essential to the semantically correct use of the term “literacy” ( Cranney and Dunn, 2011b ; Cranney and Morris, 2021 ).

An additional point made by both Cranney and Dunn (2011b) and Cranney and Morris (2021) is that no one person would be highly competent in applying knowledge/skills/attitudes in all areas of psychology; nevertheless “psychological literacy implies a relatively well-integrated and functional set of schemas that across individuals may show some variability in expression, but in terms of central tendency, can be recognized and assessed as ‘psychological literacy”' ( Cranney and Dunn, 2011b , p.8). These schemas could reflect integrating themes such as those recently identified in the APA Introductory Psychology (IP) Initiative, including “Psychological science relies on empirical evidence and adapts as new data develop”; “Our perceptions and biases filter our experiences of the world through an imperfect personal lens”; and “psychology values diversity, promotes equity, and fosters inclusion in pursuit of a more just society” ( APA, 2021 ; Gurung and Neufeld, 2022 ).

An extension of the conceptualization of PL is demonstrated by one of the 5-rated papers ( Hulme and Winstone, 2017 ), and many of the 4-rated papers. Essentially, an educator who accepts that psychological literacy should be the outcome of psychology education (at whatever level) should be psychologically literate themselves. As we discuss in the Education Landscape section, this includes using evidence-informed teaching strategies, as well as other aspects of modeling PL for students. Like the papers on cultural responsiveness (e.g., Coulson and Homewood, 2016 ; Dudgeon et al., 2018 ), this aspect of PL is quite complex but, we would argue, worth aspiring to.

Some argue that PLC is also an extension of the concept of PL; but we treat it as worthy of consideration in its own right (for excellent discussions leading to the same conclusion, see Harre et al., 2011 ; Trapp and Akhurst, 2011 ; Mair et al., 2013 ; moreover; Job et al., 2011 argue that PLC is a process, not an outcome). A very recent paper gives a general definition of PL as the “intentional values-driven application of psychology Knowledge to achieve personal, professional, and community goals” ( Cranney et al., 2022a ; p.3). That is, PL is seemingly values-neutral: one could use one's psychological knowledge and skills (a) to achieve highly individualistically oriented goals, or (b) for the “common good” as reflected in community welfare goals ( Sokol and Kuebli, 2011 ). However, while one could argue that the definition of PL is “values neutral”, the definition of PLC is weighted more toward “community welfare”. McGovern et al. (2010) did emphasize the “global” aspect of PLC, and Charlton and Lymburner (2011) made this more explicit with their term “psychologically literate global citizen”, and argued that “although the development of a global… citizen is obviously a multidisciplinary pursuit, psychology as a discipline is particularly well suited to this task” (p.234). Although (a) Cranney et al. (2022a , b ) make a broad distinction between local, national/regional, and global communities, and (b) one's community-related goals could focus more on a single community domain, in this paper for the sake of simplicity we treat the terms PLC and PLGC as equivalent. The transdisciplinary concept of global citizenship (GC) has been defined as involving “the understanding of global interrelatedness, and the capacity to live, work and contribute positively as a member of global communities” ( Cranney et al., 2012a ; p.iii). Clearly, GC can be acquired by various transdisciplinary routes. A possible way to think about this trans-disciplinarity and the relationships amongst these various concepts is proposed in Figure 3 .

Figure 3 . A proposal as to how psychological literacy (PL) relates (a) to the concept of psychologically literate citizenship (PLC; for simplicity's sake, this concept is considered here to be equivalent to the concept of psychologically literate global citizenship), and (b) to the concept of global citizenship (GC). The trajectory from PL through PLC to GC necessarily involves increased interaction (indicated by more open boundaries) with at least one ‘other-discipline literacy' (ODL) and that discipline's parallel potential development into the concepts of “other-discipline literate citizenship” (ODLC) and GC. Two ODL's are depicted here; but of course there are multiple ODLs that PL would interact with. GC is characterized by a deep appreciation of and respect for other disciplines/professions, reflected in very high levels of CR and V&E. Adv, Advanced; CR, Cultural responsivity; V&E, Values & Ethics.

What is the prevalence of papers that address the measurement of PL?

Table 4 presents the data relevant to PL measurement, which is closely tied to consideration of definitions of the concept of PL. As Roberts and Gasson (2018) point out, PL can be measured in the educational context through formal assessments; however, the capacity to measure PL in any situation and with any population could also be useful in terms of measuring the impact of “giving psychology away” ( Miller, 1969 ) in formal and informal (e.g., public) settings. Nevertheless, graduate employers apparently are less interested in the latter form of measurement ( McGovern et al., 2010 ). The measurement of PL has usually taken a group definition approach, with different measurement instruments for different capabilities, usually attempting to measure at least some of McGovern et al. (2010) capabilities. Attempts by Roberts and colleagues ( Roberts et al., 2015 ; Heritage et al., 2016 ) and by Burton et al. (2013) using primarily student self-report measures produced more than one factor, whereas Amsel et al. (2014) , using student performance measures, produced a single factor. Subsequent reviews by Newell et al. (2020 , 2021) have questioned the validity of the student self-report measures; indeed, measures that take curricular input variables (i.e., teaching strategies) into account may be needed in order to increase measurement validity. Very recently, and not analyzed in this review, Machin and Gasson (2022) describe the ongoing development of the scenario-based Test of Psychological Literacy – Revised (ToPL-R), whereby the:

“…scoring process [is] based on the Situation Judgement Test (SJT) methodology… Similar to a multiple choice question, respondents are presented with five possible answers to each scenario, each of which has been rated by experts as to how ‘correct' it is. By using a multiple choice style response format, the test can be completed in less time and scored electronically, making it fit for purpose (i.e., able to quickly and accurately assess psychological literacy)” (p. 21–23).

As highlighted by a peer-reviewer, a further gap in this measurement research is longitudinal studies across educational programs and further professional training, which could yield rich data for further conceptual development. In summary, more innovative research is required in this area, and both longitudinal studies and ToPL-R provide promising future directions.

How prevalent are papers that consider PL as a unifying concept in psychology?

As indicated in Table 4 , five papers other than McGovern et al. (2010) make reference to this idea. Cranney et al. (2012a) stated that “our aim was to have stakeholders understand each others' diverse needs at the same time as encouraging them to collaboratively progress a future-oriented agenda for our students, our discipline and profession, and our world (primarily through drawing on the concept of psychological literacy)” (p.23). Three papers ( Hulme and Winstone, 2017 ; Winstone and Hulme, 2017 ; Winstone and Kinchin, 2017 ) strongly argue that psychological literacy is an integrative concept in psychology, primarily because (a) adopting psychological literacy as a pedagogical philosophy (teaching approach) necessarily requires educators to role-model being psychologically literate in their educational practice, a process which promotes both the science and practice of psychology, and in doing so, (b) educators are more likely to produce graduates who are psychologically literate, and capable of promoting psychological science and practice in their diverse communities, thus both unifying and benefiting the discipline and profession of psychology as a whole. As Hulme and Winstone (2017) , p.264; argued, “… the discipline of psychology… has a clear integrative disciplinary concept, that of psychological literacy… which unites the discipline (i.e., what is to be taught) with pedagogy (i.e., how it should be taught)”. Landrum and McCarthy (2018) further highlight the potential for PL as a unifying concept in psychology:

“Our undergraduate pipeline is the key to our discipline's future, for the advancement of science, the practice of the profession, and in the public interest. Psychology educators should strive to create intentional learners… Intentional learners adapt to new environments, use methods of inquiry to integrate data from multiple sources, and foster lifelong learning. Intentional learners become empowered through the acquisition of knowledge and demonstration of skills, and they become informed about the world by using multiple forms of inquiry, and become responsible for their personal behavior and espouse civic values. Whether the label be intentional learners or the psychologically literate, attainment of these goals is too important to leave to chance.” (p.60).

The authors of all these papers were affiliated with native English language nations, and as noted elsewhere in this review, this idea of PL as a unifying concept in psychology may be challenged when scholars from other cultural traditions contribute robustly to this debate. Indeed, the relevance of this idea has already been challenged in the peer-review process for this paper, for example: How relevant is the concept of PL to graduate psychological scientists focusing on very narrow topics of research, or to graduate regulated psychological practitioners in specific areas of specialization? Our answer to these constructive criticisms lie in the common foundational PL capabilities (see also the General Discussion section), that, for example, (a) lead psychological scientists to share their findings with the general public through the use of evidence-based communication strategies (building, e.g., on their foundational communication, values & ethics, and application to the community domain capabilities), and (b) lead members of different psychological practice specializations to respect, rather than denigrate each other's certified capabilities (building, e.g., on their foundational cultural responsivity and values & ethics capabilities). Such psychologically literate behaviors (resulting from foundational education) should contribute to the public perception of psychology as a unified and credible discipline and profession, with multiple benefits to many stakeholders.

PL education landscape

What is the prevalence of pl papers that have relevance to education, and what is the distribution across different levels/domains of education.

All papers were judged to have relevance to education (Q.11: “Is this paper particularly relevant to psychology education (ie at least 50% of content)?”). This is not surprising, given the focus of Boneau (1990) and McGovern et al. (2010) on undergraduate psychology education. Papers contributing to the case for PL as an outcome for psychology education are summarized in Morris et al. (2021) . Although not scored in this study, it became apparent that all papers were aimed at an educator audience, except for three: Bernstein et al. (2020) is a first-year psychology text book; Dunn and Halonen (2020) provides career advice to psychology major students; and Morris et al. (2018) book describes the psychological science of many self-management tools, and is accessible to the public but also used as a text book for undergraduate students. The distribution of papers across the different levels of education is apparent in Table 5 , with the majority of papers being relevant to the undergraduate domain. Only 10 papers (9.1%) covered more than one education level.

Table 5 . Paper numbers as a function of level of education, and as a function of teaching and assessment strategies.

PL is particularly relevant to pre-tertiary psychology education. In their review of pre-tertiary education in Europe, Sokolova and Williamson (2020) state that one of the aims of the European Federation Psychology Teachers Association is to “increase psychological literacy across the nation” (p.343). National Psychology Curriculum Roundtable (NPCR) (2015) proposes an Australian curriculum which features PL. Hulme (2014) argues that for both pre-tertiary and undergraduate psychology education, emphasizing PL development enables adaptive application to the real world of diverse and changing career destinations; multiple curricular examples are provided. The key role of pre-tertiary education in the conceptualization of PL and in the appreciation of psychology by the general public is considered further in the General Discussion section.

At the undergraduate level, the scientific literacy aspect of PL is illustrated by Pownall et al. (2021) description and evaluation of a blog-writing assessment which “fosters psychological literacy by encouraging students to engage with the interplay between subject knowledge and real-world applications” (p.1). Illustrating the professionalism domain of PL is Hamilton et al. (2018) who, as summarized by Morris et al. (2021) , “provide a compelling, research-based rationale for offering psychology major students a work-integrated learning (WIL) or service learning experience, and then review the current literature on these strategies” (p.13; see also Reddy et al., 2013 ). Bringle et al. (2016 , 2022) provide a comprehensive theoretical and practical coverage across the undergraduate curriculum of service learning strategies for PLC development.

At the graduate level, Green et al. (2011) argue that graduate programs in positive psychology contribute toward greater PL and GC. Nissley and Atwood (2015) argue for PL as a teaching approach, and they propose specifically that in the early years of graduate training there should be a greater emphasis on application. One could argue that if all psychology major students acquired a moderate level of PL, then graduate training would involve acquiring a more advanced level of PL in a particular specialization (e.g., clinical, community, forensic, sports). The current lack of publications at the graduate level may reflect graduate educators' lack of awareness of, or lack of explicit focus on, this concept.

The moderately large number of papers addressing PL at an interprofessional level illustrates the relevance of psychology to education/training in multiple disciplines/professions - unsurprising given psychology's “hub” disciplinary status ( Boyack et al., 2005 ). This is an area full of intriguing challenges and opportunities, as Morris et al. (2021) illustrate in their summary of papers in this area. For example, Pusateri et al. (2019) argue against Department of Psychology educators delivering tailored psychology content into training programs for other professions, given (a) the too-high resource implications, and (b) mainstream psychology units oriented toward developing PL would necessarily deliver useful graduate capabilities relevant to all professions. Perhaps in contrast to Pusateri et al. (2019) argument, (a) Horn et al. (2021) describe how a German teacher training program productively adopts a PL teaching and outcome approach (similarly, see Zacharova et al., 2015 ), (b) Martin et al. (2020) describe how knowledge from psychological science benefits the undergraduate training of researchers from all science disciplines, and (c) Cranney and Morris (2021) describe integrating psychological science informed self-management capability building in units from any discipline. Regardless of how PL is developed during the training of other professionals, this plays a key role in Miller (1969) argument that psychological scientists and educators should “give psychology away” (p. 1071).

There are relatively few public education papers but nevertheless more so than at the graduate level. Most of these education-focused authors have a “big picture” awareness of the potential public good that should result from a more psychologically literate public. For example, Sawano (2011) describes media-based public education in Indonesia, and Morris et al. (2018) describe a set of evidence-informed tools for self-management in a way that is accessible to the general public. Both Cranney et al. (2012b) and Banyard and Hulme (2015) argue for the central role of pre-tertiary and undergraduate psychology in developing the PL of the general public, both because the large number of such graduates constitute a sizable proportion of the general population, and because of the informal educational “ripple effect” such graduates may have in their communities. Luong and Butler (2022) test a brief educational intervention designed to reduce fundamental attribution error in a general public sample. Clearly more direct public education approaches (such as described by Sawano, 2011 ) are required. Peak psychology discipline/professional bodies have the resources available to play a direct central role in increasing PL in the public domain through public education, with multiple potential benefits including increased support for the discipline and profession of psychology, and increased psychological health in the general population.

How prevalent are papers that provide educators with practical teaching and assessment strategies for PL?

Table 5 presents the data relevant to teaching and assessment strategies. Although there are more papers that cover teaching than assessment strategies, overall there are sufficient papers to allow educators to adapt the described strategies to their own programs and units. Excellent starting points are Mair et al. (2013) , the two Taylor and Hulme (2015a , 2018a) compendia and Hulme and Cranney (2022) , each of which contain multiple examples. One of the pioneering programs that utilized active learning, scaffolding, collaborative learning and authentic assessment in developing PL was at the University of Stirling, as described by Watt (2013) – a source of inspiration for all psychology educators. Nevertheless, as Cranney et al. (2022a , b ) point out, there may be particular PL capabilities, such as communication, that require more effective teaching strategies and more objective assessments – this may require more innovative student partnership and technological solutions.

How prevalent are papers that address PL as a pedagogical philosophy, with its various aspects?

Data relevant to PL as a pedagogical philosophy are presented in Table 6 . Note that responses to Q21 and 22 are contingent on a “Yes” response to Q.20. Seventeen papers were rated “Yes” for both Q.21 and 22, and we give some brief examples mostly from this subgroup; however, see Morris et al. (2021) for an overview of some relevant papers. McGovern et al. (2010) , p. 14; provide a list of faculty characteristics that are part of program quality benchmarks for PL (see also APA, 2011 ; McGovern, 2011 ). Bernstein (2011) notion of a scientist-educator includes two aspects of PL as a pedagogical philosophy: a commitment to PL as the desired outcome of psychology education, and the adoption of evidence-based teaching strategies (see also Dunn and McMinn, 2015 ; Cranney and Morris, 2021 ). Both Mair et al. (2013) and Hulme and Cranney (2022) provide a strong rationale for adopting PL as a pedagogical philosophy, and also provide multiple examples including program-level approaches.

Table 6 . Paper numbers as a function of aspects of psychological literacy (PL) as a pedagogical philosophy (PP).

Hulme and Winstone (2017) build on the notion of PL as an integrative disciplinary concept by explaining how professional values (evidence-based practice; acting ethically; professional competence) lead educators to take a risk-management rather than risk-averse approach to pedagogical innovation. The authors model PL by using McGovern et al. (2010) PL capabilities to illustrate how educators can safely and ethically approach (in a values-driven way) innovation to support the development of PL in their students.

“Within psychology…we believe that a psychologically literate teacher is one who is well equipped to deliver innovative teaching that is creative and moves the discipline forwards, and can practice within the bounds of their competence within a given educational context. This may, of course, require professional development and scholarship on the part of the educator, to stretch the bounds of their competence, and this too, draws upon the psychological literacy skills of the teacher. Thus psychological literacy might effectively act as a safety net to reassure the teacher during times of uncertainty… Perhaps the best strategy… to foster innovation, is to prioritize the development of psychological literacy and its delivery within our academic community” ( Hulme and Winstone, 2017 , p. 272)

This in-depth critically reflexive approach to the development of one's professional practice in guiding students toward the acquisition of PL parallels the work of McGovern (2011) and of those authors espousing the centrality of cultural responsivity (e.g., Coulson and Homewood, 2016 ; Dudgeon et al., 2018 ). In the next section, we present a renewed perspective on how educators can support the development of PL.

General discussion

In this section we (a) briefly summarize the answers to the questions posed in this review; (b) discuss some key implications, (c) discuss the limitations of the study, (d) present a perspective on how psychologically literate graduates emerge from psychology education, (e) consider the vital role and outcomes of introductory psychology, and (f) provide recommendations for future work regarding PL and psychology education.

General PL Landscape:

• There is moderate internationality and some collaborative internationality in the PL literature, but there is room for improvement, despite taking into account that only English language papers were included in this review.

• Publication type is heterogeneous, with an insignificant increase in journal articles across the years; total number of publications have not changed significantly over the four time periods examined.

• A number of papers that have made key conceptual contributions were identified and briefly described.

• A substantial number of papers (52.7%) made reference to PL as a general capacity, and even more (69.6%) made reference to PL as a group of capacities.

• A substantial number of papers (60.7%) made reference to the terms psychologically literate citizen, psychologically literate global citizen, or global citizen; a model for the interrelatedness amongst these concepts is proposed.

• There are relatively few papers (12.5%) that address the measurement of PL (other than through formal in-curricular assessment), and this is identified as a gap in the PL literature.

• There are a few papers (5.4%) that explicitly discuss PL as a unifying concept in psychology.

• All papers had relevance to education, with the greatest proportion of papers relevant to the undergraduate level (84.8%) and the least to the graduate level (1.8%).

• There are a substantial number of papers that provide educators with practical teaching (69.6%) and assessment (31.3%) strategies for PL.

• There are a substantial number of papers (54.5%) that address PL as a pedagogical philosophy, including the facets of evidence-informed teaching (24.1%) and (otherwise) role-modeling PL (23.2%).

In terms of implications, although significant work has been undertaken in the conceptual development of PL, further research and policy work is required to reach consensus amongst international educators and researchers. This will allow more precise operationalization and measurement, whether such measurement is through instruments that can be applied to any population, or through formal assessment in educational contexts. The general definition by Cranney et al. (2022a , b ) of PL as the “intentional values-driven application of psychology Knowledge to achieve personal, professional, and community goals” (p. 3), can be contrasted with the group definition consisting of: knowledge (including skills); research methods; application to personal domain; application to professional domain; application to community domain; communication; critical thinking; values & ethics; and cultural responsivity. The first five are the core discipline capabilities; the last are generic capabilities that are deeply contextualized to psychology, including being informed by psychological science. Most importantly, the paper is a provocation that argues for international consensus-seeking regarding psychology education outcomes, particularly at undergraduate and pre-tertiary levels. Relatedly, significant work is being undertaken in the area of PL as a pedagogical philosophy, with the challenge being to provide educators with opportunities to develop this promising practice.

Limitations of this research include the restriction to English language papers and it is noted, for example, that the term has long been used, with a somewhat different meaning, in Russian psychology education ( Karandashev, 2011 ). We invite scholars in different cultures and in non-English-language speaking countries to engage with this Anglo-American analysis of PL, and to challenge and enrich our thinking about the concept. Also excluded by applying the criterion of “culturally different meaning of psychological literacy” was a paper by Cotter et al. (2021) , because it describes a “system” (rather than individual) as psychologically literate. As Morris et al. (2021) described:

“Cotter, a psychologist, led his multidisciplinary co-author team at a mental health hospital to use evidence-based psychological interventions (including relational, emotional and problem-based coping strategies, and positive psychology interventions) to support front-line staff during the initial COVID-19 emergency. Essentially the psychologist is using his psychological knowledge and skills to change organizational behavior (what he refers to as the “psychological literacy” of the ‘system')” (p. 18).

However, it could be argued that what Cotter et al. (2021) described in this paper was PL-in-action, as orchestrated by the psychologist's integration of evidence-based psychological strategies into the workplace, in order to deal with the extreme demands on staff during this stressful time. That is, any employee with appropriate psychological training should be able to flexibly apply psychological strategies in their workplace, to optimize human functioning in usual or unusual situations – thus displaying their psychological literacy.

Another limitation of this paper is the lag between the search strategies and the submission of the paper for publication, which meant that some 3 current in-press or recently published papers were not included in the Covidence analysis. This is reflective of the ongoing activity in this topic area. A further limitation is the potential for bias on the part of raters, and although procedures were adopted to reduce any such bias, replication by other research groups clearly would be good practice.

Significantly, the peer-review process for this paper prompted us to think more deeply about the past 14 years of work in this area. Is the “general” definition of PL merely one component of McGovern et al. (2010) group definition of PL, as one peer-reviewer commented? Our initial reaction was to argue that what is meant by the general definition (e.g., “intentional values-driven application of psychology Knowledge to achieve personal, professional, and community goals”, Cranney et al., 2022a , p.3), is that it represents a “capstone” of all the other PL “group” capabilities. For example, one needs knowledge of research methods to evaluate a behavior change program, as well as knowledge of the target behavior and the target population (be that oneself, clients or employees, or the general public) and context-specific communication and critical & creative thinking skills, values and ethics, and cultural responsivity.

On further consideration, we realized that we have not clearly articulated that the PL “group” capabilities are “inputs” to the final “output” of general PL. There may be different patterns of emphasis on learning outcomes (as one peer-reviewer argued) at different educational levels (e.g., less emphasis on application of knowledge to the professional domain at pre-tertiary level) and in different cultures and nations (e.g., more emphasis on application of knowledge to the professional domain in some UK undergraduate programs), and so there will be some differences in what general PL “looks like” at the end of that educational program (see the argument in Section Interim discussion about variable but well integrated PL schemas; Cranney and Dunn, 2011b ). In Figure 4 , we demonstrate this idea in terms of Cranney et al. (2022a , b ) suggested capabilities. Alternatives would be nationally agreed intended learning goals/outcomes/capabilities. Nevertheless, this alternative conceptualization could accommodate a list of internationally agreed broad capabilities, whereby there may be national differences in emphasis on each expected learning outcome/capability. In terms of operationalizing the “capstone” output of “general” PL, there could be several options. Firstly, operationalization could be in terms of measuring the agreed PL group capabilities (although this introduces the issue of whether we are measuring PL inputs rather than outputs). Secondly, operationalization could be in terms of measuring the IPI “themes” ( APA, 2021 ), as representing the ways in which a psychologically literate individual should think and behave . However, again we encounter challenges in regards to whether this first-year level schema is well enough developed for the “standard” baccalaureate-level moderate amount of PL.

Figure 4 . General Psychological Literacy (PL) as the Outcome of Example Psychology Major's Teaching and Assessment Strategies that are Aligned with Intended Learning Outcomes/Goals/Capabilities. # 1 . Teaching and assessment (T&A) strategies are (to be determined) inputs to the intended learning outcomes/goals/capabilities, but through constructive alignment (backward design), the T&A strategies are influenced by the intended learning outcomes/goals/capabilities, which previously have been conceptualized as “group” psychological literacy (PL), here exemplified by Cranney et al. (2022a , b ) proposed capabilities. The final output is “general” PL (see final paragraphs of the General Discussion for emerging ideas regarding this general PL definition). The numbers are approximations of what would normally be expected for a standard undergraduate major, which we assume consists of a minimum 10 core units with additional core or elective units within a 24-unit baccalaureate degree program. Although there may be different emphases for national goals (e.g., USA – “liberal education” or “open” major - vs. Italy – where most students are “bound” to become practicing psychologists or psychological scientists), we focus here on a universal core, with variations in such national goals expressed in terms of additional core and elective units (= ++) required (e.g., see #3). The percentages in the second column are estimates only, to give the reader a sense of differences in the patterns of emphasis on different intended learning outcomes, which then influences the pattern of PL outcome. Introductory psychology units, whether at pre-tertiary or university level, would be the equivalent of one unit, and in comparison to the psychology major pattern, have greater emphasis on knowledge, and less emphasis on application to the professional domain. # 2 General PL is here exemplified by Cranney et al. (2022a) , p.3; definition: “intentional values-driven application of psychology Knowledge to achieve personal, professional, and community goals”. The 3 application domains of PL interact with each other (see Cranney and Morris, 2011 ; Cranney et al., 2022b , for further detail). A moderate level of general PL is expected after taking the minimum core psychology major units. See the final paragraphs of the General Discussion regarding new ideas regarding pre-tertiary education. # 3 . Application of Knowledge to the Professional Domain includes evidence-based employability skills (as a result of career-development learning strategies) regardless of career destination, and are essential for both types of psychology majors. The “open” major may require additional units to prepare students for more diverse career destinations and, given the liberal education tradition, one might expect the same for application to the community domain. In contrast, with “psychologist-bound” majors, there is the expectation that graduates will receive more career preparation in graduate training, and so additional units may be allocated according to local or national needs.

Thirdly, prompted by a peer-reviewer, we chose to draw on empirically supported motivation theories (e.g., Ryan and Deci, 2000 ) and the broader psychological and philosophical literature [e.g., Nussbaum (1997 , 2006) notion that liberal education should lead to the capacity for constructive self-criticism, which is essential for human development and wellbeing] to re-invigorate previous conceptualizations of psychologically literate graduates of psychology education programs (e.g., Cranney and Morris, 2011 ; Hulme et al., 2015 ). Finally, for the sake of illustrative simplicity, we have chosen just two educational program/level examples; two different categories of psychology major, one “liberal education/open” psychology major, where a small percentage of graduates become professional psychologists (e.g., UK, USA), and the other “psychologist-bound”, where the majority of graduates become professional psychologists (e.g., Italy; Job et al., 2011 ). Essentially, we argue that there should be the same core coverage of a certain pattern of emphasis on the different learning outcomes/goals/capabilities, but that these two categories of psychology major would be distinguished by additional core or elective content. In both cases, a moderate level of general PL would be acquired. If one were to include the further example of introductory psychology (IP) at pre-tertiary or tertiary level, then the pattern of coverage of the capabilities would differ, and of course coverage would be much less, however each capability would still receive some minimal coverage, and the extent of psychological literacy acquired would be low. In terms of teaching and assessment strategies, in Figure 4 , we have designated these as “to be determined”, because this level of detail is beyond the scope of this paper. As indicated in the Section How prevalent are papers that provide educators with practical teaching and assessment strategies for PL?, resources already exist – these need to be categorized and aligned with the learning outcomes/capabilities.

As indicated above, the “unpacking” of the PL definition in Figure 4 draws on previous conceptualizations by Cranney and Morris (2011) and Hulme et al. (2015) who specified general domains to which psychological knowledge, skills and attitudes, acquired through education, could be applied. This notion is more fully developed in Cranney et al. (2022b) , ( Figure 2 ), whereby Bronfenbrenner (1979) model of the various socio-cultural influences on individual's thoughts, emotions, and behaviors is essentially “tipped on its head” through the “arming” of individuals with psychological capabilities. That is, through psychology education, individuals learn how to, and can choose whether to, actively influence those socio-cultural factors, rather than be passively influenced by them. These ideas are congruent with philosophical notions relating capabilities such as self-criticism to quality of life and wellbeing ( Nussbaum and Sen, 1993 ; Nussbaum, 1997 , 2006 ). These ideas are also congruent with Fraissl (2022) German-language analysis of the potential outcomes of psychology education, including that psychologically educated people look at everyday phenomena through the lens of their psychological training. As a result, they can behave in a more self-determined way to manage their everyday lives (see Figure 4 , top component of general PL). This also is expressed in Job et al. (2011) argument regarding the benefits of PL at the individual level:

“ Having competence and skills that allow one to fulfill a role in finding appropriate solutions to problems is likely to create positive emotions and to increase the individual's sense of self-efficacy…. That is, one's beliefs about one's capabilities to exercise influence over events that affect one's own and other people's lives. It may also foster the sense of being an agent of change rather than a passive observer …” (p. 169).

These ideas are also congruent with the ingredients of Ryan and Deci (2000) empirically supported Self-Determination Theory. Both the group and general definitions of PL proposed by Cranney et al. (2022a , b ) are motivational in nature, as reflected in the Figure 4 “unpacking” of PL. A sub-theory of Self-determination Theory proposes three psychological needs: competence, autonomy, and relatedness. The three general PL components in Figure 4 (a) reflect autonomy in choice of goal, and competence acquired through continuous learning, and (b) emphasize the importance of constructive relationships (relatedness) in all domains of our lives. Note also that the satisfaction of these basic psychological needs leads to increased wellbeing ( Bahrami and Cranney, 2018 ). Not explicit in Figure 4 is Cranney and Morris (2011) call for psychologically literate individuals/graduates to utilize their capabilities in leadership roles in these different domains, but of course that would be each individual's choice, and it may not be until psychologically literate citizenship is acquired, that such individuals feel compelled to lead.

Finally, as a peer-reviewer constructively commented, we acknowledge that this review has been psychology-major-centric - partly understandable given the origin of the modern meanings of the term ( McGovern et al., 2010 ). The vast majority of high school and university/college students' exposure to the discipline of psychology is through just one or two introductory psychology (IP) units, and so it is critically important for the future wellbeing of both the discipline and the general public, to promote quality outcomes of such psychology education exposure (e.g., Geiss, 2019 ; APA, 2021 ; Nolting and Geiss, 2022 ). There has been criticism of the typical USA-textbook-driven “ topical” approach to the outcomes at this level, that is, a broad coverage of the various topics of “basic” (e.g., biological, cognitive, social, developmental psychology) and more “applied” topics (most commonly, psychological disorders, but sometimes also general health & wellness, organizational or forensic psychology, neuropsychology). One criticism is the fact that topic-based knowledge changes with new research findings. Alternative approaches have included the “ perspectives” approach, which involves, for example, applying major theoretical perspectives in the field (e.g., behaviorist, biological, cognitive, evolutionary, humanistic, psychodynamic) to psychological phenomena, thus promoting transferable critical thinking skills, in the sense of being able to apply multiple perspectives in problem solving (see Nolting and Geiss, 2022 , for an overview). Following a long tradition of German-language psychology textbooks and curriculum design, Nolting and Geiss (2022) present an integrative systems approach to IP:

“ It is a simple fact that the scientific field of psychology can be divided into domains and topics, but this is not the case for psychology's subject matter – mind and behavior – which has to be regarded as a coherent system or organism. Therefore, when structuring a course or textbook, some kind of integration or synopsis is necessary, too. This is the primary purpose of the integrative approach” (p. 2).

The authors provide an integrative schema (distinguishing between the outcomes of receiving/understanding and influencing/changing/acting-on, which is somewhat similar to Figure 1 of Cranney et al., 2022b ) for the common aspects of what psychology is about. They claim that the “special benefits of the integrative pattern are (1) making the general principles more coherent, and (2) facilitating transfer to a vast range of human behavior” (p. 4). This approach holds much promise for IP, and the authors provide practical teaching examples in Supplementary material . In parallel with the development of that paper, the APA, 2021 , p. 1; IP Initiative, in specifying IP learning outcomes, appears to “blend” a topical approach (“Psychology Content: Identify basic concepts and research findings”), a psychological thinking approach (“Scientific Thinking: Solve problems using psychology methods”), and an integrative thinking approach (“Key Themes: Provide examples of psychology's integrative themes”).

Similar to APA (2021) and Nolting and Geiss (2022) , we argue that desirable IP learning outcomes would include: (a) understanding instructor-chosen areas within each core discipline knowledge topic, including knowledge of research methods; (b) acquisition of minimal critical thinking and scientific reasoning skills that would allow critical analysis of claims about human thinking and behavior (e.g., knowing about common cognitive biases such as confirmation bias, and the difference between correlation and causation; knowing the fundamentals of scientific approaches to the investigation of human behavior), and (c) minimal capacity to apply this knowledge and skill to solving problems (or in the motivational language of Figure 4 , achieving goals). At the pre-tertiary level, this latter capacity would focus on application to the personal domain (e.g., using evidence-based strategies to deal with procrastination) and community domain (e.g., using evidence-based strategies to increase inclusivity behaviors in the high school environment). Indeed, the above three factors could be reframed as a minimal level of psychological literacy (and is somewhat congruent with the positions of Fraissl, 2022 ; and Nolting and Geiss, 2022 ), which could inform the final column output in Figure 4 (i.e., what “general” PL consists of) – but this possibility requires further collaborative research and scholarship.

Meanwhile, we should acknowledge that national and local specification of IP learning outcomes is often constrained by three factors. Firstly, there may be lack of knowledge and foresight regarding the high impact of IP; this underlines the need for those passionate about this subject to collaborate in order to influence such leaders to improve the quality of IP outcomes. Secondly, there is often a lack of adequate training of IP educators; IP advocates need to collaborate to share training and other teaching resources. Thirdly, at a tertiary level, under-funding of both undergraduate programs and graduate professional training programs may lead to psychology departments choosing to utilize IP income to fund psychology professional graduate training by offering low-cost IP topic knowledge lecture delivery and assessment, rather than more resource-intensive but deeper learning strategies (e.g., case analyses, problem-based learning, Nolting and Geiss, 2022 ). Again, educators must advocate for the value of IP ( and liberal/open psychology major education at the tertiary level) and share resources. One interim practical solution may be to deliver broad introductory knowledge “cheaply” through lectures or assigned textbook or other readings (with formative quizzes and other strategies to encourage ongoing study), but within tutorial/lab settings, engage students in multiple formative and summative assessable challenges to (a) critically analyze claims about human behavior and (b) creatively solve behavioral problems (or, achieve goals) in personal or community domains, through the application of their developing psychological knowledge, skills, and attitudes (e.g., see Morris and Cranney, 2022 ; re. application to the personal domain).

In conclusion, we argue that one advantage of the reconceptualization of “group” PL is that it allows for both international consensus regarding broad intended learning outcomes/capabilities and national/cultural/educational-level variation on that consensus, through different degrees of emphasis on each of the “group” capabilities. Consensus regarding how to operationalize and measure “general” PL may continue to be a challenge, but that challenge again presents opportunities, such as continued international and culturally-based debate, and the known strengths of multimethod convergence ( Stanovich, 2013 ). We invite informed comment on whether this initial revised conceptualization of PL, based on this review and provoked by peer-review commentary, adds value to the processes and outcomes of psychology education.

Recommendations:

1. To facilitate the conceptual development of PL (and thus its effective measurement/assessment) as well as the further integration of PL into psychology education as the desired learning outcome, as suggested by a peer-reviewer, international efforts are required by relevant stakeholders to reach a consensus on shared aims, intended learning outcomes/goals/capabilities, and thus practices and tools, in developing psychological literacy in formal and informal educational contexts.

2. A suggested starting point would be an international peak psychology discipline organization examining recent propositions regarding the outcomes of psychology education (particularly at the undergraduate, pre-tertiary and public education levels), and seeking consensus, in collaboration with stakeholders. Such an outcome could then trigger follow-up actions at national and institutional levels (see Cranney et al., 2022a for further suggestions). As suggested by a peer-reviewer, cultural differences in psychology education systems may translate to differences in emphasis on individual learning outcomes/goals/capabilities in those local/national educational contexts. These possibilities are accommodated by the alternative conceptualization outlined in this section (see Figure 4 ). The “unpacking” of the general concept of PL should also provoke further development of the operationalization and measurement of PL.

3. Consensus regarding broad psychology education outcomes should facilitate greater focus on curriculum backward design with active and explicit teaching strategies for PL, and authentic assessment and valid measurement of PL. This process could be facilitated by (a) organizational support of formal educator professional development and sharing of practice/resources - institutionally, nationally, and internationally (including beyond English language boundaries), and (b) student partnership curricular innovation.

4. Peak national and international disciplinary organizations could take a greater role in promoting PL (and its inherent benefits for public health and wellbeing, as well as for the organizations themselves), especially through education, including public education.

Finally, O'Hara (2007 ) use of the term PL, although more in the vein of PLC, seems particularly relevant in our COVID-19 world, and echoes Miller's (1969) call to “give psychology away” (p.1074) to improve the human condition. As Morris et al. (2021) summarized:

“O'Hara used “psychological literacy” to refer to an adaptive and positive application of psychology to meet the challenges of the tumultuous 21st century… [arguing] that (a) the world's population requires PL in order to deal with the rapid global cultural shifts and consequent lack of certainty in our everyday lives, and… (b) education is key to this endeavor.” (p.4).

Data availability statement

The original contributions presented in the study are included in the article/ Supplementary material , further inquiries can be directed to the corresponding author/s.

Author contributions

JC conceived the paper. CC managed the literature searches and the Covidence platform. JC, SM, KN, and CC contributed (in that order) to the Covidence decisions. JC wrote the initial drafts of the Sections Introduction, Results and Discussion, and Conclusions and Recommendations sections. CC and KN co-wrote the initial draft of the Method. All authors contributed significantly to the subsequent editing of the paper and approved the final submission manuscript.

Acknowledgments

We would like to thank (a) Rebecca Tyler for assistance with Figures 2 – 4 , (b) the UNSW School of Psychology for covering the article processing fee, and (c) the general support of the author's institutions in the process of completing this research.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2022.913814/full#supplementary-material

1. ^ Boneau (1990) gives the first known published reference to the term. The study involved the collation of 1000 key terms which students should understand. For Boneau, this knowledge then defined psychological literacy. In the second precursor paper, O'Hara (2007 ) used “psychological literacy” in the sense of McGovern et al. (2010) “psychologically literate citizenship”.

2. ^ Essentially, Cranney and Morris (2021) make a distinction between two dimensions – having the theoretical knowledge underlying psychological skills and attitudes (or not), and using/applying psychological knowledge/skills/attitudes (or not). Psychologically literate individuals are high on both dimensions.

3. ^ For example, Morris et al. (2021) ; Cranney et al. (2022a , b ); Machin and Gasson (2022) . Relevant points from these papers were nevertheless integrated into the discussion.

Amsel, E., Allen, L., and Bauer, R. (2014). Psychological Literacy: Its Nature, Acquisition, and Application. Paper presented at the 4th International Conference of Psychology Educators, August, Flagstaff AZ . Available online at: http://www.psychliteracy.com/literature (accessed April 5, 2022).

APA (2011). American Psychological Association. Principles for Quality Undergraduate Education in Psychology . Available online at: http://www.apa.org/education/undergrad/principles.aspx (accessed October 6, 2021).

Google Scholar

APA (2013). American Psychological Association The APA Guidelines for the Undergraduate Psychology Major: Version 2.0 . Available online at: http://www.apa.org/ed/precollege/about/undergraduate-major.aspx (accessed October 6, 2021).

APA. (2021). American Psychological Association The APA Introductory Psychology Initiative . Available online at: https://www.apa.org/ed/precollege/undergrad/introductory-psychology-initiative (accessed October 6, 2021).

Bahrami, Z., and Cranney, J. (2018). Integrated conative model of well-being: from motives to well-being. J. Happiness Stud. Interdiscipl. Forum Subj. Well-Being 4, 961–981. doi: 10.1007/s10902-017-9845-2

CrossRef Full Text | Google Scholar

Banyard, P., and Duffy, K. (2014). Student representations of psychology in the UK. Psychol. Teach. Rev. 2, 110–120. Available online at: https://files.eric.ed.gov/fulltext/EJ1146515.pdf

Banyard, P., and Hulme, J. A. (2015). Giving psychology away: how George Miller's vision is being realised by psychological literacy. Psychol. Teach. Rev . 2, 93–101. Available online at: https://files.eric.ed.gov/fulltext/EJ1146565.pdf

Beins, B., Landrum, E., and Posey, D. (2011). Specialized Critical Thinking: Scientific and Psychological Literacies (STP Presidential Task Force on Psychological Literacy White Paper) . Washington DC: American Psychological Association.

Bernstein, D. (2011). “A scientist-educator perspective on psychological literacy”, in The Psychologically Literate Citizen: Foundations and Global Perspectives , ed. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 281–295. doi: 10.1093/acprof:oso/9780199794942.003.0069

Bernstein, D., Pooley, J. A., Cohen, L., Goldthorpe, B., Provost, S., Cranney, J., et al. (2020). Psychology: An International Discipline in Context: Australian and New Zealand Edition. (3rd ed.) . Melbourne, Australia: Cengage Learning. ISBN 978-0-17-021841-2.

Boneau, C. A. (1990). Psychological literacy: a first approximation. Am. Psychol . 45, 891–900. doi: 10.1037/0003-066X.45.7.891

PubMed Abstract | CrossRef Full Text | Google Scholar

Boyack, K. W., Klavans, R., and Börner, K. (2005). Mapping the backbone of science. Scientometrics 64, 351–374. doi: 10.1007/s11192-005-0255-6

Bringle, R. G., Reeb, R., Naude, L., Ruiz, A., and Ong, F. (2022). “Service learning: An innovative pedagogy for the psychology curriculum,” in International Handbook of Psychology Learning and Teaching , eds. J. Zumbach, D. Bernstein, S. Narciss, and P. Marsido (New York, NY: Springer).

Bringle, R. G., Reeb, R. N., Brown, M. A., and Ruiz, A. I. (2016). Service Learning in Psychology: Enhancing Undergraduate Education for the Public Good . Washington, DC: American Psychological Association. doi: 10.1037/14803-000

Bronfenbrenner, U. (1979). The Ecology of Human Development . Cambridge: Harvard University Press.

Brooker, A., and Woodyatt, L. (2019). Psychological wellbeing and distress in higher education. Student Success 10, i–vi. doi: 10.5204/ssj.v10i3.1419

Bryan, J., Ranzjin, R., Balfour, C., Tuckey, M., Hayward, R., Pearson, E., et al. (2012). “Increasing the work-readiness of Australian psychology undergraduates by changing the curriculum,” in Teaching Psychology Around the World (Volume 3) , eds. S. McCarthy, K. Laurie Dickson, J. Cranney, A. Trapp, and V. Karandashev (Newcastle upon Tyne, UK: Cambridge Scholars Publishing) 164–179.

Burton, L. J., Chester, A., Xenos, S., and Elgar, K. (2013). Peer mentoring to develop psychological literacy in first-year and graduating students. Psychol. Learn. Teach. 12, 136–146. doi: 10.2304/plat.2013.12.2.136

Burton, L. J., and McDonald, K. J. (2011). “Introductory psychology and psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 91–103. doi: 10.1093/acprof:oso/9780199794942.003.0032

Butler, H. A., and Halpern, D. F. (2013). “Developing and assessing skills of psychological literacy and critical thinking in our students,” in Assessing Teaching and Learning in Psychology: Current and Future Perspectives , ed. D. S. Dunn, S.C. Baker, C.M. Mehrotra, R.E. Landrum, and M. A. McCarthy, (Belmont, CA: Wadsworth Cengage Learning) 82–93.

Charlton, S., and Lymburner, J. (2011). “Fostering psychologically literate citizens: A Canadian perspective” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 234–248. doi: 10.1093/acprof:oso/9780199794942.003.0060

Cormack, S., Bourne, V., Deuker, C., Norton, L., O'Siochcru, C., and Watling, R. (2014). The future of pedagogical action research in psychology. Psychol. Teach. Rev . 20, 95–109. Available online at: https://files.eric.ed.gov/fulltext/EJ1146475.pdf

Cotter, P., Jhumat, N., Garcha, E., Papasileka, E., Parker, J., Mupfupi, I., et al. (2021). A systemic response to supporting frontline inpatient mental health staff in coping with the COVID-19 outbreak. Ment. Health Rev. J . 26, 18–31. doi: 10.1108/MHRJ-05-2020-0026

Coulson, D., and Homewood, J. (2016). Developing psychological literacy: is there a role for reflective practice? J. Uni. Teach. Learn. Pract. 13, 1–18. doi: 10.53761/1.13.2.5

Cranney, J. (2008). Sustainable and Evidence-based Learning and Teaching Approaches to the Undergraduate Psychology Curriculum . Available online at: http://altf.org/wp-content/uploads/2016/08/fellowships_report_cranney_dec08.pdf (accessed October 6, 2021).

Cranney, J. (2013). Towards psychological literacy: a snapshot of evidence-based learning and teaching. Australian J. Psychol. 65, 1–4. doi: 10.1111/ajpy.12013

Cranney, J. (2020). “Why should psychological literacy and self-management capability be key outcomes for undergraduate students?,” in Innovations in a Changing World , eds. K. Andrews, F. A. Papps, V. Mancini, L. Clarkson, K. Nicholson Perry, G. Senior, and E. Brymer (Melbourne: Navatas Ltd) 3–13.

Cranney, J., Andrews, A., and Morris, S. (2016). Curriculum Renewal to Build Student Resilience and Success: Phase 1 [ID12-2381]: Final Peer-reviewed Report. Australian Government Office for Learning and Teaching . Available online at: https://ltr.edu.au/resources/ID12-2381_UNSW_Cranney_Final%20report.pdf (accessed October 6, 2021)

Cranney, J., Botwood, L., and Morris, S. (2012a). National Standards for Psychological Literacy and Global Citizenship: Outcomes of Undergraduate Psychology Education, 2012 Final Report. ALTC/OLT National Teaching Fellowship . Available online at: http://altf.org/wp-content/uploads/2016/08/Cranney_J_NTF_Final-Report_2012.pdf (accessed October 5, 2021).

Cranney, J., Carroll, P., Crowe, S., Littlefield, L., Montgomery, I., Baker, C., et al. (2012b). “Promoting psychological literacy within the community”, in Psychology 2020: The 2011–2012 Presidential Initiative on the future of psychological science in Australia , eds. S. Crowe (Melbourne: Australian Psychological Society), 57–65. Available online at: https://www.psychology.org.au/Training-and-careers/psychological-science/Academics/2011-12-Presidential-Initiative (accessed October 5, 2021).

Cranney, J., and Dunn, D. S. (2011a). The Psychologically Literate Citizen: Foundations and Global Perspectives . New York, NY: Oxford University Press. doi: 10.1093/acprof:oso/9780199794942.001.0001

Cranney, J., and Dunn, D. S. (2011b). “Psychological literacy and the psychologically literate citizen: New frontiers for a global discipline,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 3–12. doi: 10.1093/acprof:oso/9780199794942.003.0014

Cranney, J., and Dunn, D. S. (2011c). “What the world needs now is psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 322–336. doi: 10.1093/acprof:oso/9780199794942.003.0077

Cranney, J., Dunn, D. S., Hulme, J. A., Nolan, S. A., Morris, S., and Norris, K. (2022a). “Psychological literacy and undergraduate psychology education: an international provocation,” in Frontiers in Education. vol. 7. doi: 10.3389/feduc.2022.790600

Cranney, J., Dunn, D. S., Hulme, J. A., Nolan, S. A., Morris, S., and Norris, K. (2022b). Graduate capabilities for undergraduate psychology education: a proposal. [Supplementary Material to Cranney, J., Dunn, D.S., Hulme, J.A., Nolan, S.A., Morris, S., and Norris, K. (2022). Psychological literacy and psychology undergraduate education: an international provocation. in Frontiers in Education . vol. 7. Retrieved https://www.frontiersin.org/articles/10.3389/feduc.2022.790600/full#supplementary-material .

Cranney, J., and Morris, S. (2011). “Adaptive cognition and psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 251–268. doi: 10.1093/acprof:oso/9780199794942.003.0063

Cranney, J., and Morris, S. (2018). “Undergraduate capstone experiences and psychological literacy,” in Teaching Psychology Around the World: Volume 4 , eds. G. J. Rich, A. Padilla-Lopez, L. K. de Souza, L. Zinkiewicz, J. Taylor, and J. L. S. B. Jaafar (Newcastle on Tyne, UK: Cambridge Scholars Press), 306–328.

Cranney, J., and Morris, S. (2021). “Psychological literacy in undergraduate psychology education and beyond,” in Handbook on the State of the Art in Applied Psychology , eds. P. Graf and D. Dozois (Hoboken, NJ: Wiley-Blackwell) 315–337.

Cranney, J., Morris, S., and Botwood, L. (2015). “Psychological literacy in undergraduate psychology education,” in The Oxford Handbook of Undergraduate Psychology Education , ed. D. S. Dunn (New York, NY: Oxford University Press) 863–872.

Cranney, J., Morris, S., Krebs-Lazendic, L., and Hutton-Bedbrook, K. (2020). “Back to the education future–evidence-based student-centred approaches to online curriculum design and delivery,” in Tertiary Online Teaching and Learning: TOTAL Perspectives and Resources for Digital Education , ed. S. McKenzie, F. Garivaldis, and K. R. Dyer (Singapore: Springer Nature) 119–128. doi: 10.1007/978-981-15-8928-7_10

Cranney, J., Morris, S., Krochmalik, A., and Botwood, L. (2013). “Assessing psychological literacy,” in Assessing Teaching and Learning in Psychology: Current and Future Perspectives , ed. D. S. Dunn, S.C. Baker, C.M. Mehrotra, R.E. Landrum, and M. A. McCarthy (Belmont, CA: Wadsworth Cengage Learning) 95–106.

Cranney, J., Morris, S., Martin, F., Provost, S., Zinkiewicz, L., Reece, J., et al. (2011). “Psychological literacy and applied psychology in undergraduate education,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 146–164. doi: 10.1093/acprof:oso/9780199794942.003.0041

Davidson, G. R., and Morrissey, S. A. (2011). “Enhancing ethical literacy of psychologically literate citizens,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 41–55. doi: 10.1093/acprof:oso/9780199794942.003.0023

Denson, N., and Ing, M. (2011). “Educational psychology and psychological literacy in higher education,” in The Psychologically Literate Citizen: Foundations and Global Perspectives, eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 104–118. doi: 10.1093/acprof:oso/9780199794942.003.0034

Dickson, K. L. (2012). “The hallmark of psychology: Science and application,”, in Teaching Psychology Around the World: Volume 3 , ed. S. McCarthy, K. L. Dickson, J. Cranney, V. Karandashev and A. Trapp (Newcastle on Tyne, UK: Cambridge Scholars Press) 214–230.

Dudgeon, P., Darlaston-Jones, D., and Bray, A. (2018). “Teaching Indigenous psychology: A conscientisation, de-colonisation and psychological literacy approach to curriculum,” in Teaching Critical Psychology: International Perspectives , eds. C. Newnes and L. Golding (New York, NY: US Routledge/Taylor and Francis Group) 123–147. doi: 10.4324/9781315209319-7

Dudgeon, P., Darlaston-Jones, D., and Clark, Y. (2011). “Changing the lens: Indigenous perspectives on psychological education,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 72–90. doi: 10.1093/acprof:oso/9780199794942.003.0029

Dunn, D. S., Cautlin, R. L., and Gurung, R. A. R. (2011). “Curriculum matters: Structure, content and psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 15–26.

Dunn, D. S., and Halonen, J. S. (2020). The Psychology Major's Companion: Everything You Need to Know to Get Where You Want to Go (2nd ed) . New York, NY: Worth Publishers.

Dunn, D. S., Hammer, E. Y., and Weiten, W. (2015). “Psychology of adjustment: An opportunity for teaching and promoting psychological literacy,” in The Oxford Handbook of Undergraduate Psychology Education , ed. D. S. Dunn (New York, NY: Oxford University Press) 431–438. doi: 10.1093/oxfordhb/9780199933815.013.031

Dunn, D. S., and McMinn, J. G. (2015). “Teaching psychology: Reflecting on the art and science of quality pedagogy,”, in The Oxford Handbook of Undergraduate Psychology Education , ed. D. S. Dunn (New York, NY: Oxford University Press) 11–23. doi: 10.1093/oxfordhb/9780199933815.013.003

Elliot, D. (2020). Publishing with dissertation students: a covert strategy for developing psychology students' employability skills? Psychol. Teach. Rev . 26, 96–102. Available online at: https://files.eric.ed.gov/fulltext/EJ1257777.pdf

Fraissl, D. (2022). Psychologische Bildung, Eine philosophische Annaherung. Wiesbaden, Germany: Springer. doi: 10.1007/978-3-658-37696-3

Freedman, G., Oates, I. G., and Kirk, S. E. (2021). Applying social psychology to a global crisis: Student engagement in a laboratory class during the COVID-19 pandemic. Schol. Teach. Learn. Psychol. [preprint]. doi: 10.1037/stl0000227

Geiss, P. G. (2019). Rethinking psychological literacy for introductory courses in pre-tertiary and higher psychology education. Psychol. Teach. Rev . 25, 46–53. Available online at: https://files.eric.ed.gov/fulltext/EJ1216455.pdf

Goedeke, S., and Gibson, K. (2011). What do new psychology students know about psychology? Australian Psychol. 46, 133–139. doi: 10.1111/j.1742-9544.2011.00028.x

Green, R. A., Conlon, E. G., and Morrissey, S. A. (2017). Task values and self-efficacy beliefs of undergraduate psychology students. Australian J. Psychol . 69, 112–120. doi: 10.1111/ajpy.12125

Green, S., Robinson, P. L., and Oades, L. G. (2011). “The role of positive psychology in creating the psychologically literate citizen,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 119–130. doi: 10.1093/acprof:oso/9780199794942.003.0036

Gurung, R. A. R., and Neufeld, G. (2022). Transforming Introductory Psychology: Expert Advice on Teacher Training, Course Design, and Student Success . Washington: American Psychological Association. doi: 10.1037/0000260-000

Halonen, J. S., Dunn, D. S., Baker, S., and McCarthy, M. A. (2011). “Departmental program approaches for educating psychologically literate citizens,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 131–145. doi: 10.1093/acprof:oso/9780199794942.003.0038

CrossRef Full Text

Halpern, D. (2010). “Introduction: A call to action,” in Undergraduate education in psychology: a blueprint for the future of the discipline, ed. D. F. Halpern (Washington, DC: American Psychological Association) 3–7. doi: 10.1037/12063-000

Halpern, D. F., and Butler, H. A. (2011). “Critical thinking and the education of psychologically literate citizens,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 27–40. doi: 10.1093/acprof:oso/9780199794942.003.0020

Hamilton, K., Morrissey, S. A., Farrell, L. J., Ellu, M. C., O'Donovan, A., Weinbrecht, T., et al. (2018). Increasing psychological literacy and work readiness of Australian psychology undergraduates through a capstone and work-integrated learning experience: current issues and what needs to be done. Aust. Psychol . 53, 151–160. doi: 10.1111/ap.12309

Harre, N., Milfont, T. L., Helton, W., and Mead, A. (2011). “Sustainability and the psychologically literate citizen,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 220–233. doi: 10.1093/acprof:oso/9780199794942.003.0057

Harris, R., Pownall, M., Thompson, C., Newell, S. J., and Blundell-Birtill, P. (2021). Students' understanding of psychological literacy in the UK undergraduate curriculum. Psychol. Teach. Rev. 27, 56–68. Available online at: https://files.eric.ed.gov/fulltext/EJ1304625.pdf

PubMed Abstract | Google Scholar

Heritage, B., Roberts, L. D., and Gasson, N. (2016). Psychological literacy weakly differentiates students by discipline and year of enrolment. Front Psychol. 7, 162. doi: 10.3389/fpsyg.2016.00162

Higgins, J. P., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., et al. (2011). The cochrane collaboration's tool for assessing risk of bias in randomised trials. BMJ 343, d5928. doi: 10.1136/bmj.d5928

Horn, D., Grotzbach, D., and Drechsel, B. (2021). Fostering preservice teachers' psychological literacy by counseling pupils on their self-regulated learning - Didactical concept of a theory-practice learning. Psychol. Learn. Teach. 20, 279–229. doi: 10.1177/1475725720973517

Hulme, J. (2014). Psychological Literacy - from classroom to real world. Psychologist 27, 932–935. Available online at: https://www.bps.org.uk/psychologist/psychological-literacy-classroom-real-world

Hulme, J., and Cranney, J. (2022). “Psychological literacy and learning for life,” in International Handbook of Psychology Learning and Teaching , eds. J. Zumbach, D. Bernstein, S. Narciss, and P. Marsico (New York, NY: Springer). Available online at: http://www.psychliteracy.com/literature

Hulme, J. A. (2019). “Giving psychology away” to non-psychologists. Comment on Psychology curricula for non-psychologists: a framework recommended by the EFPA board of educational affairs. Psychol. Learn. Teach . 18, 134–136. doi: 10.1177/1475725719827710

Hulme, J. A., and Kitching, H. J. (2017). The nature of psychology: reflections on university teachers' experiences of teaching sensitive topics. Psychol. Teach. Rev . 23, 4–14. Available online at: https://files.eric.ed.gov/fulltext/EJ1146401.pdf

Hulme, J. A., Skinner, R., Worsnop, F., Collins, E., Banyard, P., Kitching, H. J., et al. (2015). Psychological literacy: a multifaceted perspective. Psychol. Teach. Rev . 21, 13–24. Available online at: https://files.eric.ed.gov/fulltext/EJ1146570.pdf

Hulme, J. A., and Winstone, N. E. (2017). Do no harm: risk aversion versus risk management in the context of pedagogic frailty. Knowl. Manage. E-Learn. 9, 261–274. doi: 10.34105/j.kmel.2017.09.016

James, A. I. (2014). Cross-age mentoring to support A-level pupils' transition into Higher Education and undergraduate students' employability. Psychol. Teach. Rev. 20, 79–94. Available online at: https://centaur.reading.ac.uk/73903/1/Crossage%20mentoring_PTR%20Paper_Author%20copy.pdf

Job, R., Lotto, L., and Tonzac, C. (2011). “Psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 167–177. doi: 10.1093/acprof:oso/9780199794942.003.0044

Karandashev, V. (2011). “Psychological literacy goals in psychology teaching in Russian education,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 206–219. doi: 10.1093/acprof:oso/9780199794942.003.0055

Kent, A., and Skipper, Y. (2015). Making a difference with psychology: reporting on a module to develop psychological literacy in final year undergraduates. Psychol. Teach. Rev . 21, 35–47. Available online at: https://files.eric.ed.gov/fulltext/EJ1146561.pdf

Landrum, R. E., and McCarthy, M. A. (2018). Measuring the benefits of a bachelor's degree in psychology: promises, challenges, and next steps. Schol. Teach. Learn. Psychol. 4, 55–63. doi: 10.1037/stl0000101

Lilienfeld, S. O., Pydych, A. L., Lynn, S. J., Latzman, R. D., and Waldman, I. D. (2017). 50 differences that make a difference: a compendium of frequently confused term pairs in psychology. F. Education 2, 37. doi: 10.3389/feduc.2017.00037

Luong, R., and Butler, K. (2022). Informal psychology education and the fundamental attribution error: Testing the effectiveness of accessible online passages. Current Psychol. (in press) doi: 10.1007/s12144-021-02432-w

Machin, M. A., and Gasson, N. (2022). “An introduction to careers in the psychological sciences,” in The Australian Handbook for Careers in Psychological Science , eds. M. A. Machin, T. M. Machin, P. N. Hoare, and C. Jeffries (CC BY NC 4.0. University of Southern Queensland), 17–38. Available online at: https://usq.pressbooks.pub/psychologycareers/ (accessed March 14, 2022).

Mair, C., Taylor, J., and Hulme, J. (2013). An introductory guide to psychological literacy and psychologically literate citizenship . Available online at: https://www.heacademy.ac.uk/sites/default/files/resources/psychological_literacy_and_psychologically_literate_citizenship.pdf (accessed March 14, 2022).

Martin, F., Tose, H., and McBain, B. (2020). “A new Bachelor of Science: Embedding scientific and psychological literacy,” in Teaching Psychology Around the World: Volume 5 , ed. G. J. Rich, A. Padilla-Lopez, L. Ebersohn, J. Taylor, and S. Morrissey (Newcastle on Tyne, UK: Cambridge Scholars Press), 187–204.

McGovern, T. V. (2011). “Virtues and character strengths of psychologically literate faculty,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 296–350. doi: 10.1093/acprof:oso/9780199794942.003.0071

McGovern, T. V., and Brewer, C. L. (2012). “Undergraduate education in psychology,” in Handbook of psychology: History of Psychology , eds. D. K. Freedheim and I. B. Weiner (New York, NY: Wiley) 507–529. doi: 10.1002/9781118133880.hop201025

McGovern, T. V., Corey, L. A., Cranney, J., Dixon, W. E., Holmes, J. D., et al. (2010). “Psychologically literate citizens,” in Undergraduate Education in Psychology: Blueprint for the Discipline's Future , ed. D. Halpern (Washington, DC: American Psychological Association) 9–27. doi: 10.1037/12063-001

Miller, G. A. (1969). Psychology as a means of promoting human welfare. Am. Psychol . 24, 1063–1075. doi: 10.1037/h0028988

Morris, S., and Cranney, J. (2022). Promoting student wellbeing through dedicated units on the psychological Science of wellbeing: rationale, nature, and student evaluations. Psychol. Learn. Teach. (in press). doi: 10.1177/14757257221098024

Morris, S., Cranney, J., Baldwin, P., Mellish, L., and Krochmalik, A. (2018). The Rubber Brain: A Toolkit for Optimising Your Study, Work, and Life . Bowen Hills, QLD: Australian Academic Press.

Morris, S., Cranney, J., Jeong, J. M., and Mellish, L. (2013). Developing psychological literacy: Student perceptions of graduate attributes. Aust. J. Psychol. 65, 54–62. doi: 10.1111/ajpy.12010

Morris, S., Norris, K., and Cranney, J. (2021). “Psychological Literacy,” in Oxford Bibliographies in Psychology , ed. Dana S. Dunn (New York, NY: Oxford University Press).

Morrissey, S., Weinbrecht, T., Farrell, L., O'Donovan, A., and Hamilton, K. (2018). “Facilitating psychological literacy in third-year undergraduate students through university-industry work integrated learning,” in Teaching Psychology Around the World: Volume 4 , ed. G. J. Rich, A. Padilla-Lopez, L. K. de Souza, L. Zinkiewicz, J. Taylor, and J. L. S. B. Jaafar (Newcastle on Tyne, UK: Cambridge Scholars Press), 329–348.

Murdoch, D. D. (2016). Psychological literacy: proceed with caution, construction ahead. Psychol. Res. Behav. Manag . 9, 189–199. doi: 10.2147/PRBM.S88646

Murdoch, D. D., Gregory, A., and Eggleton, J. M. (2015). Why psychology? An investigation of the training in psychological literacy in nursing, medicine, social work, counselling psychology, and clinical psychology. Canad. Psychol. Psychol. Canad . 56, 136–146. doi: 10.1037/a0038191

National Psychology Curriculum Roundtable (NPCR) (2015). The Australian Curriculum: Psychological Science . Available online at: https://www.psychology.org.au/Training-and-careers/psychological-science/Educators/Secondary-school (accessed October 6, 2021).

Newell, S. J., Chur-Hansen, A., and Strelan, P. (2020). A systematic narrative review of psychological literacy measurement. Aust. J. Psychol . 72, 123–132. doi: 10.1111/ajpy.12278

Newell, S. J., Chur-Hansen, A., and Strelan, P. (2021): A construct validity analysis of the concept of psychological literacy. Aust. J. Psychol. 73, 462–474. doi: 10.1080/00049530.2021.1922069

Newstead, S. E. (2015). Psychological literacy and ‘the emperor's new clothes. Psychol. Teach. Rev . 21, 3–12. Available online at: https://files.eric.ed.gov/fulltext/EJ1146631.pdf

Nissley, G. E., and Atwood, M. (2015). “Holistic learning to holistic help: Use of case-based education in promoting psychological literacy in clinical psychological education,” in Telling stories: The art and science of storytelling as an instructional strategy , ed. K. Brakke and J. A. Houska (Washington, DC, US Society for the Teaching of Psychology) 136–144. Available online at: https://www.dropbox.com/s/wuplg1q7nvd7548/tellingstories.pdf?dl=0 ] (accessed October 6, 2021).

Nolting, H.-P., and Geiss, P. G. (2022). System first: An integrative approach to introductory psychology. Teach. Psychol. [preprint]. doi: 10.1177/00986283211058659

Nussbaum, M. C. (1997). Cultivating Humanity: A Classical Defense of Reform in Liberal Education . Cambridge: Harvard University Press. doi: 10.2307/j.ctvjghth8

Nussbaum, M. C. (2006). Education and democratic citizenship: Capabilities and quality education. J. Human Develop . 7, 385–395. doi: 10.1080/14649880600815974

Nussbaum, M. C., and Sen, A. (1993). Quality of Life . Oxford: Clarendon Press. doi: 10.1093/0198287976.001.0001

O'Hara, M. (2007). Psychological literacy for an emerging global society: another look at Rogers' “persons of tomorrow” as a model. Person-Centered Exper. Psychother. 6, 45–60. doi: 10.1080/14779757.2007.9688427

Pownall, M., Havelka, J., and Harris, R. (2021). Scientific blogs as a psychological literacy assessment tool. Teach. Psychol. [preprint]. doi: 10.1177/00986283211027278

Pusateri, T. P., Landrum, R. E., Gurung, R. A. R., and Halonen, J. S. (2019). Pragmatic suggestions for improving psychology's perceived relevance for non-majors: comment on Dudke et al., 2019. Psychol. Learn. Teach. 18, 140–143. doi: 10.1177/1475725719831494

QAA. (2019). Quality Assurance Agency for Higher Education UK. Subject Benchmark Statement: Psychology . Available online at: https://www.qaa.ac.uk/docs/qaa/subject-benchmark-statements/subject-benchmark-statement-psychology.pdf (accessed October 6, 2021).

Reddy, P., Lantz, C., and Hulme, J. A. (2013). Employability in psychology: A guide for departments. York: Higher Education Academy . Available online at: https://www.heacademy.ac.uk/knowledge-hub/employability-psychology-guide-departments (accessed: March 14, 2021).

Roberts, L. D., and Gasson, N. (2018). “Facilitating psychological literacy: The importance of conceptualisation and measurement,” in Teaching Psychology Around the World: Volume 4 , ed. G. J. Rich, A. Padilla-Lopez, L. K. de Souza, L. Zinkiewicz, J. Taylor, and J. L. S. B. Jaafar (Newcastle on Tyne, UK: Cambridge Scholars Press) 293–305.

Roberts, L. D., Heritage, B., and Gasson, N. (2015). The measurement of psychological literacy: a first approximation. F. Psychol. 6, 1–12. doi: 10.3389/fpsyg.2015.00105

Ryan, R. M., and Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am. Psychol . 55, 68–78. doi: 10.1037/0003-066X.55.1.68

Sawano, S. W. (2011). “An Indonesian perspective on psychological literacy,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 178–190. doi: 10.1093/acprof:oso/9780199794942.003.0046

Sokol, B. W., and Kuebli, J. E. (2011). “Psychological literacy: Bridging citizenship and character,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 268–280. doi: 10.1093/acprof:oso/9780199794942.003.0067

Sokolova, L., and Williamson, M. (2020). “The journey into psychology starts at school: Pre-tertiary psychology education in Europe,” in Teaching Psychology Around the World: Volume 5 , ed. G. J. Rich, A. Padilla-Lopez, L. Ebersohn, J. Taylor, and S. Morrissey (Newcastle on Tyne, UK: Cambridge Scholars Press) 340–353.

Stanovich, K. (2013). How to Think Straight About Psychology: International Edition . 10th Ed. Boston: Pearson.

Takooshian, H., and Landi, G. (2011). “Psychological literacy: an alumni perspective,” in The Psychologically Literate Citizen: Foundations and Global Perspectives, eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 306–321. doi: 10.1093/acprof:oso/9780199794942.003.0073

Taylor, J. (2019). “Psychological literacy for all: an overview of this ‘literacy' and how it is relevant for students of all disciplines,” in EDULEARN19: Conference Proceedings of the 11th International Conference on Education and New Learning Technologies . Spain: Palma. doi: 10.21125/edulearn.2019.1126

Taylor, J., and Coady, S. (2019). “Measuring elements of psychological literacy in students: comparing gender, educational level and discipline,” in EDULEARN19: Conference Proceedings of the 11th International Conference on Education and New Learning Technologies. Spain: Palma. p. 5033–5038. doi: 10.21125/edulearn.2019.1248

Taylor, J., and Hulme, J. (2015b). Introducing a compendium of psychological literacy case studies: reflections on psychological literacy in practice. Psychol. Teach. Rev . 21, 25–34. Available online at: https://files.eric.ed.gov/fulltext/EJ1146628.pdf

Taylor, J., and Hulme, J. (2018a). International edition of the psychological literacy compendium . Available online at: http://www.psychliteracy.com/resources (accessed April 5, 2022)

Taylor, J., and Hulme, J. A. (2015a). Psychological Literacy: A Compendium of Practice . Available online at: http://eprints.bournemouth.ac.uk/22906/ (accessed April 5, 2022)

Taylor, J., and Hulme, J. A. (2018b). “An overview of psychological literacy in practice from the UK,” in Teaching Psychology Around the World: Volume 4 , ed. G. J. Rich, A. Padilla-Lopez, L. K. de Souza, L. Zinkiewicz, J. Taylor, and J. L. S. B. Jaafar (Newcastle on Tyne, UK: Cambridge Scholars Press) 362–379.

Tomcho, T. J., and Foels, R. (2017). Psychological literacy in applied psychology disciplines: back to, or beyond, the basics? Teach. Psychol. 44, 108–116. doi: 10.1177/0098628317692613

Trapp, A., and Akhurst, J. (2011). “A UK Perspective on psychological literacy and citizenship,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 191–205. doi: 10.1093/acprof:oso/9780199794942.003.0051

Trapp, A., Banister, P., Ellis, J., Latto, R., Miell, D., and Upton, D. (2011). The Future of Undergraduate Psychology in the United Kingdom . York, UK: UK Higher Education Academy Psychology Network. Available online at: https://groups.psychology.org.au/Assets/Files/Future%20UG%20UK.pdf (accessed: April 5, 2022).

Turner, M., and Davila-Ross, M. (2015). Using oral exams to assess psychological literacy: the final year research project interview. Psychol. Teach. Rev . 21, 48–68. Available online at: https://files.eric.ed.gov/fulltext/EJ1146560.pdf

Watt, R. (2013). Developing the psychologically literate citizen at the University of Stirling . Available online at: https://www.advance-he.ac.uk/knowledge-hub/developing-psychologically-literate-citizen-university-stirling (accessed April 5, 2022)

White, F. (2011). “The social psychology of intergroup harmony and the education of psychologically literate citizens,” in The Psychologically Literate Citizen: Foundations and Global Perspectives , eds. J. Cranney and D. S. Dunn (New York, NY: Oxford University Press) 56–71. doi: 10.1093/acprof:oso/9780199794942.003.0025

Winstone, N. E., and Hulme, J. A. (2017). “Integrative disciplinary concepts: The case of psychological literacy,” in Pedagogic Frailty and Resilience in the University , eds. I. M. Kinchin and N. E. Winstone (Rotterdam: Sense Publishers) 93–107. doi: 10.1007/978-94-6300-983-6_7

Winstone, N. E., and Kinchin, I. M. (2017). Teaching sensitive issues: psychological literacy as an antidote to pedagogic frailty. Psychol. Teach. Rev . 23, 15–29. Available online at: https://files.eric.ed.gov/fulltext/EJ1146406.pdf

Zacharova, Z. J., Sokolova, L., and Lemesova, M. (2015). “Interactive teaching of psychology using design-based research in developing an interactive psychology course,” in Rethinking Education by Leveraging the ELearning Pillar of the Digital Agenda for Europe! (Bucharest: National Defence University Publish House), vol. 1. p. 186–193.

Keywords: psychological literacy, psychology education, undergraduate, citizenship, pedagogical approach, Covidence systematic review, international consensus

Citation: Cranney J, Morris S, Norris K and Connolly CE (2022) Charting the psychological literacy landscape: Systematic review highlighting psychology education. Front. Educ. 7:913814. doi: 10.3389/feduc.2022.913814

Received: 06 April 2022; Accepted: 21 July 2022; Published: 12 August 2022.

Reviewed by:

Copyright © 2022 Cranney, Morris, Norris and Connolly. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Jacquelyn Cranney, j.cranney@unsw.edu.au

Mindfulness Interventions for OCD: A Systematic Review and Meta-Analysis

Saul Mcleod, PhD

Educator, Researcher

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, Ph.D., is a qualified psychology teacher with over 18 years experience of working in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

• This meta-analysis examined the effectiveness of mindfulness interventions (MIs), including mindfulness-based interventions (MBIs) and mindfulness-informed therapies (MITs), for obsessive-compulsive and related disorders (OCRDs).
• Results of the meta-analysis provide initial evidence for medium to large benefits of mindfulness interventions in reducing overall symptom severity across OCRDs. However, benefits were limited for common symptoms like obsessions and depression.
• Treatment gains were not sustained at follow-up, suggesting mindfulness may not solve the problem of OCD relapse.
• This highlights the need to continue optimizing mindfulness protocols by incorporating strategies that target these symptoms directly, rather than just overall severity.
• Studies from lower-income countries using acceptance and commitment therapy (ACT) contributed larger reductions in OCD symptoms and depression.
• Tolin (2023) argues there is an overlap between ERP and acceptance/mindfulness approaches in OCD treatment. Both emphasize decreasing compulsions and exposure exercises, though they differ in emphasis. ERP emphasizes exposure more, while acceptance focuses on letting go of control of thoughts.

Obsessive-compulsive and related disorders (OCRDs) are serious mental health conditions involving obsessions and compulsions that can significantly impair functioning (American Psychiatric Association, 2013).

First-line treatments like exposure therapy have limitations, including high relapse rates and complexity in practice (Gillihan et al., 2012; Olatunji et al., 2013).

Mindfulness interventions emphasize present-moment awareness with an attitude of non-judgment (Kabat-Zinn, 2004). They have shown promise for improving both physical and mental health across disorders (de Vibe et al., 2017; Demarzo et al., 2015; Goldberg et al., 2022).

The mindfulness skill of non-reactivity, or allowing thoughts and feelings to come and go without reacting, may be particularly helpful for managing intrusive thoughts in OCD (Hawley et al., 2017; Landmann et al., 2019).

Despite high anxiety and depression comorbidity in OCRDs (Fineberg et al., 2013), no systematic review has evaluated mindfulness interventions for the full range of OCRDs and common symptoms like obsessions and depression.

Establishing an evidence base for these interventions can inform treatment guidelines and help address the limitations of first-line approaches.

• This systematic review and meta-analysis followed PRISMA guidelines (Page et al., 2021). Five databases were searched for RCTs on mindfulness interventions for OCRDs.
• Eligibility criteria: 8+ years old with OCRD diagnosis, mindfulness interventions compared to any control groups, validated OCRD outcome measures.
• 26 RCTs with 1281 participants were included. Meta-analyses calculated pooled standardized mean differences (SMDs) in OCRD severity between mindfulness and control groups.
• Subgroup analyses examined individual symptoms and depression. Meta-regressions explored the influence of study characteristics on effectiveness.
• 1281 participants aged 13.9-46.2 years (mean 31.2), 34.5% male.
• 88.5% had formal OCRD diagnosis, 11.5% by self-report. 15 studies focused on OCD, 6 on BDD, 5 on TTM. No studies on hoarding or skin picking.
• Most from high income countries. Race/ethnicity data lacking.
• Mindfulness interventions had medium effects on OCD at post-treatment (SMD = -0.62) and small effects at follow-up (SMD = -0.48). High heterogeneity (I2=90%) but minimal publication bias.
• Medium effects were found for body dysmorphic disorder (BDD) at post-treatment (SMD = -0.65) and follow-up (SMD = -0.74). No heterogeneity or publication bias.
• Large effects for trichotillomania (TTM) at post-treatment (SMD = -1.52), low heterogeneity. No follow-up data.
• Small reductions in OCD obsessions at post-treatment (SMD = -0.32) but not follow-up. Small decreases in OCD depression at post-treatment only.
• Meta-regressions: larger OCD reductions in lower-income countries and with ACT. MBCT trended towards greater OCD and depression reductions

Implications

• First-line treatments for OCRDs have limitations – mindfulness may help address these.
• Considering the high psychiatric burden of OCRDs, mindfulness interventions represent a promising treatment approach both on their own and as an adjunct to established therapies.
• Mindfulness was less effective for obsessions and comorbid depression, highlighting areas for refinement.
• Lower-income countries showed greater benefits, suggesting potential for cost-effective treatment.
• Tailoring protocols like ACT and MBCT that directly target OCD and depression appears promising.
• Lack of long-term follow-up data for most OCRDs indicates need for studies on sustainability.

Future Research

More research is vital to strengthen the evidence base so these interventions can effectively help address the significant patient need.

• Further research needed across OCRD diagnoses, symptoms, demographics, and adverse effects.
• Further high quality RCTs with larger samples are needed to establish stronger evidence across the range of OCRDs, patient demographics, and sustainable outcomes.
• Research is lacking on hoarding disorder, skin picking disorder, long-term outcomes for TTM, and adverse effects of MIs. More high-quality RCTs on OCRDs are needed.
• It is unclear if acceptance/mindfulness represent “intentional” treatments where the key components drive effectiveness, or “inadvertent” treatments where incidental exposure exercises are active. More dismantling research is needed.
• Potential risks of mindfulness for OCD patients have not been thoroughly examined.

Strengths & Limitations

The study had many methodological strengths, including:

• Rigorous PRISMA guidelines followed, with low risk of bias overall
• Controlled for exposure techniques to isolate unique effects of mindfulness
• Assessed multiple OCRDs and common symptoms like obsessions and depression
• Examined influence of methodological factors through meta-regressions
• Used statistical methods to assess heterogeneity and publication bias

However, this study was limited in a few ways:

• High heterogeneity in analyses, reflecting the diverse nature of OCRDs
• Small samples for BDD and TTM limit generalizability of findings
• Meta-regressions likely underpowered due to few studies
• Groupings for exposure techniques may be imperfect without study protocols
• Most studies lacked data on race/ethnicity and adverse effects
• It’s unknown if effects are due to mindfulness specifically versus other treatment components.
• It’s unclear which specific OCD patients mindfulness works best for (e.g. severity level, subtype).

This is the first comprehensive meta-analysis demonstrating medium to large benefits of MIs for overall symptom severity across multiple OCRDs – OCD, BDD, and TTM. The effects were sustained without exposure techniques, highlighting the unique therapeutic benefits of mindfulness.

Mindfulness allows practicing non-reactivity to intrusive thoughts, a key skill for managing obsessions in OCD. However, the meta-analysis found only transient benefits for obsessions and no effects on comorbid depression.

This indicates the need to optimize protocols by incorporating strategies that directly target these symptoms, rather than just overall severity. Exploring evidence-based MIs like MBCT and ACT appears fruitful, as they showed greater OCD reductions.

The lack of follow-up data for TTM and absence of trials for other OCRDs underscores the need for more research across disorders.

Considering the high psychiatric comorbidity with OCRDs, using mindfulness as an adjunct or stand-alone treatment offers promise but requires more rigorous study.

Primary Paper

Pseftogianni, F., Panagioti, M., Birtwell, K., & Angelakis, I. (2023). Mindfulness interventions for obsessive–compulsive and related disorders: A systematic review and meta-analysis of randomized controlled trials. Clinical Psychology Review, 233-243. https://doi.org/10.1037/cps0000132

Other References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.).

de Vibe, M., Bjørndal, A., Fattah, S., Dyrdal, G. M., Halland, E., & Tanner-Smith, E. E. (2017). Mindfulness-based stress reduction (MBSR) for improving health, quality of life and social functioning in adults: A systematic review and meta-analysis. Campbell Systematic Reviews, 13 (1), 1-264. https://doi.org/10.4073/csr.2017.11

Demarzo, M. M. P., Montero-Marin, J., Cuijpers, P., Zabaleta-del-Olmo, E., Mahtani, K. R., Vellinga, A., Vicens, C., Lopez-del-Hoyo, Y., & García-Campayo, J. (2015). The efficacy of mindfulness-based interventions in primary care: A meta-analytic review. Annals of Family Medicine, 13 (6), 573–582. https://doi.org/10.1370/afm.1863

Fineberg, N. A., Haddad, P. M., Carpenter, L., Gannon, B., Sharpe, R., Young, A. H., Joyce, E., Rowe, J., Wellsted, D., Nutt, D. J., & Sahakian, B. J. (2013). The size, burden and cost of disorders of the brain in the UK. Journal of Psychopharmacology, 27 (9), 761–770. https://doi.org/10.1177/0269881113495118

Gillihan, S. J., Williams, M. T., Malcoun, E., Yadin, E., & Foa, E. B. (2012). Common pitfalls in exposure and response prevention (EX/RP) for OCD. Journal of Obsessive-Compulsive and Related Disorders, 1 (4), 251–257. https://doi.org/10.1016/j.jocrd.2012.05.002

Goldberg, S. B., Riordan, K. M., Sun, S., & Davidson, R. J. (2022). The empirical status of mindfulness-based interventions: A systematic review of 44 meta-analyses of randomized controlled trials. Perspectives on Psychological Science, 17 (1), 108–130. https://doi.org/10.1177/1745691620968771

Hawley, L. L., Rogojanski, J., Vorstenbosch, V., Quilty, L. C., Laposa, J. M., & Rector, N. A. (2017). The structure, correlates, and treatment related changes of mindfulness facets across the anxiety disorders and obsessive compulsive disorder. Journal of Anxiety Disorders, 49 , 65–75. https://doi.org/10.1016/j.janxdis.2017.03.003

Kabat-Zinn, J. (2004). Wherever you go, there you are: Mindfulness meditation for everyday life. Piatkus Books.

Landmann, C., Tuschen-Caffier, B., Moritz, B., Külz, S., & K, A. (2019). Mindfulness predicts insight in obsessive–compulsive disorder over and above OC symptoms: An experience-sampling study. Behaviour Research and Therapy, 121 , Article 103449. https://doi.org/10.1016/j.brat.2019.103449

Olatunji, B., Davis, M., Powers, M., & Smits, J. (2013). Cognitive-behavioral therapy for obsessive-compulsive disorder: A meta-analysis of treatment outcome and moderators. Journal of Psychiatric Research, 47 (1), 33–41. https://doi.org/10.1016/j.jpsychires.2012.08.020

Page, M., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., Mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S., Chou, R., Glanville, J., Grimshaw, J., Hróbjartsson, A., Lalu, M., Li, T., Loder, E., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Journal of Clinical Epidemiology , 134, 178–189. https://doi.org/10.1016/j.jclinepi.2021.03.001

Tolin, D. F. (2023. Mindfulness and acceptance for OCD: New direction or more of the same? Clinical Psychology: Science and Practice, 23 (3), 248-250. https://doi.org/10.1037/cps0000142

Further Reading

• Adamis, A. M., Jessup, S. C., & Olatunji, B. O. (2023). Strengths and shortcomings of a mindful approach to the treatment of obsessive-compulsive and related disorders. Consulting Psychology Journal: Practice and Research, 72 (4), 244–247. https://doi.org/10.1037/cps0000160
• Tolin, D. F. (2023). Mindfulness and acceptance for OCD: New direction or more of the same? Consulting Psychology Journal: Practice and Research, 72 (4), 248-250. https://doi.org/10.1037/cps0000142
• Twohig, M. P. (2009). The application of acceptance and commitment therapy to obsessive-compulsive disorder .  Cognitive and Behavioral Practice ,  16 (1), 18-28.

Learning Check

• How might mindfulness practice help people manage intrusive thoughts or obsessions in OCD? What skills does it teach that could be useful?
• If you were designing a study to test mindfulness for an OCRD, what considerations would guide your choices about the sample, protocol design, comparisons, and outcome measures?
• What do acceptance-based therapists do with OCD patients? How might an ERP protocol be adapted to incorporate elements of mindfulness?
• What might account for the larger benefits of mindfulness found in lower-income countries compared to higher-income ones? What are the implications?
• SAVE ARTICLE

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on June 22, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, June 22). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved November 3, 2023, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, what is your plagiarism score.

• 1) Form your topic
• 2) Craft your search
• 3) Use the databases
• 4) Get articles, books, and journals
• 5) Reading and Writing
• 6) Citing in APA Style
• Free Resources
• Last Updated: Oct 30, 2023 12:06 PM
• URL: https://guides.nyu.edu/psychology

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

• 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
• 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
• 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
• PMID: 30089228
• DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

• Guidelines as Topic
• Meta-Analysis as Topic*
• Publication Bias
• Review Literature as Topic
• Systematic Reviews as Topic*