Skip to main content
Banner Image
WNHS Internet |   WNHS Hub |  HealthPoint |   WA Health Library Sites
Ask Us!

Research Tips: Systematic Reviews Step-by-Step

Research tips, support, strategies and FAQs

What is a Systematic Review?

Systematic Reviews

An effective systematic review "collates all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question"  (Cochrane Handbook for Systematic Reviews of Interventions) .

The Centre for Reviews and Dissemination at York estimates that a  team will take 9-24 months to complete a systematic review. Systematic reviews can be of interventions (i.e. randomised controlled trials) or observations (i.e. case control or cohort studies).

A systematic review should have:

  • clearly stated objectives with pre-defined eligibility criteria for studies
  • explicit, reproducible methodology
  • a systematic search that attempts to identify all studies
  • assessment of the validity of the findings of the included studies (e.g. risk of bias)
  • systematic presentation, and synthesis, of the characteristics and findings of the included studies

“the quality of a systematic review depends on the quality of the studies appraised. It can be difficult to reach meaningful conclusions from reviews of low-level evidence, and thus, systematic reviews are commonly limited to high-level evidence (Level I or II) studies (RCTs)”.

Wright, R.W.,  Brand, R.A., Dunn,W. & Spindler, K.P. (2007). How to Write a Systematic Review. 

Why are systematic reviews necessary?

Problems with systematic reviews

Introduction to systematic reviews by Susan Shenkin)

Systematic Review versus Literature Review

  

      Criteria           

 

    Systematic Reviews          

            

  Literature Reviews

Question

Focused on a single question (often PICO based)

Not necessarily focused on a single question - may describe an overview

Protocol

A peer reviewed protocol or plan is included

No protocol included

Background

Summarises the available literature

Summarises the available literature

Objectives

Clear objectives are identified

Objectives may or may not be identified

Inclusion/Exclusion Criteria

Criteria stated before the review is conducted

Criteria not stated

Search Strategy

Comprehensive and systematic (stated in the document)

Strategy not explicitly stated (not always comprehensive or systematic)

Process of Selecting Articles

Usually clear and explicit

Not described in a literature review

Process of Evaluating Articles

Comprehensive evaluation of study quality

Evaluation of study quality may or may not be included

Process of Extracting Information

Usually clear and specific

Not clear or explicit

Results & Data Synthesis

Clear summaries of studies based on high quality evidence

Summary based on studies where the quality of the articles may not be specified. May also be influenced by the reviewer's theories, needs and beliefs.

Discussion

Written by an expert or group of experts with a detailed and well grounded knowledge of the issues

Written by an expert or group of experts with a detailed and well grounded knowledge of the issues

(from Curtin University Library's Systematic Review guide)

Loading ...

Systematic Reviews Step-by-Step

Use the tabs above for information about the following steps:

  1. Scope the literature
  2. Form a team
  3. Formulate the question, develop & register the protocol
  4. Systematically search the literature
  5. Data extraction
  6. Quality appraisal
  7. Data synthesis
  8. Interpreting results, presenting the review

 

Step 1: Scope the literature

A systematic review is most useful where:

  • there is a substantive research question
  • several empirical studies have been published 
  • there is uncertainty about the results

 

Step 2: Form a team

The highest quality reviews will have input from experts in:

  • the subject being reviewed
  • systematic review methodology
  • information retrieval
  • statistics
  • reference management
  • other aspects such as qualitative research methods if needed

 

(References: Higgins, J. & Thomas, J. (2019). Cochrane Handbook for Systematic Reviews of Interventions Wright, R.W.,  Brand, R.A., Dunn,W. & Spindler, K.P. (2007). How to Write a Systematic Review. Clinical Orthopaedics and Related Research, 455, pp. 23–29; Simons, M. (2011).  Guidelines for writing systematic reviews; Curtin University Systematic Reviews; Duke University: Systematic Reviews: The Process; University of Edinburgh, Systematic reviews and meta-analyses: a step-by-step guide )

Step 3: Formulate the question, develop & register the protocol

  • Establish a clear & answerable question. Consider using PICOT to focus the question (see Ask the Right Question box). May require several revisions & more scoping of literature.
  • Establish whether the question has already been answered in the published literature or is registered as an ongoing review (search DARE  now available in PubMed Health,  Cochrane, and PROSPERO, or the Campbell Collaboration for social interventions)
  • Write a review protocol to formulate the review question and methods before retrieving the literature, thereby minimizing bias. Include the clear & answerable question; rationale for the review; the search strategy (search terms & resources);  inclusion & exclusion criteria (such as publication date range, language, main focus, explicit methodology, outcome measurements); quality checklist to assess individual studies; data extraction & management strategy;  data synthesis strategy;  dissemination strategy;  timeframe. 
  • RevMan – the software used for preparing & maintaining Cochrane protocols & reviews
  • Register your review to ensure others know it is ongoing. Register with Cochrane for interventions or accuracy of diagnostic tests; with Campbell Collaboration for social interventions in education, crime & justice, social welfare; with PROSPERO for reviews in health or social care. New Cochrane protocols are automatically uploaded to PROSPERO. Each organisation will have its own suggestions for the review protocol  

Step 4: Systematically search the literature

  • At least 2 reviewers are required. They first independently screen titles & abstracts based on the research question (in terms of PICOT). Based on this, full-text articles are retrieved for a second stage screening. The final selection is then submitted for data extraction.
  • Sources to search include: Cochrane Library; DARE (Database of Abstracts of Reviews of Effects); Medline; Embase; PubMed Clinical Queries; subject specific databases (e.g. CINAHL, PsycInfo); conference proceedings; hand search key journals; bibliographies & references listed in primary sources; guidelines (often based on SRs); grey literature (see our Grey Literature page); forward citation searching of seminal articles; foreign language literature (do not limit searches to English); clinical trials (see Trials Registries tab on Grey Literature page ), contact scholars/experts in the field, search the Internet. See the following for more details:
  • The search process should be meticulously documented as it develops. Include:
  1. databases used
  2. date of search
  3. dates of coverage provided by each database
  4. search terms used
  5. total number of publications found
  6. number of relevant publications
  7. limits applied

Include the PRISMA flowchart in the review. Use one of the following online tools for managing your search results:

  • EndNote (freely available to Department of Health staff);
  • Covidence ("improves healthcare evidence synthesis by improving the efficiency and experience of creating and maintaining Systematic Reviews." Supported by The Alfred Hospital, Monash University, National ICT Australia & University of London);
  • EPPI-Reviewer 4 (software for all types of literature reviews including systematic reviews & meta-analyses)

 

Step 5: Data extraction

  • Devise a  standardized form to assist with data extraction.
  • This will generally include details of the reference, study objectives, study design, population, intervention, control, outcome, comments on study quality.

 

Step 6: Quality appraisal

Each study should be evaluated in terms of its:
  • Methodological quality - the extent to which the design & conduct are likely to have prevented systematic errors (bias)
  • Precision - a measure of the likelihood of random errors (confidence interval)
  • External validity - the extent to which the results are generalisable to a particular target population
  • See the appropriate checklists in the Appraise & Calculate boxes. Also see:

Step 7: Data synthesis

  • After including & excluding studies according to the quality appraisal, the analysis of data & results should initially begin with a simple descriptive evaluation of each study, often presented in a table.
  • If the study results are very heterogeneous it may be more appropriate to summarise the data narratively.
  • A statistical analysis (meta-analysis) should include numerical and graphical presentations of the data, look at the strength and consistency of the evidence and investigate reasons for inconsistencies.
  • See Calculate box and

 

Step 8: Interpreting results, presenting the review

  • Most information for this can be presented in the analysis of data & results table. However, the strengths & weaknesses of the included studies should be discussed.

 

(References: Higgins, J. & Thomas, J. (2019). Cochrane Handbook for Systematic Reviews of Interventions Wright, R.W.,  Brand, R.A., Dunn,W. & Spindler, K.P. (2007). How to Write a Systematic ReviewClinical Orthopaedics and Related Research, 455, pp. 23–29; Simons, M. (2011).  Curtin University Systematic Reviews; Duke University: Systematic Reviews: The Process; University of Edinburgh, Systematic reviews and meta-analyses: a step-by-step guide )

Loading ...