Systematic Reviews (A How-To Guide): Protocol & Searching

In order to be conducted in a manner that is logical, transparent, reproducible, thorough and subject to a minimum level of bias, systematic reviews need to follow a well-defined, comprehensive, and explicit methodology, or "protocol.''  This protocol must be developed and agreed upon in advance of any attempt to accumulate, manage, or evaluate search results.

The protocol is used as a roadmap throughout the research process and ensures research integrity, accountability, and that the research team remains focused on the original research question.

Elements of the Protocol

• Research question, rationale and objectives
• Ample relevant literature to warrant a systematic review/meta-analysis
• Inclusion and exclusion criteria
• Information sources to be searched
• Proposed key search terms and search strategy
• Methods and tools for screening, data extraction, quality assessment, and bias analysis 
• Data management process and tools
• Team members and contact details
• Timeline

Protocol Development: Checklists*

There are several checklists available to guide the development of a protocol. 

• Joanna Briggs Institute Systematic Review Checklist
• PRISMA-P (2015) Protocol Article

Protocol Development: The Importance of Registering 

It is highly recommended that a protocol is registered once it has been finalized. The development of registries evolved due to the significant increase in systematic reviews being conducted. Protocol registries perform several important functions.

• Reduce duplication of systematic reviews
• Enhance transparency
• Demonstrate credibility 

Registries

Protocols can be registered with one or more registries. Listed below are two well-known registeries. 

• PROSPERO International
  • Guide for Registering with PROSPERO
• Joanna Briggs Institute

*Additional guidance for protocol development is provided in the 'Further Guidance' section of this guide.

Formulating a research question for a systematic review may be a lengthy process that requires rethinking and revisions.  Follow these steps to begin your process.

Background Searching

• Conduct exploratory searches to scope the literature on the topic and determine how your review will differ from others
• Write down keywords and concepts
• Identify key studies related to your topic
• Find systematic reviews on related topics by searching in registries such as:
  • Prospero
  • Cochrane Library
  • Research Registry
  • INPLASY

Form the Question

• Research questions for systematic reviews should be narrow and specific
• If relevant, choose a question framework such as PICO or SPIDER to structure your question
  • Guide to Research Question Frameworks - City, University of London
  • Guide to Frameworks for Research Questions - University of Maryland

Focus & Revise

• During the earliest stages of the search process, your research question will likely change in scope or terminology
• Use Thesaurus, Subject Terms, or Medical Subject Headings in databases to identify alternative keywords and concepts

Systematic reviews require searching in a variety of information sources including but not limited to subscription databases, open access databases, and grey literature.  Consider the various types of information you will need to fully answer your research question, and what types of resources you will need to search for this information.

Grey Literature Definition

"Grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserve by libraries and institutional repositories, but not controlled by commercial publishers; i.e., where publishing is not the primary activity of the producing body."

- 2010 International Conference on Grey Literature, Prague

Loosely defined, grey literature consists of any information not published in a commercial, peer-reviewed journal.  Some types of grey literature used in systematic reviews include, but are not limited to:

• Abstracts
• Conference Proceedings
• Dissertations & Theses
• Pre-Prints
• Clinical Trials
• Clinical or Practice Guidelines
• Working Papers
• White Papers
• Reports
• Standards & Patents
• Policy & Legislation

Why Search Grey Literature?

Searching for grey literature can help reduce the effects of publication bias that exist in peer-reviewed literature.  Peer-reviewed journals often favor publishing studies with significant and positive results, so that studies with conflicting or negative results remain unpublished.  It is important to survey as much literature as possible for a systematic review, including studies with negative or low-impact results.  

Many library databases include filters for different types of grey literature, and there are various online and open-access sources to find grey literature.  Grey literature searches require multiple searches across several platforms to survey what is available.

Find links to selected sources for grey literature below.

Further Reading