Transformative Education
for Health Professionals

Methodology of Evidence-Informed Guidelines

The process of developing WHO guidelines encompasses the synthesis of all available published research and grey literature evidence; formal assessment of the quality of evidence; consideration of resource use and costs; and consideration of values and preferences. The formal assessment of quality of evidence includes the use of a transparent system for assessing evidence and rating recommendations following the GRADE methodology. This process links evidence to recommendations and explains the reason that judgements were taken at each step along the way. By design, the process was steered by the WHO secretariat with the support of the core guideline development group that included content experts for specialties involved, a methodologist and representatives of potential stakeholders and that maintained a geographic and gender balance.

Using a multi-pronged methodological strategy for creating evidence-informed guidelines, the approach was inclusive of a comprehensive scoping literature review and analysis of systematic reviews covering other related published evidence, in addition to a review of the grey literature.   

The process of developing the guidelines began in 2009 with an extensive scoping of the literature on health professional education, gathering expert opinion through the formation of a large reference group that met three times in 2010, and building consensus that culminated in the first meeting of the Core Guidelines Development Group in Divonne, France in May 2011. The guidelines have been developed in parallel with a strong implementation platform with the Medical Education Partnership Initiative (MEPI), the Nursing Education Partnership Initiative (NEPI), the United States President's Emergency Plan for AIDS Relief (PEPFAR), and other partners. The secretariat is also engaging additional multi-sector stakeholders and civil society to galvanize support for a global advocacy movement around the need for health professional education and training reform.

The second meeting of the group was called to:

  • advise on the priority of questions and scope of the guideline;
  • advise on the choice of important outcomes for decision-making;
  • comment on the evidence used to inform the guideline;
  • advise on the interpretation of evidence, with explicit consideration of the overall balance of resource use, values and preferences, benefits and risks;
  • formulate recommendations, taking into account diverse values and preferences according to GRADE.

Every effort was made to comply with standards for reporting, processing and using evidence as set by the WHO Guidelines Review Committee (GRC). The Guidelines Development Group used a multi-pronged methodological approach to bring together the evidence available in support of the guidelines, so that countries can use them with reasonable assurance that they will facilitate sound policy development in matters of health professional education.  The following steps were taken:

  • An outcomes framework, based on the guiding principles for transforming and scaling up health professional education and training, was designed to inform the development of the recommendations (Annex 14).
  • A comprehensive literature review was undertaken to provide as wide an understanding of the areas involved as possible initially, followed by the drafting of PICO (population/intervention/comparison/outcome) questions and the commissioning of systematic reviews, covering other related published evidence and grey literature relating to these questions. A template was used for conducting systematic reviews in order to ensure uniformity and comparability and the data collected was recorded in a decision table (see Annex 7 for decision tables).
  • Evidence tables were developed based on the systematic reviews (see here).  The systematic reviewers assessed the risk of bias in each of the randomized studies included, guided by the following criteria:
    • Inadequate sequence generation;
    • Inadequate allocation concealment;
    • Lack of blinding of participants, providers data collectors, outcome adjudicators and data analysts;
    • Incompleteness of outcome data;
    • Selective outcome reporting, and other bias.

The systematic reviewers assessed the risk of bias in each included non-randomized studies guided by the following criteria:

  • Failure to develop and apply appropriate eligibility criteria (e.g. under-or over-matching in case-control studies, selection of exposed and unexposed subjects in cohort studies, selection of exposed and unexposed in cohort studies from different populations);
  • Flawed measurement of both exposure and outcome  (e.g. differences in measurement of exposure such as recall bias in case-controlled studies, differential surveillance for outcome in exposed and unexposed in cohort studies);
  • Failure to adequately control confounding (e.g. failure of accurate measurement of all known prognostic factors, failure to match for prognostic factors and/or adjustment in statistical analysis);
  • Incomplete follow-up.
  • With regard to how the data was synthesized, it should be noted that the same PICO question did not necessarily assess the same outcomes.  Even when they did, they did not necessarily use the same measurement instruments or approaches. Even when using the same instruments or approaches, they did not report enough statistical data to allow a meta-analysis of the data.  As a consequence, we report the results in a narrative manner and when available, we describe the statistical results as reported by the authors of the original studies. This means that in some cases a range of relative effects (e.g. odds ratio or relative ratio) may be presented, while in other cases a relative effect with or without a confidence interval may be presented.
  • In addition to the above, and in order to strengthen the issue of relevance, which is one of the three outcomes guiding these recommendations, two surveys were conducted. A feasibility and acceptability survey gathered the views of 136 stakeholders and potential beneficiaries of the recommendations from all WHO regions; and a civil society survey (169 respondents) provided views and expectations on the main areas of interest in the guidelines. This was also a strategy to better identify the roles and contribution of civil society to the transformational education agenda and movement and, at the same time, to deepen its engagement as a key stakeholder in the work of the Core Group. Discussions were conducted via e-mail and during two workshops that took place in Divonne-les-Bains (France) in May 2011 and in Washington D.C. (USA) in March 2012.
  • Decision tables were developed by the Guidelines Development Group based on all the evidence provided from the evidence tables based on systematic reviews, literature reviews and the feasibility and acceptability study.