Hong, Quan Nha (2018).  Revision of the mixed methods appraisal tool (MMAT): A mixed methods study. McGill Family Medicine Studies Online, 13: e08 

Download thesis here

Abstract

Background: Systematic mixed studies reviews (SMSRs), i.e., systematic reviews combining qualitative, quantitative and mixed methods studies, are growing in popularity owing to their potential to provide a rich and practical understanding of complex health interventions and problems. Due to the heterogeneity of the included studies, one challenging issue concerns the critical appraisal of studies. A critical appraisal tool was developed to address this challenge: the Mixed Methods Appraisal Tool (MMAT). The MMAT includes criteria for appraising the methodological quality of five categories of studies: (a) qualitative studies, (b) randomized controlled trials, (c) non-randomized studies, (d) quantitative descriptive studies, and (e) mixed methods studies. Pilot studies provided proof-of-concept for the feasibility of the MMAT and a need for further development.

Objectives: The overall objective of this project was to revise the MMAT. The specific objectives were to identify the changes that need to be made in the MMAT and the most relevant criteria that should be included in the MMAT.

Methods: A sequential exploratory mixed methods design was used. A first phase consisted in a qualitative descriptive study. Semi-structured interviews with researchers having used the MMAT were conducted to identify the strengths, limits, and areas for improvement of the tool. Then, the team composed of 12 researchers with complementary expertise in qualitative, quantitative and mixed methods research met to discuss the results and plan the next step. In a second phase, a modified e-Delphi study was performed with experts in qualitative, survey and mixed methods studies to identify the most relevant critical appraisal criteria. Consensus was reached when at least 80% of experts judged a criterion 'very' or 'extremely' relevant. In addition, a mapping of criteria from 33 existing critical appraisal tools was performed to identify the core criteria for randomized controlled trials and non-randomized studies. The results of these two phases informed the development of a revised version of the MMAT (version 2018).

Results: For the first phase, a total of 20 participants from eight different countries were interviewed. Thirteen main themes were identified and grouped into the dimensions of usefulness, i.e., utility and usability. The themes related to utility concerned the coverage, completeness, flexibility, and other utilities of the tool (educational tool). Those on usability were related to the tool's learnability, efficiency, satisfaction and errors that could be made due to difficulties in understanding or selecting the criteria to rate. For the second phase, respectively 73 and 56 experts participated in Round-one and Round-two of the modified e-Delphi study. The experts were from 11 different countries. Consensus was reached for six qualitative criteria, eight survey criteria, and seven mixed methods criteria. The mapping of the criteria of randomized controlled trials and non-randomized studies led to add new criteria in the MMAT to covers the different categories of bias addressed in critical appraisal tools. On the basis of these results, of the 19 criteria in the MMAT (version 2011), four were removed, seven were reformulated, five were replaced, and ten new were added. Explanations were added in the user manual as well as an algorithm to help reviewers judge and select the criteria to use.

Discussion and conclusion: This project addressed the usefulness and content validity of the MMAT. A revised version of the MMAT was developed and includes 25 criteria on five categories of studies. Changes from the previous version concerned mainly the number of criteria, the user manual, and the overall scoring. This revised version will need to be pilot tested and the website will be modified. Continuous development of the MMAT is required and future research should focus on its validity, reliability, and usefulness.