Kluchnyk, Maria (2018). Can alerts from smartphone application improve how family medicine residents prepare for certification examination? A mixed methods study. McGill Family Medicine Studies Online, 13: e11
Abstract
Background:In 2014, to help family medicine residents prepare for their certification examination, the ITPCRG team (McGill University-Department of Family Medicine) created an IAM smartphone application. This mobile app was inspired by the concept of spaced education (SE). We operationalized SE as follows: access to clinical information on the 99 Priority Topics in Family Medicine and their Key Features, a weekly alert to read one new topic, open ended test questions, and answer feedback. SE has positive effects on long-term knowledge retention in highly focused specialties. However, SE has not yet been studied in the context of the family medicine residency. This study aimed to describe residents' participation in SE, explore the factors that influenced sustainable participation in SE mediated by weekly alerts on the IAM app, and describe strategies for IAM app improvement from residents' perspectives.
Objectives:Quantitative: (1) To estimate the extent to which an intervention (weekly alerts on the mobile app as a mediator of SE) stimulates residents to visit the IAM app during their first year of training, (2) To identify participants for the second qualitative phase (residents following alerts most closely and least closely). Qualitative: (1) To describe factors, from the perspectives of residents, which facilitate or hinder sustainable participation in SE mediated by weekly alerts, (2) To describe strategies recommended by residents for improvement of the IAM app as an information system for SE.
Methodology and methods: Using a sequential explanatory mixed methods research design, two sequential studies were performed. Phase 1: Pre-experimental study was conducted to describe app use among incoming first year family medicine residents at McMaster (2015) and to identify participants for the second qualitative phase (residents following alerts most closely and least closely). Phase 2: In qualitative descriptive study, 13 residents were interviewed. Semi-structured interviews were analyzed using thematic analysis (deductive-inductive approach) to describe residents' perspectives.
Results:In phase 1, 81 of 96 (84%) first year residents consented to participate in the study. Typical participants were between 25 and 31 years old, predominantly female (71%) and predominantly using iOS devices (76%). Of 81 residents who consented to participate in the study, 65 residents (80%) installed and used the app. A count of page visits per month revealed that the effect of alerts from the IAM app was not sustained. The stimulating effect of alerts was observed only during the first two months of the study. For these who used the IAM app, the number of topics of the week each resident visited within 1-week of the alert (or alert visits) varied from 0 to 34 (mean=1.7, SD=4.5), and the total number of pages each participant visited (total page visits) varied from 1 to 442 (mean=28.3, SD=61.4). In phase 2, both barriers and facilitators to resident's sustainable participation in SE mediated by weekly alerts appeared to fall into five dimensions: user-related factors, information content factors, mobile app design factors, alert system factors, and service factors. Strategies for improvement of the IAM app appeared to fall into four dimensions: information content factors, mobile app design factors, alert system factors, and service factors.
Discussion and conclusion:These study findings were consistent with the literature on SE. Inspired by the DeLone and McLean Information Systems (IS) Success Model, five dimensions (user-related factors, information content factors, mobile app design factors, alert system factors, and service factors) appear to be potential predictors of users' sustainable participation in SE. Results of this study were used to modify the IAM app according to residents' needs. The next phase of the IAM app project will test a new version of the IAM app and measure the effectiveness of SE.