User Engagement with A Multimodal Conversational Agent for Self-Care and Chronic Disease Management: A Retrospective Analysis

Recruitment and Study Setting

We included users who engaged with the conversational agent (referred to as “Albert” hereafter) through the Albert Health app on either iOS or Android platforms between January 1, 2022, and December 31, 2023. The app supports the Turkish language and is primarily used in Turkey.

Users accessed the app through three primary channels: (1) downloading it independently from app stores, (2) receiving invitations from healthcare professionals during hospital visits, and (3) being referred by insurance companies as part of health management programs. Inclusion criteria required users to be located in Turkey and to have interacted with the conversational agent at least once after completing the initial tutorial. Users who entered an invalid birthdate (e.g., before 1900 or after their registration date) were excluded from the analysis.

During installation, users were prompted to provide informed consent for the use of their data for research purposes. Upon consenting, they completed an onboarding survey that collected demographic information and details about any health conditions. To encourage continued engagement, a one-time reminder notification was sent on the eighth day after installation.

Mobile Health Assistant

Albert is a digital health platform developed by Albert Health. It comprises multiple health and chronic disease management programs and is freely available for download from the Google Play Store and Apple App Store. As of 2023, Albert has served approximately 150,000 users through partnerships with public and private healthcare institutions and industry stakeholders [24]. The app functions as a multimodal health assistant, offering both voice and screen-based interaction. Screen-based interactions include text input and the use of assistive buttons (initiated by user or in response to app notifications). Both interaction modes (voice and screen-based) required the same number of steps to complete tasks within the CA interface. Albert is built on conversational development platforms, including Google Dialogflow and Rasa, [25, 26] which provide natural language processing (NLP) capabilities such as intent recognition (i.e., identifying the user’s goal or request) and entity extraction (i.e., detecting specific information such as medication names or symptom details). To enable voice input and output, the app integrates Google Speech-to-Text (STT) and Google Text-to-Speech (TTS) services. Voice responses are delivered in a male voice and enhanced using Speech Synthesis Markup Language (SSML) to produce natural and expressive output. These components are orchestrated by the app’s central processing system, referred to as the AI Engine, which integrates NLP models (Rasa and machine learning frameworks provided by Dialogflow) and STT services. The AI Engine receives user audio in streaming format, converts it to text via STT, and then passes the transcribed text to the NLP models for interpretation and guidance. Figure 1; Table 1 illustrate the flow of user engagement and provide a sample interaction to explain this process.

In addition to the CA, Albert provides alternative methods for accessing health information. Users can browse pre-organized content through traditional navigation tabs (such as a library module with a limited number of health-related articles). While these navigation paths offer structured access to information, they require users to know what they are looking for and navigate through multiple screens. In contrast, the CA allows users to directly request specific information through natural language. Within the scope of this study, we did not report engagement with non-conversational features of Albert.

Albert offers multiple programs tailored to either general health management (GEN) or disease-specific self-management (DMP). Each program includes its own version of the CA, adapted from the core Albert system to address specific user needs. DMPs focus on managing particular health conditions—such as diabetes, hypertension, heart failure, and asthma—by providing condition-specific educational content and enabling users to record relevant clinical parameters. In contrast, GEN programs provide general health support and lifestyle guidance without targeting a specific disease.

Intents

We categorized each user–CA conversation into six intent groups based on whether the interaction was clinical (Health Information, Medication-related, Clinical Parameter) or non-clinical in nature (Fallback, Small Talk, and Technical Queries/Support). Interactions involving questions about health topics or access to care—such as FAQs, wellness tips, pharmacy locations, appointment scheduling, or general advice—were grouped under Health Information. Conversations related to medication usage, schedules, or recording were classified as Medication-related. Instances in which users logged clinical data (e.g., blood pressure, blood glucose, body mass index, or symptom journals) were assigned to the Clinical Parameter group. The Fallback category captured responses triggered when the CA was unable to understand the user’s input. Small Talk included casual or social interactions, and Technical Queries/Support encompassed questions related to app use or troubleshooting. (Table 2).

Data Privacy

To protect user privacy, all data were deidentified, and each user was assigned a randomly generated identifier at the time of registration. Encryption protocols were applied to secure the data throughout collection and storage. Given the retrospective nature of this study, the analysis was limited to the existing dataset, and no additional data were collected.

Data Analysis

We analyzed user interactions data (i.e., each query) from users who completed the CA tutorial and used the chat function. Interaction data from tutorial sessions—which occur when a user downloads and initiates the app for the first time—were excluded. The dataset included user demographics, program participation, and detailed engagement metrics (e.g., number and types of interactions; mode of interaction—voice or screen-based interaction).

We first conducted descriptive statistics to summarize user demographics and usage patterns. The Shapiro-Wilk test was used to assess normality. As the data were not normally distributed, we applied the Mann-Whitney U test for group comparisons and calculated Cliff’s Delta to estimate effect size.

To examine whether interaction mode (voice vs. screen-based interaction) varied by gender, intent group, age group, or program group, we constructed contingency tables and used Pearson’s Chi-square tests, with Cramer’s V reported as the effect size.

Lastly, we conducted logistic regression to identify predictors of long-term engagement. Predictor variables included program group (DMP vs. GEN), frequency of speech versus screen-based interaction, internally calculated mean confidence score, age group, and level of clinical context in the interaction. The mean confidence score represents the CA’s natural language understanding confidence level for each user interaction, as provided by the underlying NLP platforms (Dialogflow and Rasa). This score ranges from 0 to 1, with higher values indicating greater confidence in correctly interpreting the user’s intent. The level of clinical context was categorized based on the ratio of clinical to non-clinical interactions: high (ratio > 1, indicating more clinical than non-clinical intents), medium (ratio = 1, indicating equal distribution), and low (ratio < 1, indicating fewer clinical than non-clinical intents). Clinical intents include Health Information, Medication-related, and Clinical Parameter categories as defined previously. Continuous variables were standardized, and Wald-based 95% confidence intervals (CIs) were derived for each odds ratio (OR). We assessed multicollinearity using variance inflation factors (VIFs), and model discrimination was evaluated by plotting the receiver operating characteristic (ROC) curve and calculating the area under the curve (AUC).

Demographics

Of the 24,537 users, 56% of those who reported their gender (n = 15,723) identified as female, and 44% identified as male. Additionally, 36% of the total sample (n = 8,814) did not report their gender. Regarding age, 31% did not disclose their age. Among those who did, 6% (n = 1,012) were aged 0–15, 30% (n = 4,673) were 15–30, 44% (n = 7,584) were 30–45, 17% (n = 3,035) were 45–60, and 3% (n = 568) were 60 or older (see Table 3).

The Shapiro-Wilk test (p < 0.001) indicated non-normal distribution for age, gender, and program type. A binomial test was conducted to assess whether the gender distribution deviated significantly from a 50/50 split. Results showed that female users (56%) were significantly more likely to engage with the CA than male users (p < 0.0001).

Health Condition-Specific Findings

Among all participants, 7.75% (n = 1,901) were enrolled in a DMP, accounting for 8.4% of total interactions and averaging 5.03 interactions per user. In contrast, 92.2% (n = 22,628) participated in a GEN program, contributing 91.6% of total interactions, with an average of 4.60 interactions per user (Table 4).

A Mann-Whitney U test (W = 20,739,146, p = 0.008) indicated a statistically significant difference in interaction frequency between DMP and GEN groups. However, the effect size was negligible (Cliff’s Delta = − 0.0357, 95% CI: − 0.0634 to − 0.0080), suggesting that although DMP users engaged slightly more frequently, the practical difference was minimal.

Engagement and Interaction

Users interacted with Albert most frequently to access health information (32%), followed by logging clinical parameters (16%) and addressing medication-related topics (12%) within the clinical context (Table 5).

Regarding the mode of interaction, females used voice slightly more often (53.1%), whereas males showed a modest preference for screen-based interaction (52.6%) (χ²(1) = 248.89, p < 0.001). Marked differences appeared across intent types: fallback (80.2%) and small talk (64.5%) were predominantly voice-driven, while clinical parameter interactions were mostly initiated via screen-based interaction (60.9%) (χ²(15) = 11,681, p < 0.001). Similarly, non-clinical intents occurred more frequently via voice (66.9%), while clinical tasks were more often completed through screen-based interaction (χ²(3) = 6,457.68, p < 0.001).Age-related trends were also observed. Younger users (ages 0–15) favored voice input, whereas older users (60+) were more likely to use screen-based interaction, though the effect was modest (χ²(4) = 330.55, p < 0.001). Users enrolled in disease management programs showed a higher preference for voice (54.3%) compared to those in general health programs (52.3%) (χ²(1) = 14.24, p < 0.001). Intent type showed the strongest observed association with interaction mode (Cramér’s V = 0.28), consistent with a moderate effect size. Through a manual review of a random subsample (n = 200), we identified that the majority (approximately 80%) were due to non-meaningful speech or slang usage. In addition, 10% of these instances involved users terminating the session prematurely while the remaining 10% were attributable to the speech-to-text system failing to capture specific medical terminology or infrequently used drug names.

User engagement frequency varied notably over time. Interactions peaked sharply on the first day after registration and declined rapidly within the first 10 days. Of the total user base, 14,194 users (58%) completed only one session, while 10,335 users (42%) engaged in two or more sessions (Fig. 2). A secondary peak occurred around day 8, corresponding with the scheduled reminder notification, followed by a stabilization at lower engagement levels. This distinction between single-session and multi-session users was used as a key outcome in the logistic regression analysis.

Logistic Regression Analysis

The logistic regression model (see Table 6) demonstrated moderate discriminative performance (Residual deviance = 31,890, AIC = 31,914, AUC = 0.65). Gender did not have a meaningful effect on long-term usage and was therefore excluded from the final model. Membership in the GEN group was associated with lower odds of long-term engagement compared to the DMP group (OR = 0.67, 95% CI: 0.60–0.74, p < 0.001). A higher percentage of voice-based interactions was positively associated with long-term engagement (OR = 1.005, 95% CI: 1.004–1.006, p < 0.001), while a higher percentage of screen-based interaction was associated with lower odds (OR = 0.994, 95% CI: 0.993–0.995, p < 0.001).

Interestingly, a higher mean confidence score in the CA’s responses was significantly associated with lower odds of sustainable engagement (OR = 0.43, 95% CI: 0.33–0.56, p < 0.001). Younger adults (ages 15–30 and 30–45) were significantly less likely to engage over time (OR ≈ 0.76–0.77), whereas older users (45–60, 60+) showed weaker or nonsignificant differences compared to the reference group. Medium clinical context (OR = 1.56) and low clinical context (OR = 1.07) both significantly increased the odds of long-term engagement relative to high clinical context. Variance inflation factors (VIF) for all predictors showed minimal multicollinearity (< 2).

Limitations

This study has several limitations. First, the data are observational and derived from user interactions within a single mobile application, which limits the generalizability of the findings to other CAs or health apps. Second, reliance on self-reported demographic information —combined with a substantial proportion of users choosing not to disclose their age or gender—may introduce bias and reduce the accuracy of demographic insights. Additionally, the high proportion of single-session users may reflect selection bias, as the dataset primarily represents individuals who opted to engage with the CA at least once. The tendency to use other UI features (e.g., navigating to the app library section for reading materials) may inform the CA use behavior, however, other features were not investigated within the scope of this study. The imbalance between general health management and disease-specific user groups may have further limited comparative analyses across subgroups. Cultural and linguistic factors unique to the target population may also impact the generalizability of results to other settings. Moreover, the study did not assess the role of graphical user interface elements in promoting sustainable engagement. The lack of qualitative data on user satisfaction and reasons for discontinuation limits a more nuanced understanding of user experience. Future research should incorporate qualitative interviews or surveys to explore user disengagement and guide targeted design improvements. Finally, the retrospective design precludes causal inference regarding the relationship between demographics and engagement patterns.

Disclosure Statement

S.C., Z.P.P., M.D. and A.Y. are employees of Albert Health (developer of the mobile app). No other disclosures were reported.

Ethics Declarations

The study was approved by the Biruni University Ethics Committee (Approval No. 2023/83 − 18) and was conducted in accordance with the Declaration of Helsinki. Participants received no payment or other compensation for their participation in the survey. Informed Consent was obtained from all participants.

Competing Interests

The authors declare no competing interests.

Clinical Trial Number

Not applicable.