The Review of Usability Evaluation Methods on Tele health or Telemedicine Systems

Ladan Soltanzadeh; Amin Babazadeh Sangar; Kambiz Majidzadeh

Journal ID (publisher-id): FHD

Title: Frontiers in Health Informatics

Abbreviated Title: Front Health Inform

ISSN (electronic): 2322-3596

License (open-access):

This is an Open Access article distributed under the terms of the Creative Commons Attribution License, (http://creativecommons.org/licenses/by/3.0/) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Date received: 21 January 2022

Date accepted: 15 March 2022

Publication date (print): 2020

Publication date (electronic): 11 April 2022

Volume: 10

Issue: 1

Electronic Location Identifier: 112

DOI: [10.30699/fhi.v11i1.357]

The review of usability evaluation methods on telehealth or telemedicine systems

Ladan Soltanzadeh¹ , Amin Babazadeh Sangar^2, ¹ and Kambiz Majidzadeh²

^[1]PhD. Student in Software Engineering, Department of Computer Engineering, Urmia Branch, Islamic Azad University of Urmia, Urmia, Iran

^[2]Assistant Professor, Department of Computer Engineering, Urmia Branch, Islamic Azad University of Urmia, Urmia, Iran

Author notes:

Correspondence to: ^[*]Corresponding author: Amin Babazadeh Sangar

Abstract

Introduction:

Telemedicine in the pandemic of coronavirus disease 2019 (COVID-19) has responded to societal distancing in medical treatments by protecting health workers while also managing available resources. To attain best practices in telemedicine, a platform must be functional, and both patients and clinicians must be satisfied with the technology. To ensure the benefits of telehealth systems, usability refers to how easy the user interfaces of telehealth systems are to use. In this study, the usability of telemedicine systems has been investigated.

Material and Methods:

The authors of this study review the study from 2015 to 2021 using a combination of the keywords "health", "telemedicine", "telemedicine", "mobile health", "usability" "Software", "System" and "Program", which led to the extraction of 119 articles in this field.

Results:

Articles in the field of remote health software and evaluation of the usability of remote health applications in the form of applications based on mobile health technologies, web-based applications or a combination of both types with sample devices Primary are wearable electronics, sensors or robots.

Conclusion:

In this study, most of the remote health software are mobile based and their usability has been evaluated by a questionnaire. Satisfaction is the most important usability attributes to consider when designing Health mobile apps.

Key Words: Healthcare, Telemedicine, Telehealth, Mobile Health, System, Software, Application, Usability

MATERIAL AND METHODS

In this systematic review, the most essential phrase was "usability of the telehealth or telemedicine system". The scope was limited to English literature. Six digital databases were searched to find the target articles: IEEE Xplore, the Web of Science (WoS), Science Direct, PubMed, Google Scholar, and Elsevier's Scopus. In this study, a review was conducted to search for every article related to healthcare, apps, and usability to 2021. A mix of keywords containing "Healthcare", "Tele Health", "Tele Medicine", " mobile health " combined with the "OR" and "AND" operators followed by "Usability" and "AND "Software" and "OR" "System" and "OR" "Application" were used for search.

The initial query resulted in 350 papers. After reviewing the titles of the articles by two of the authors, 235 articles remained and after deleting the duplicate articles and reviewing the abstracts, 170 articles remained. In the final full-text review, 119 articles were included in the study of usability factors in health and health care, and to better explain the concepts and methods of papers in the final set in the study of usability factors in health and health care. The telemedicine system and how to check the usability of those system was assessed. Each article's authors, title, journal, year of publication, usability check model and abstract were entered into an excel file (Fig 1).

Fig 1

Flowchart of study selection

RESULTS

The telemedicine or telehealth systems in which their usability were assessed and included in this review are such as:

smoking cessation mobile apps, tele rehabilitation mobile apps, pain management mobile apps, cardiopulmonary resuscitation mobile apps, diabetes care mobile apps, a self-management of anxiety and depression mobile apps, rheumatoid arthritis mobile apps, oncology cancer care mobile apps, hypertension Management mobile apps, urology care mobile apps, pediatric otolaryngology mobile apps, self-care for pregnant women.

The usability attributes which assessed in each of the articles are shown in Table 1.

Table 1

Usability factors

Usability Factors	References
Learnability	[15, 19-35]
Satisfaction	[19, 20, 25-27, 29-34, 36-96]
Efficiency	[19, 23, 25, 26, 28-35, 38, 40, 41, 43-45, 47, 53-56, 58, 59, 63, 65, 66, 69, 74, 75, 78-80, 82, 83, 86-89, 91, 93-95, 97-102]
Effectiveness	[13, 19, 20, 22, 23, 25-29, 32-35, 38, 40-44, 47, 53-56, 59, 60, 63, 65-67, 70, 74, 75, 78, 79, 81-83, 85-89, 91-96, 98, 99, 102-111]
Memorability	[35]
Errors	[21-23, 25, 27, 29, 76, 77, 79, 88, 98, 106, 107]
Safety	[20, 27, 56, 92, 106, 111-113]
Performance	[19, 20, 23, 25-28, 40, 45, 54, 65, 70-72, 77, 80, 82, 101, 106, 107, 112, 114, 115]
Design	[19-23, 25-33, 35, 40, 42-44, 53, 54, 65, 69, 71, 72, 76-79, 81, 82, 85, 88, 91, 92, 95, 97, 98, 101, 103-105, 107, 109, 110, 112, 114-117]
Navigation	[13, 19-23, 25, 27-33, 35, 38, 40, 42, 43, 49, 53, 54, 66, 67, 72, 75, 79, 80, 82, 83, 88, 91, 106, 107, 109, 114, 117]
Compatibility	[19, 45, 71, 76, 106]
Visibility	[29, 45, 97, 106, 107, 114]

The usability evaluation methods which assessed in each of the articles are shown in Table 2.

DISCUSSION

This study aims to survey usability evaluation methods on tele health or telemedicine systems that run on the Web or Android platform and determine the variety of usability attributes in them. This review focused on the usability evaluation of applications or software designed for telehealth, telemedicine, and health care systems such as telecare monitoring system, mobile applications for Pain Assessment and Management, self-management of hypertension, HIV Interventions, Diabetes, Multiple Sclerosis, Cancer, cardiovascular disease, supporting health care during pregnancy, Psychological Experiences of Web-Based Psychosocial Interventions, interventions for substance use disorder [57, 99].

The results of this study show that mobile phones were the most used because mobile phones can both connect to sensors, run mobile applications, and receive and store data and information and send to o the server located in the health care center for monitoring by health care providers. Mobile apps have a great potential to support patients in healthcare, and to encourage healthy behavioral changes, a set of factors that has impact on the app effectiveness is related to the quality of those features that lead to positive user experiences when using the app. there are so many health mobile apps available, but app usage by patients is low [100], Evaluation of Mobile application usability may be influenced by patient factors such as age, sex, and psychological needs. Mobile Apps can help patients connect with health care providers by supporting email communication and sharing home-monitored data [100]. The need for mobile health approaches to address particular aging features of older adults grows as the population of older adults as a possible consumer group of mobile health grows [10].

The World Health Organization's Global Observatory for eHealth defines mobile health “medical and public health practice supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices” [118].

Table 2

Usability methods

Usability Evaluation Method		Articles
Interview	Interview	[13, 14, 16, 20, 25, 26, 36-44, 47, 49, 53, 54, 57-59, 97-99, 103-105, 114, 116]
	Semi Structured Interviews	[13, 37, 39, 50, 51, 58, 62, 64, 66, 99, 114]
	Qualitative Interview	[45, 55, 57, 59, 103, 105]
	Post study phone interview	[28, 36]
questionnaire	Questionnaire	[19, 25-32, 45, 46, 48, 54-57, 60, 64-66, 69-71, 77, 78, 80-82, 85, 90-92, 101, 107, 113, 115, 117, 119, 120]
	Post-Study System Usability Questionnaire (PSSUQ)	[27, 45, 76, 79, 98]
	Computer Systems Usability Questionnaire (CSUQ)	[36, 61, 80, 84]
	Telehealth Usability Questionnaire	[34, 94, 96, 108]
	System Usability Scale (SUS)	[19, 22, 46, 48, 50, 51, 63, 64, 67, 77, 79, 85, 89, 99, 100, 106, 109, 112]
	Technology Acceptance Model-2 (TAM-2)	[77]
	Client Satisfaction Questionnaire-8, (CSQ-8)	[52]
	User Experience Questionnaire (UEQ)	[65]
NASA Task Load Index (NASA TLX)		[84, 106]
Observations		[37, 47, 50, 51, 55, 60, 66, 68, 69, 118]
Descriptively Reported		[55, 56]
Expert Review		[46]
Think a Loud		[45, 47, 67]
Eye Tracking		[21, 67]
Motion tracking device		[21, 67, 69]

Smartphones have made it easier for healthcare practitioners and the general population to collaborate, gathers data, and makes clinical decisions [22, 88]. The need for mobile health approaches to address particular aging features of older adult's increases as the population of older adults as a potential user group grows. Usability research, however, indicate that mobile health is still not planned properly for older adults and their preferences [87, 117], Older patients are a significant target population for Web-based health information since numerous ailments, such as cancer, diabetes, and hypertension, are diseases that affect the elderly. In elderly individuals who access Web-based health information, improvements in self-efficacy, blood pressure, hemoglobin levels, and cholesterol levels have been noted [69, 83].

To evaluate the usability of mHealth apps, the most common data collection technique utilized in the studies was questionnaire, followed by field study, interview, observation, think-aloud, and app-use generated data.

The SUS was the most common standard questionnaire used by the studies in the review. The SUS, the Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire, and the Post Study System Usability Questionnaire were the most commonly used scales that studies used to create their own questionnaires.

In this review, we found that questionnaire was the most common data collection technique of the included studies; however, researchers either used standard questionnaires, such as SUS or USE, which were not specifically designed for the mental health domain, or adapted a standard questionnaire or developed a new one. Owing to the great variety of the questionnaires, there is a need to establish a common standardized usability questionnaire targeted specifically at mHealth apps.

Tele Health and Telemedicine are a timely response to the limitations that COVID-19 societal distancing imposes on traditional healthcare delivery methods and the COVID-19 pandemic is favoring digital transitions. Health-care organizations quickly adopted digital solutions and advanced technology tools in response to the pandemic's first phase. Telehealth has emerged as a critical component of patient healthcare delivery during the COVID-19 pandemic.

Telehealth can help with enhancing compliance and clinical efficiency, as well as reducing in-person contacts for clinical routes, which is critical during the COVID-19 epidemic [68, 82]. In the most of the reviewed articles, the factors of the ISO model, 9241-11, or Dr. Nielson's model were used to evaluate the usability of the telehealth systems. The effectiveness, efficiency, satisfaction, learnability, memorability, and errors factor were measured in the majority of the reviewed articles, but given the issue of telehealth and the fact that users vary depending on the type of disease and care or intervention required, age and physical condition must improve the usability of systems for upgrading. To make the system designed for users more usable, it is suggested that the usability of telehealth systems be evaluated using twelve factors [16]. Questionnaires are the most often used approach for evaluating health-care systems. The results showed that Satisfaction, Efficiency, Design and Effectiveness are the most important usability attributes to consider when designing Health mobile apps. the authors had previously designed a telemedicine system for heart control for diabetics [121, 122], but its usability had not been studied, so for future work, they will decide to design a telehealth system and study its usability based on the usability evaluation model with twelve factors included Learnability, Satisfaction, Efficiency, Effectiveness, Memorability, Errors, Safety, Performance, Design, Navigation, Compatibility, and Visibility [121, 122].

CONCLUSION

This study provides usability evaluations methods and usability attributes in telemedicine or telehealth systems especially mhealth apps. the mhealth apps are the most developing due for the availability of mobile devices to users, their popularity, and device affordances. the standard questionnaires are the most used for systems usability evaluation. the satisfaction, efficiency, design and effectiveness are the most important systems usability attributes.

AUTHOR’S CONTRIBUTION

All authors contributed to the literature review, design, data collection and analysis, drafting the manuscript, read and approved the final manuscript.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest regarding the publication of this study.

FINANCIAL DISCLOSURE

No financial interests related to the material of this manuscript have been declared.

References

1. Carreiro S, Newcomb M, Leach R, Ostrowski S, Boudreaux ED, Amante D. Current reporting of usability and impact of mHealth interventions for substance use disorder: A systematic review. Drug Alcohol Depend. 2020;215:108201.

2. Zhou L, Bao J, Parmanto B. Systematic review protocol to assess the effectiveness of usability questionnaires in mhealth app studies. JMIR Res Protoc. 2017;6(8):e151.

3. Luna-Perejon F, Malwade S, Styliadis C, Civit J, Cascado-Caballero D, Konstantinidis E, et al. Evaluation of user satisfaction and usability of a mobile app for smoking cessation. Comput Methods Programs Biomed. 2019;182:105042.

4. Tubb MR, Meulen MBV, Pallerla H, Regan S, Doarn CR. Clinical evaluation of e-Quit worRx: A mobile app to enhance smoking cessation shared decision making in primary care. Mhealth. 2019;5

5. Gowarty MA, Longacre MR, Vilardaga R, Kung NJ, Gaughan-Maher AE, Brunette MF. Usability and acceptability of two smartphone apps for smoking cessation among young adults with serious mental illness: Mixed methods study. JMIR Ment Health. 2021;8(7):e26873.

6. Lai B, Wilroy J, Young HJ, Howell J, Rimmer JH, Mehta T, et al. A mobile app to promote adapted exercise and social networking for people with physical disabilities: Usability study. JMIR Form Res. 2019;3(1):e11689.

7. Wilroy J, Lai B, Currie M, Young HJ, Thirumalai M, Mehta T, et al. Teleassessments for enrollment of adults with physical or mobility disability in a home-based exercise trial in response to COVID-19: Usability study. JMIR Form Res. 2021;5(11):e29799.

8. Sun T, Dunsmuir D, Miao I, Devoy GM, West NC, Görges M, et al. In‐hospital usability and feasibility evaluation of Panda, an app for the management of pain in children at home. Paediatr Anaesth. 2018;28(10):897–905.

9. Birnie KA, Nguyen C, Amaral TD, Baker L, Campbell F, Lloyd S, et al. A parent–science partnership to improve postsurgical pain management in young children: Co-development and usability testing of the Achy Penguin smartphone-based app. Can J Pain. 2018;2(1):280–91.

10. Ehrler F, Weinhold T, Joe J, Lovis C, Blondon K. A mobile app (BEDSide mobility) to support nurses' tasks at the patient's bedside: Usability study. JMIR Mhealth Uhealth. 2018;6(3):e57.

11. Tonga E, Williamson E, Srikesavan C, Özen T, Sarıtaş F, Lamb SE. A hand exercise mobile app for people with rheumatoid arthritis in Turkey: Design, development and usability study. Rheumatol Int. 2021;41(6):1151–60.

12. Constantinescu G, Kuffel K, King B, Hodgetts W, Rieger J. Usability testing of an mHealth device for swallowing therapy in head and neck cancer survivors. Health Informatics J. 2019;25(4):1373–82.

13. International Organization for Standardization. Ergonomics of human-system interaction - part 11: Usability: Definitions and concepts [Internet].

14. Salari R, Kalhori SRN, Ghazisaeedi M, Jeddi M, Nazari M, Fatehi F. Mobile-based and cloud-based system for self-management of people with type 2 diabetes: Development and usability evaluation. J Med Internet Res. 2021;23(6)

15. Bevan N. Measuring usability as quality of use. Software Quality Journal. 1995;4(2):115–30.

16. Tark R, Metelitsa M, Akkermann K, Saks K, Mikkel S, Haljas K. Usability, acceptability, feasibility, and effectiveness of a gamified mobile health intervention (Triumf) for pediatric patients: Qualitative study. JMIR Serious Games. 2019;7(3):e13776.

17. van Beukering M, Velu A, van den Berg L, Kok M, Mol BW, Frings-Dresen M, et al. Usability and usefulness of a mobile health app for pregnancy-related work advice: Mixed-methods approach. JMIR Mhealth Uhealth. 2019;7(5):e11442.

18. Welbie M, Wittink H, Westerman MJ, Topper I, Snoei J, Devillé W. A mobile patient-reported outcome measure app with talking touchscreen: Usability assessment. JMIR Form Res. 2019;3(3):e11617.

19. The Health Resources, Services Administration. Telehealth programs [Internet]. :2020 [cited: 01 Dec 2021]. Available from: https://www.hrsa.gov/rural.

20. Shah UM, Chiew TK. A systematic literature review of the design approach and usability evaluation of the pain management mobile applications. Symmetry. 2019;11(3)

21. Agha Z, Weir CR, Chen Y. Usability of telehealth technologies. International Journal of Telemedicine and Applications. 2013;

22. Almeida AF, Rocha NP, Silva AG. Methodological quality of manuscripts reporting on the usability of mobile applications for pain assessment and management: A systematic review. Int J Environ Res Public Health. 2020;17(3):785.

23. Wildenbos GA, Peute L, Jaspers M. Aging barriers influencing mobile health usability for older adults: A literature based framework (MOLD-US). Int J Med Inform. 2018;:114: 66–75.

24. International Organization for Standardization. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) [Internet].

25. Gowarty MA, Aschbrenner KA, Brunette MF. Acceptability and usability of mobile apps for smoking cessation among young adults with psychotic disorders and other serious mental illness. Front Psychiatry. 2021;:12:656538.

26. Frías A, Solves L, Navarro S, Palma C, Farriols N, Aliaga F, et al. Technology-based psychosocial interventions for people with borderline personality disorder: A scoping review of the literature. Psychopathology. 2020;53(5-6):254–63.

27. Thirumalai M, Rimmer JH, Johnson G, Wilroy J, Young HJ, Mehta T, et al. TEAMS (Tele-Exercise and Multiple Sclerosis), a tailored telerehabilitation mHealth app: Participant-centered development and usability study. JMIR Mhealth Uhealth. 2018;6(5):e10181.

28. Dunsmuir D, Wu H, Sun T, West NC, Lauder GR, Görges M, et al. A postoperative pain management mobile app (panda) for children at home after discharge: Usability and feasibility. JMIR Perioper Med. 2019;2(2)

30. McCall T, Ali MO, Yu F, Fontelo P, Khairat S. Development of a mobile app to support self-management of anxiety and depression in African American women: A usability study. JMIR Form Res. 2021;5(8):e24393.

31. Bruggers CS, Baranowski S, Beseris M, Leonard R, Long D, Schulte E, et al. A prototype exercise–empowerment mobile video game for children with cancer, and its usability assessment: Developing digital empowerment interventions for pediatric diseases. Front Pediatr. 2018;

32. Ingadottir B, Blondal K, Thue D, Zoega S, Thylen I, Jaarsma T. Development, usability, and efficacy of a serious game to help patients learn about pain management after surgery: An evaluation study. JMIR Serious Games. 2017;5(2):e10.

33. Radhakrishnan K, Toprac P, O'Hair M, Bias R, Kim MT, Bradley P, et al. Interactive digital e-Health game for heart failure self-management: A feasibility study. Games Health J. 2016;5(6):366–74.

34. Vanosdoll M, Ng N, Ho A, Wallingford A, Xu S, Matin SB, et al. A novel mobile health tool for home-based identification of neonatal illness in Uganda: Formative usability study. JMIR Mhealth Uhealth. 2019;7(8):e14540.

35. Stuti D, Karanam C, Gómez-Orozco C, Gómez-Marín O. Mobile phone intervention for heart failure in a minority urban county hospital population: Usability and patient perspectives. Telemed J E Health. 2017;23(7):544–54.

36. Pimmer C, Tulenko K. The convergence of mobile and social media: Affordances and constraints of mobile networked communication for health workers in low-and middle-income countries. Mobile Media & Communication. 2016;4(2):252–69.

37. Nguyen M, Waller M, Pandya A, Portnoy J. A review of patient and provider satisfaction with telemedicine. Curr Allergy Asthma Rep. 2020;20(11):72 .

38. Wang Y, Zeng L, Yao S, Zhu F, Liu C, Laura AD, et al. Recommendations of protective measures for orthopedic surgeons during COVID-19 pandemic. Knee Surg Sports Traumatol Arthrosc. 2020;28(7):2027–35.

39. Sangar AB, Rastari S. A model for increasing usability of mobile banking apps on smart phones. Indian Journal of Science and Technology. 2015;8(30):1–9.

40. Frías A, Palma C, Salvador A, Aluco E, Navarro S, Farriols N, et al. B·RIGHT: Usability and satisfaction with a mobile app for self-managing emotional crises in patients with borderline personality disorder. Australas Psychiatry. 2021;29(3):294–8.

41. Metelmann B, Metelmann C, Schuffert L, Hahnenkamp K, Brinkrolf P. Medical correctness and user friendliness of available apps for cardiopulmonary resuscitation: Systematic search combined with guideline adherence and usability evaluation. JMIR Mhealth Uhealth. 2018;6(11):e190.

42. Fu HN, Adam TJ, Konstan JA, Wolfson JA, Clancy TR, Wyman JF. Influence of patient characteristics and psychological needs on diabetes mobile app usability in adults with type 1 or type 2 diabetes: Crossover randomized trial. JMIR Diabetes. 2019;4(2):e11462.

43. Harvey C, Koubek R, Bégat V, Jacob S. Usability evaluation of a blood glucose monitoring system with a spill-resistant vial, easier strip handling, and connectivity to a mobile app: Improvement of patient convenience and satisfaction. J Diabetes Sci Technol. 2016;10(5):1136–41.

44. Diamantidis CJ, Ginsberg JS, Yoffe M, Lucas L, Prakash D, Aggarwal S, et al. Remote usability testing and satisfaction with a mobile health medication inquiry system in CKD. Clin J Am Soc Nephrol. 2015;10(8):1364–70.

45. Gladman T, Tylee G, Gallagher S, Mair J, Rennie SC, Grainger R. A tool for rating the value of health education mobile apps to enhance student learning (MARuL): Development and usability study. JMIR Mhealth Uhealth. 2020;8(7):e18015.

46. Liew MS, Zhang J, See J, Ong YL. Usability challenges for health and wellness mobile apps: mixed-methods study among mHealth experts and consumers. JMIR Mhealth Uhealth. 2019;7(1):e12160.

47. Nitsch M, Dimopoulos CN, Flaschberger E, Saffran K, Kruger JF, Garlock L, et al. A guided online and mobile self-help program for individuals with eating disorders: An iterative engagement and usability study. J Med Internet Res. 2016;18(1):e7.

48. Rodríguez S, Sanz AM, Llano G, Navarro A, Parra-Lara LG, Krystosik AR, et al. Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study. PLoS One. 2020;15(5):e0233269.

49. Teo CH, Ng CJ, Lo SK, Lim CD, White A. A mobile web app to improve health screening uptake in men (ScreenMen): Utility and usability evaluation study. JMIR Mhealth Uhealth. 2019;7(4):e10216.

50. Thies K, Anderson D, Cramer B. Lack of adoption of a mobile app to support patient self-management of diabetes and hypertension in a federally qualified health center: Interview analysis of staff and patients in a failed randomized trial. JMIR Hum Factors. 2017;4(4):e24.

51. Chantler T, Paton C, Velardo C, Triantafyllidis A, Shah SA, Stoppani E, et al. Creating connections- the development of a mobile-health monitoring system for heart failure: Qualitative findings from a usability cohort study. Digit Health. 2016;:2: 2055207616671461.

52. Ding H, Fatehi F, Russell AW, Karunanithi M, Menon A, Bird D, et al. User experience of an innovative mobile health program to assist in insulin dose adjustment: Outcomes of a proof-of-concept trial. Telemed J E Health. 2018;24(7):536–43.

53. Tay I, Garland S, Gorelik A, Wark JD. Development and testing of a mobile phone app for self-monitoring of calcium intake in young women. JMIR Mhealth Uhealth. 2017;5(3):e27.

54. Chapman Smith SN, Brown PC, Waits KH, Wong JS, Bhatti MS, Toqeer Q, et al. Development and evaluation of a user-centered mobile telestroke platform. Telemed J E Health. 2019;25(7):638–48.

55. Pinem AA, Yeskafauzan A, Handayani PW, Azzahro F, Hidayanto AN, Ayuningtyas D. Designing a health referral mobile application for high-mobility end users in Indonesia. Heliyon. 2020;6(1):e03174.

56. Stütz T, Emsenhuber G, Huber D, Domhardt M, Tiefengrabner M, Janneke Oostingh G, et al. Mobile phone-supported physiotherapy for frozen shoulder: Feasibility assessment based on a usability study. JMIR Rehabil Assist Technol. 2017;4(2)

57. Liu YC, Chen CH, Lin YS, Chen HY, Irianti D, Jen TN, et al. Design and usability evaluation of mobile voice-added food reporting for elderly people: Randomized controlled trial. JMIR Mhealth Uhealth. 2020;8(9):e20317.

58. Brinkel J, May J, Krumkamp R, Lamshöft M, Kreuels B, Owusu-Dabo E, et al. Mobile phone-based interactive voice response as a tool for improving access to healthcare in remote areas in Ghana: An evaluation of user experiences. Trop Med Int Health. 2017;22(5):622–30.

59. Moeini S, Watzlaf V, Zhou L, Abernathy RP. Development of a weighted well-being assessment mobile app for trauma affected communities: A usability study. Perspect Health Inf Manag. 2020;18(Winter):1o.

60. Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the gulf region. JMIR Res Protoc. 2016;5(3):e93.

61. Kristjansdottir OB, Borosund E, Westeng M, Ruland C, Stenberg U, Zangi HA, et al. Mobile app to help people with chronic illness reflect on their strengths: Formative evaluation and usability testing. JMIR Form Res. 2020;4(3):e16831.

62. Heiney SP, Donevant SB, Adams SA, Parker PD, Chen H, Levkoff S. A smartphone app for self-management of heart failure in older African Americans: Feasibility and usability study. JMIR Aging. 2020;3(1):e17142.

63. Marques ADB, Moreira TMM, Jorge TV, Rabelo SMS, Lima deCarvalho REF, Felipe GF. Usability of a mobile application on diabetic foot self-care. Rev Bras Enferm. 2020;73(4):e20180862 .

64. Rasche P, Mertens A, Miron-Shatz T, Berzon C, Schlick CM, Jahn M, et al. Seamless recording of glucometer measurements among older experienced diabetic patients–A study of perception and usability. PLoS One. 2018;13(5):e0197455.

65. Castellano-Tejedor C, Moreno J, Knittle K, Nurmi J, Ginchev T, Parramón G, et al. Assessing the user experience and usability of the PRECIOUS system: a randomized controlled trial in obese patients. Inform Health Soc Care. 2020;45(4):410–27.

66. Marzuki MFM, Yaacob NA, Yaacob NM, Hassan MRA, Ahmad SB. Usable mobile app for community education on colorectal cancer: development process and usability study. JMIR Hum Factors. 2019;6(2):e12103.

67. Pereira-Azevedo N, Osório L, Fraga A, Roobol MJ. Rotterdam prostate cancer risk calculator: Development and usability testing of the mobile phone app. JMIR Cancer. 2017;3(1)

68. Heynsbergh N, Heckel L, Botti M, Livingston PM. A smartphone app to support carers of people living with cancer: A feasibility and usability study. JMIR Cancer. 2019;5(1):e11779.

69. Fu MR, Axelrod D, Guth AA, Wang Y, Scagliola J, Hiotis K, et al. Usability and feasibility of health IT interventions to enhance self-care for lymphedema symptom management in breast cancer survivors. Internet Interv. 2016;:5: 56–64.

70. Hill JR, Harrington AB, Adeoye P, Campbell NL, Holden RJ. Going remote-Demonstration and evaluation of remote technology delivery and usability assessment with older adults: Survey study. JMIR Mhealth Uhealth. 2021;9(3):e26702.

71. Bente BE, Roderick van 't Klooster JWJ, Schreijer MA, Berkemeier L, van Gend JE, Hendrik Slijkhuis PJ, et al. The Dutch COVID-19 contact tracing app (the CoronaMelder): Usability study. JMIR Form Res. 2021;5(3):e27882.

72. English LL, Dunsmuir D, Kumbakumba E, Ansermino JM, Larson CP, Lester R, et al. The pediatric risk assessment (PARA) mobile app to reduce postdischarge child mortality: Design, usability, and feasibility for health care workers in Uganda. JMIR Mhealth Uhealth. 2016;4(1):e16.

73. Bolle S, Romijn G, Smets EM, Loos EF, Kunneman M, van Weert JC. Older cancer patients' user experiences with web-based health information tools: A think-aloud study. J Med Internet Res. 2016;18(7):e208.

74. Barbabella F, Poli A, Hanson E, Andréasson F, Salzmann B, Döhner H, et al. Usage and usability of a web-based program for family caregivers of older people in three European countries: A mixed-methods evaluation. Comput Inform Nurs. 2018;36(5):232–41.

75. Agnisarman SO, Chalil Madathil K, Smith K, Ashok A, Welch B, McElligott JT. Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergon. 2017;:58: 424–34.

76. Anzinger H, Elliott SA, Hartling L. Comparative usability analysis and parental preferences of three web-based knowledge translation tools: Multimethod study. J Med Internet Res. 2020;22(3):e14562.

77. Lauritsen L, Andersen L, Olsson E, Søndergaard SR, Nørregaard LB, Løventoft PK, et al. Usability, acceptability, and adherence to an electronic self-monitoring system in patients with major depression discharged from inpatient wards. J Med Internet Res. 2017;19(4):e123.

78. Kapoor A, Nambisan P. Usability and acceptance evaluation of ACESO: A web-based breast cancer survivorship tool. J Cancer Surviv. 2018;12(3):316–25.

79. Wang T, Dolezel D. Usability of web-based personal health records: An analysis of consumers' perspectives. Perspect Health Inf Manag. 2016;13(Spring)

80. Takano A, Miyamoto Y, Kawakami N, Matsumoto T. Web-based cognitive behavioral relapse prevention program with tailored feedback for people with methamphetamine and other drug use problems: Development and usability study. JMIR Ment Health. 2016;3(1):e1.

81. Neville C, Da Costa D, Rochon M, Peschken CA, Pineau CA, Bernatsky S, et al. Development of the lupus interactive navigator as an empowering web-based eHealth tool to facilitate lupus management: Users perspectives on usability and acceptability. JMIR Res Protoc. 2016;5(2):e44.

82. Bennion MR, Hardy GE, Moore RK, Kellett S, Millings A. Usability, acceptability, and effectiveness of web-based conversational agents to facilitate problem solving in older adults: Controlled study. J Med Internet Res. 2020;22(5):e16794.

83. Tao D, Shao F, Wang H, Yan M, Qu X. Integrating usability and social cognitive theories with the technology acceptance model to understand young users' acceptance of a health information portal. Health Informatics J. 2020;26(2):1347–62.

84. García-Casal JA, Martínez-Abad F, Cid-Bartolomé T, Smith SJ, Llano-Ordóñez K, Perea-Bartolomé MV, et al. Usability study and pilot validation of a computer-based emotion recognition test for older adults with Alzheimer's disease and amnestic mild cognitive impairment. Aging Ment Health. 2019;23(3):365–75.

85. Gerłowska J, Skrobas U, Grabowska-Aleksandrowicz K, Korchut A, Szklener S, Szczęśniak-Stańczyk D, et al. Assessment of perceived attractiveness, usability, and societal impact of a multimodal robotic assistant for aging patients with memory impairments. Front Neurol. 2018;:9: 392.

86. Marien S, Legrand D, Ramdoyal R, Nsenga J, Ospina G, Ramon V, et al. A web application to involve patients in the medication reconciliation process: A user-centered usability and usefulness study. J Am Med Inform Assoc. 2018;25(11):1488–500.

87. Klaassen B, van Beijnum BJF, Held JP, Reenalda J, van Meulen FB, Veltink PH, et al. Usability evaluations of a wearable inertial sensing system and quality of movement metrics for stroke survivors by care professionals. Front Bioeng Biotechnol. 2017;:5: 20.

88. Jia Y, Wang W, Wen D, Liang L, Gao L, Lei J. Perceived user preferences and usability evaluation of mainstream wearable devices for health monitoring. PeerJ. 2018;:6: e5350.

89. Lee H, Kim J, Kim S, Kong HJ, Joo H, Lee D, et al. Usability evaluation of user requirement-based teleconsultation robots: A preliminary report from south Korea. Methods Inf Med. 2020;59(2-03):86–95.

90. Tonino RPB, Larimer K, Eissen O, Schipperus MR. Remote patient monitoring in adults receiving transfusion or infusion for hematological disorders using the VitalPatch and accelerateIQ monitoring system: Quantitative feasibility study. JMIR Hum Factors. 2019;6(4):e15103.

91. Poncette AS, Mosch L, Spies C, Schmieding M, Schiefenhövel F, Krampe H, et al. Improvements in patient monitoring in the intensive care unit: Survey study. J Med Internet Res. 2020;22(6):e19091.

92. Bonsignore L, Bloom N, Steinhauser K, Nichols R, Allen T, Twaddle M, et al. Evaluating the feasibility and acceptability of a telehealth program in a rural palliative care population: TapCloud for palliative care. J Pain Symptom Manage. 2018;56(1):7–14.

93. Lefler LL, Rhoads SJ, Harris M, Funderburg AE, Lubin SA, Martel ID, et al. Evaluating the use of mobile health technology in older adults with heart failure: Mixed-methods study. JMIR Aging. 2018;1(2):e12178.

94. Ancona GD, Murero M, Feickert S, Kaplan H, Öner A, Ortak J, et al. Implantation of an innovative intracardiac microcomputer system for web-based real-time monitoring of heart failure: Usability and patients' attitudes. JMIR Cardio. 2021;5(1):e21055.

95. Wallin E, Norlund F, Olsson EMG, Burell G, Held C, Carlsson T. Treatment activity, user satisfaction, and experienced usability of internet-based cognitive behavioral therapy for adults with depression and anxiety after a myocardial infarction: Mixed-methods study. J Med Internet Res. 2018;20(3):e87.

96. Kim Y, Seo J, An SY, Sinn DH, Hwang JH. Efficacy and safety of an mhealth app and wearable device in physical performance for patients with hepatocellular carcinoma: Development and usability study. JMIR Mhealth Uhealth. 2020;8(3):e14435.

97. Silveira DV, Marcolino MS, Machado EL, Ferreira CG, Moreira Alkmim MB, Resende ES, et al. Development and evaluation of a mobile decision support system for hypertension management in the primary care setting in Brazil: Mixed-methods field study on usability, feasibility, and utility. JMIR Mhealth Uhealth. 2019;7(3):e9869.

98. Cho S, Lee JH, Kim IK, Kim MG, Sik KY, Lee E. The educational and supportive mobile application for caregivers of dementia people. Stud Health Technol Inform. 2016;:225: 1045–6.

99. Moradian S, Krzyzanowska MK, Maguire R, Morita PP, Kukreti V, Avery J, et al. Usability evaluation of a mobile phone-based system for remote monitoring and management of chemotherapy-related side effects in cancer patients: Mixed-methods study. JMIR Cancer. 2018;4(2):e10932.

100. Janatkhah R, Tabari-Khomeiran R, Asadi-Louyeh A, Kazemnejad E. Usability of a disease management mobile application as perceived by patients with diabetes. Comput Inform Nurs. 2019;37(8):413–9.

101. Fucà E, Costanzo F, Bonutto D, Moretti A, Fini A, Ferraiuolo A, et al. Mobile-health technologies for a child neuropsychiatry service: Development and usability of the assioma digital platform. Int J Environ Res Public Health. 2021;18(5)

102. Sahu D, Rathod V, Phadnis A, Bansal SS. Telehealth for consultation and shoulder rehabilitation: A preliminary study on the perspectives of 30 patients during the COVID-19 lockdown. Clin Shoulder Elb. 2021;24(3):156–65.

103. Alessa T, Abdi T, Hawley MS, de Witte L. Mobile apps to support the self-management of hypertension: Systematic review of effectiveness, usability, and user satisfaction. JMIR Mhealth Uhealth. 2018;6(7):e10723.

104. Young HJ, Mehta T, Lai B. TEAMS (Tele-Exercise and Multiple Sclerosis), a tailored telerehabilitation mHealth app: Participant-centered development and usability study. JMIR Mhealth Uhealth. 2018;6(5):e10181.

105. Seo NJ, Arun Kumar J, Hur P, Crocher V, Motawar B, Lakshminarayanan K. Usability evaluation of low-cost virtual reality hand and arm rehabilitation games. J Rehabil Res Dev. 2016;53(3):321–34.

106. Himes KP, Donovan H, Wang S, Weaver C, Grove JR, Facco FL. Healthy beyond pregnancy, a web-based intervention to improve adherence to postpartum care: Randomized controlled feasibility trial. JMIR Hum Factors. 2017;4(4):e26.

107. Martinez W, Threatt AL, Rosenbloom ST, Wallston KL, Hickson GB, Elasy TA. A patient-facing diabetes dashboard embedded in a patient web portal: Design sprint and usability testing. JMIR Hum Factors. 2018;5(3)

108. Layfield E, Triantafillou V, Prasad A, Deng J, Shanti RM, Newman JG, et al. Telemedicine for head and neck ambulatory visits during COVID-19: Evaluating usability and patient satisfaction. Head Neck. 2020;42(7):1681–9.

109. Dubin JM, Wyant WA, Balaji NC, Ong WL, Kettache RH, Haffaf M, et al. Telemedicine usage among urologists during the COVID-19 pandemic: Cross-sectional study. J Med Internet Res. 2020;22(11):e21875.

110. Waqar-Cowles LN, Chuo J, Weiss PF, Gmuca S, LaNoue M, Burnham JM. Evaluation of pediatric rheumatology telehealth satisfaction during the COVID-19 pandemic. Pediatr Rheumatol Online J. 2021;19(1):170.

111. Moulaei K, Sheikhtaheri A, Ghafaripour Z, Bahaadinbeigy K. The development and usability assessment of an mHealth application to encourage self-care in pregnant women against COVID-19. J Healthc Eng. 2021;

112. Khodambashi S, Gilstad H, Nytro O. Usability evaluation of clinical guidelines on the web using eye-tracker. Stud Health Technol Inform. 2016;:228: 95–9.

113. Sengupta A, Beckie T, Dutta K, Dey A, Chellappan S. A mobile health intervention system for women with coronary heart disease: Usability study. JMIR Form Res. 2020;4(6)

114. Kessel KA, Me Vogel M, Kessel C, Bier H, Biedermann T, Friess H, et al. Mobile health in oncology: A patient survey about app-assisted cancer care. JMIR Mhealth Uhealth. 2017;5(6)

115. McClellan MA, Karumur RP, Vogel RI, Petzel SV, Cragg J, Chan D, et al. Designing an educational website to improve quality of supportive oncology care for women with ovarian cancer: An expert usability review and analysis. Int J Hum Comput Interact. 2016;32(4):297–307.

116. Tu MH, Chang P, Lee YL. Avoiding obsolescence in mobile health: Experiences in designing a mobile support system for complicated documentation at long-term care facilities. Comput Inform Nurs. 2018;36(10):501–6.

117. Talboom-Kamp E, Tossaint-Schoenmakers R, Goedhart A, Versluis A, Kasteleyn M. Patients' attitudes toward an online patient portal for communicating laboratory test results: Real-world study using the eHealth impact questionnaire. JMIR Form Res. 2020;4(3):e17060.

118. World Health Organization. mHealth: New horizons for health through mobile technologies [Internet].

119. Schnall R, Cho H, Liu J. Health information technology usability evaluation scale (Health-ITUES) for usability assessment of mobile health technology: Validation study. JMIR Mhealth Uhealth. 2018;6(1)

120. Jeffrey B, Bagala M, Creighton A, Leavey T, Nicholls S, Wood C, et al. Mobile phone applications and their use in the self-management of type 2 diabetes mellitus: A qualitative study among app users and non-app users. Diabetol Metab Syndr. 2019;:11: 84. PMID: 31636719 DOI: 10.1186/s13098.

121. Soltanzadeh L, Taheri A, Rabiee M. Implementing a prototype diabetic telemedicine system. International Journal of Computer Science Issues. 2014;11(4):86–91.

122. Soltanzadeh L, Taheri A, Rabiee M. A portable ECG device for diabetic patients's telemedicine. Frontiers in Health Informatics. 2014;3(1):9–11.

This display is generated from Gostaresh Afzar Hamara JATS XML.

Frontiers in Health Informatics

ISSUES

Views