• Logo
  • HamaraJournals
143

Views

Usability Evaluation of Hospital Information System According to Heuristic Evaluation

, , , , , , , and

Abstract

Introduction:

There are various applications and health information systems which have been developed to promote the effective retrieval of patient information, statistics, research, and education. Therefore, there is a need to design them in consistency with scientific principles of usability. To this end, the usability of hospital information sub-systems employed at the hospitals of Ahvaz were compared using heuristic evaluation method. The objective of the study was to assess the usability of hospital information system according to heuristic evaluation.

Material and Methods:

Six trained evaluators, independently determined the ADT subsystem, HIM subsystem, and NIS according to Nielsen’s 10 Heuristic Principles. Since more than half of the hospitals (about 54%) employed Sib application, no specific sampling method was used. After combining the usability problems, the average severity ratings of the problems were calculated, and then the subsystems were compared.

Results:

The number of the usability problems of the ADT information subsystem, HIM subsystem, and NIS were 40, 39, and 37, respectively. After merging the problems, the features of “user control and freedom” with 20 cases and “flexibility and efficiency of use” with six cases had the highest and the lowest inconsistencies with usability principles. The average severity ratings of the problems also varied between 1.7 and 3.

Conclusion:

Heuristic evaluation method is regarded as one of the approaches appropriate to identify usability problems in health information systems. Thus, it is advisable to utilize this method to modify the design of the systems and to improve their efficiency before their implementation in order to increase user satisfaction.

INTRODUCTION

In the field of medical sciences, various software applications and health information systems have been developed to prevent or reduce medical errors, facilitate and accelerate procedures, prevent redundancy, reduce costs, and improve patient safety [1, 2]. The hospital information system (HIS) is one of the most commonly used systems in Iran [3]. Using this system improves the efficiency and quality of care, increases satisfaction, and provides better services [4]. This system includes subsystems including para-clinical, patient management, pharmacy, accounting, storage, statistics, staff, clinic, medical records, medical equipment, social work, insurance, and merit pay [5].

This system should be designed and used appropriately and accurately, and in accordance with the scientific principles of usability [6]. Usability is the ease of using a system and the extent to which a particular product can be used to achieve specific goals in a given context with effectiveness, efficiency, and satisfaction [7].

The evaluation methods for the usability of systems are one of the methods used to ensure the proper design of software applications and health information systems. Usability deals with various software features including ease of learning, efficiency, memorization, error prevention, and user satisfaction [8]. Usability testing methods and usability inspection methods are two major groups of application evaluation methods [8-10]. Examples of evaluations with end-user participation include think aloud. Exploratory evaluation and cognitive walkthrough are two examples of Usability Inspection [11, 12]. Exploratory evaluation, which is also called heuristic evaluation, evaluates the degree of compliance of the system with the standard principles of designing the user interface. This method is one of the simplest and fastest methods that identifies usability problems by spending less time and resources [13, 14]. Introduced by Nielsen for the first time, this method has ten principles that should be taken into account in the design of user interface. After identifying system problems, it is possible to determine their severity and, eventually, their subsequent consequences [8-10, 13-17].

This method is one of the efficient and cost-effective methods for assessing the clinical information system [18]. In this method, evaluators consider each case of violation of Nielsen's principles as a usability problem. Problems that could interfere with the user's interaction with the system's user interface are known as violations. If these violations do not resolve, they will waste time, increase error rate, reduce the quality of information, and thus threaten the health of patients [14]. According to Thyvalikakath and Joshi, in order to identify the majority of problems of usability, three to five evaluators are required. This method may not identify all the usability problems of a system [9, 19].

Thyvalikakath et al. identified 50% of usability problems [9]. Also, in their study on identifying the problems of usability of the emergency information system, Khajouei et al. found that most of the problems had a relatively high degree of severity. Thus, ignoring them is an obstacle to the realization of the goals of health information systems [20]. Similarly, Nabovati et al. reported that 67% of the problems had a relatively high degree of severity. Compliance with the principles and standards of design is recommended in the early phases of system development [21].

No study has yet been conducted to identify the usability problems of HIS in Ahvaz Jundishapur University of Medical Sciences (AJUMS). Considering the importance of HIS, in this study, we evaluated the admission, discharge, transfer (ADT), health information management (HIM), and the nursing information system (NIS) subsystems used in one of the 28 hospitals which is currently using Sib software (specific and commercial name for HIS developed by Gostar-Sanam company) as an example of HIS, in order to usability evaluation of hospital information subsystems in AJUMS hospitals using heuristic evaluation method.

MATERIAL AND METHODS

This study was carried out on the HIS of 28 hospitals affiliated with AJUMS in 2017. All hospitals were governmental, of which 23 were non-teaching hospitals and 5 were teaching hospitals. Since more than half of hospitals in Khuzestan province (about 54%) use Sib system, in this study, no sampling was done. The study was focused on the Sib system installed in the information technology laboratory at the School of Allied Medical Sciences of AJUMS. With the assistance of an expert on usability, six evaluators assessed the ADT, HIM and the NIS subsystems. All evaluators had bachelor degree in health information technology and were trained by attending four theoretical sessions plus a practical one offered as a workshop as well as at least five clarification sessions (at least two hours each) on heuristic evaluation. The evaluators were eligible for the study if they passed these sessions successfully. In this study, Nielsen's ten principles were used.

Data collection was done through a standard usability checklist whose validity had been previously verified. This checklist has ten principles and each principle has unique items. The evaluators independently recorded the system features that did not conform to the principles and determined the severity of the problems on the basis of a five-point scale (0-4):

Absence of a problem (No problem)

There is no need for fixing unless there is an extra time in the project (Cosmetic)

Fixing should be given low priority (Minor)

Important to fix, so it should be given high priority (Major)

Imperative to fix before the product is released (Catastrophe)

The evaluators calculated the severity of the problems of usability based on a combination of three factors (Table 1).

Table 1

The factors affecting the severity of usability problem

1 The problem frequency: Is the problem common or rare?
2 The problem impact: Is it difficult or easy for the end users to take over the problem?
3 The problem persistence: Does it trouble the end users repeatedly or is it a one-time problem?

The score of each principle was calculated based on the mean scores of the items in that principle.

After identifying problems and providing screenshots of them, these problems were combined by evaluators and after removing duplicate items, they were organized in a single list. The data recorded in Excel. Data were analyzed through descriptive analysis including frequency, relative frequency, and mean.

Results

The ADT, HIM, and NIS subsystems were evaluated by six evaluators (Table 2) using the heuristic evaluation principles.

Table 2

General characteristics of the evaluators attended in the usability evaluation (n=8)

Evaluators Number experience
(Years)
Age range
(Years)
Gender Field of study Degree Academic rank
Supervisor 1 1 5 30-40 M Medical Informatics PhD Assistant professor
Supervisor 2 1 10 50-60 M Health Information Management MSc Lecturer
Students 6 - 20-30 F Health Information Technology BSc Student

Since no Help feature was included in these subsystems, the findings of this study were analyzed on nine principles. In total, 474 usability violations were identified by the evaluators 63, 68, 101, 58, 100 and 84, respectively. Therefore, after merging the total number of shared cases, the total number was reduced to 116, of which 40 cases (34%) were related to the ADT subsystem, 39 cases (34%) to the HIM subsystem and 37 cases (32%) to the NIS subsystem.

The highest violations of usability evaluation principles in the ADT subsystem were related to the feature "Recognition rather than recall" with seven cases (18%) whereas the lowest were associated with "Flexibility and efficiency of use" with two cases (5%). The average severity of problems in this subsystem was 2.3 (minor problem). "User control and freedom" with a severity of 2.9(major problem) had the greatest severity in this subsystem. Also, more than half of the violations (21 cases) were of minor problem severity while nine (23%) had a severity of 2.5 or mediocre (Table 3).

In the HIM subsystem, the features of "User control and freedom" with seven cases (18%) and "Flexibility and efficiency of use" with two cases(5%) had the highest and lowest number of violations, respectively. In this subsystem, over 30% of the problems were major. The average severity varied from 2.9(major problem) relating to "User control and freedom" to 1.1(minor problem) relating to the "Match between system and the real world" (Table 3).

In the NIS subsystem, the features of "User control and freedom" and "Flexibility and efficiency of use" had the highest and lowest inconsistencies with the principles of usability: seven cases (19%) and two cases (5%), respectively. The average degree of severity was 2.3 (minor problem), varying between 1.5 and 3. The features of "User control and freedom", "Error prevention" and "Recognition rather than recall" were major problems (Table 3).

Overall, among the AJUMS HIS subsystems examined, the features of "User control and freedom" and "Flexibility and efficiency" had the highest and lowest inconsistencies with the principles of usability: 20 cases (17%) and 6 cases (5%), respectively. More than half (56%) of the HIS problems were related to "User control and freedom" (17%), "Consistency and standards" (13%), "Help users recognize, diagnose, and recover from errors" (13%), "Recognition rather than recall" (13%).

Table 3

HIS usability problems per evaluators, average, and severity of errors

No Principles/ Heuristics Sub-system Six Evaluators Five Evaluators Four Evaluators Three Evaluators Two Evaluators One Evaluator Total Average severity Severity status
1 Visibility of system status ADT 1 - - - 1 1 3 2.1 Minor
HIM 2 - - 2 - - 4 2.3 Minor
NIS 1 - 2 - 1 - 4 2 Minor
HIS 4 - 2 2 2 1 11 2.2 Minor
2 Match between system and the real world ADT 1 1 - 1 - 1 4 1.9 Minor
HIM 1 1 1 1 - - 4 1.1 Minor
NIS 1 - 2 - - 1 4 2 Minor
HIS 3 2 3 2 - 2 12 1.7 Minor
3 User control and freedom ADT 2 2 - - 2 - 6 2.9 Major
HIM 2 2 - 1 2 - 7 2.9 Major
NIS 3 1 1 - 2 - 7 3 Major
HIS 7 5 1 1 6 - 20 3 Major
4 Consistency and standards ADT 2 - - 1 2 - 5 1.9 Minor
HIM 2 - - 1 2 - 5 1.9 Minor
NIS 3 - 1 1 - - 5 1.5 Minor
HIS 7 - 1 3 4 - 15 1.8 Minor
5 Help users recognize , diagnose, and recover from errors ADT 2 - - - 3 - 5 1.8 Minor
HIM 2 - - - 3 - 5 1.9 Minor
NIS 2 - - 1 2 - 5 2 Minor
HIS 6 - - 1 8 - 15 1.9 Minor
6 Error prevention ADT 1 - - 2 1 - 4 2.7 Major
HIM 1 - - 2 - - 3 2.6 Major
NIS 2 1 - 1 - - 4 2.6 Major
HIS 4 1 - 5 1 - 11 2.7 Major
7 Recognition rather than recall ADT 1 2 2 1 1 - 7 2.5 Mid
HIM 1 - 1 2 1 - 5 2.1 Minor
NIS 1 - 1 1 - - 3 3 Major
HIS 3 2 4 4 2 - 15 2.6 Major
8 Flexibility and efficiency of use ADT 2 - - - - - 2 2.5 Mid
HIM 2 - - - - - 2 2.6 Major
NIS 1 - 1 - - - 2 2.5 Mid
HIS 5 - 1 - - - 6 2.6 Major
9 Aesthetic and minimalist design ADT - 2 1 - 1 - 4 2.1 Minor
HIM - 1 1 1 1 - 4 2.5 Mid
NIS - 1 - - 2 - 3 1.9 Minor
HIS - 4 2 1 4 - 11 2.2 Minor
10 Help and documentation ADT - - - - - - - - -
HIM - - - - - - - - -
NIS - - - - - - - - -
HIS - - - - - - - - -
Total ADT 12 7 3 5 11 2 40 2.3 Minor
HIM 13 4 3 10 9 - 39 2.2 Minor
NIS 14 3 8 4 7 1 37 2.3 Minor
HIS 39 14 14 19 27 3 116 2.3 Minor

Discussion

Similar to the findings of the study, Khajouei et al. [20], the feature "Help and documentation", was not defined in HIS. This feature, which can be implemented offline and online, can be useful in solving problems that users may encounter while working with the system, can play an effective role in training users, eliminates confusion and saves the user's time by removing the need to refer to or call the system administrator.

The features "User control and freedom", "Consistency and standards", "Help users recognize, diagnose, and recover from errors" and "Recognition rather than recall" were the most common cases of inconsistencies in HIS. This problem can cause user exhaustion, and failure in successful interaction with the system and ultimately their resistance to using other information systems.

The following modifications are deemed necessary: quick access to application settings, switching between overlap windows, defining undo and canceling out features, using common standards, and observing consistency throughout the system, displaying the severity of errors, and displaying pieces of information in a way users can understand the purpose of their presentation and content. This reduces the need for users to invest intellectual and physical activity to perform actions, and to adopt a trial-and-error strategy to achieve goals, and identify possible errors and avoid them.

Thirty-one percent of the problems identified in this study were major. Therefore, their ignorance is an obstacle to the realization of the goals of the health information system. In other study, the degree of severity of most of the identified problems is reported relatively high [20, 21]. For example, Atashi et al. identified 99 applicability problems in their research, with an average degree of severity ranging from 2.4 to 3.3 [22]. The severity of more than 50% of the problems was major, which is consistent with the study by Chan et al. [13] According to the literature, information systems that have low usability can reduce the accuracy and speed, and can bring about user confusion, anger and dissatisfaction [23, 24].

One of the strengths of this study is that although the severity of the problems is determined manually and on the basis of the three components proposed by Nielsen, considering how this is calculated (i.e., the average of the problems encountered by independent evaluators) the final severity of each problem is of high accuracy. Therefore, system designers can use this information to prioritize the changes required in user interface design. What makes this method superior to others is the fact that it is cheap and easy to use in comparison with other methods, and evaluators can be easily trained on using it for identifying a large number of problems.

Conclusion

The findings of the current study show that heuristic evaluation is one of the versatile methods for identifying problems in health information systems. Hence, it could be regarded as an efficient, fast, simple, and cost-effective method for identifying usability problems.

According to the findings of this study, not abiding by some of the principles of usability, such as the "User control and freedom" feature, most likely leads to confusion, dissatisfaction, reduced speed, increased errors, and loss of information quality. Based on the findings of the current study, it can be concluded that heuristic evaluation of usability has more strengths than weaknesses. Therefore, if this method is adopted parallel to other methods, its efficiency will be maximized. Given that heuristic evaluation can be done in a short time and the list of problems is immediately available, this method will be very suitable for evaluation when there is a shortage of time for system development.

Usability evaluation can identify the cause of problems, which may bring about new errors, users' fear of working with the system, user resistance against it, and harm to the patient [25].

According to the findings, researchers have presented suggestions for solving the problems. First and most importantly, the user's offline and online help and support need to be defined in the system. Also, the most important and essential points for working with the system should be provided as a document in the system.

Second, for "Visibility of system status" feature, it is recommended that a clear cue, to inform the user of system activity at the time of system processing should be defined in the system. Also, the selected field or item need to be made more transparent compared to other fields.

Third, for the "Match between system and real world", it is recommended that logical and alphanumeric sequences should be followed in the menu options. Also, redundant fields should be removed.

Then, the most important recommendation for the "User control and freedom" is the Undo option need to be defined in a way that ensures its quick access.

Fifth, recommendation for the "Consistency and standards" includes Persian font should be used. Also, soft tones for positive feedback and harsh tones for negative feedback should be used.

Sixth, the most important recommendation regarding "Help users recognize, diagnose, and recover from errors" is severity of errors should be determined and solutions need to be proposed.

Seventh, in case of "Error prevention", it is recommended that next to each field, the number of authorized characters should be specified. Also, entering unauthorized characters in the letter fields should be avoided.

Last but not least, In case of "Aesthetic and minimalist design", it is recommended that error messages should be expressed positively.

Study limitations

1. The study was conducted on only three subsystems of HIS, and the findings are based on the evaluation of these subsystems, so the evaluation of other HIS subsystems could show more usability problems.

2. This system is used in hospitals affiliated with AJUMS. Therefore, the results cannot be fully extended to other similar systems in hospitals in the province which are produced by different vendors. Of course, many of the systems installed there have the same design and usability problems.

ACKNOWLEDGEMENT

The researchers are grateful to the Deputy Director for Research and Technology Development of AJUMS.

AUTHOR’S CONTRIBUTION

The authors agree on this final form of the manuscript, and attested that all authors contributed in the final draft of the manuscript. 

CONFLICTS OF INTEREST

The authors declare no conflicts of interest regarding the publication of this study.

FINANCIAL DISCLOSURE

This article is the result of a research project approved by the Deputy Director for Research and Technology Development of AJUMS (approval number: 33011492; Code of Ethics Committee: IRAJUMS.REC.1396.1119).

References

1. Menachemi N, Collum TH. Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy. 2011;:4: 47–55.
2. Yucel G, Cebi S, Hoege B, Ozok AF. A fuzzy risk assessment model for hospital information system implementation. Expert Systems with Applications. 2012;39(1):1211–8.
3. Saghaeiannejad IS, Saeedbakhsh S, Jahanbakhsh M, Habibi M. Assessment and comparison of hospital information systems in isfahan hospitals based on the adjusted delone and mclean model. Health Information Management. 2011;8(5)
4. Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc. 2001;8(4):299–308.
5. Azizi A, Pirnejad H. Using health information technology to improve workflow in the emergency department. Mashhad University of Medical Sciences: Graduate Students Research Festival 2013;
6. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005;74(7-8):519–26.
7. International Organization for Standardization. ISO 9241-11 ergonomic requirement for office work with visual display terminals (VDTs) _ Part 11: Guidance on usability. Geneva:ISO 1998;
8. Nielsen J. Usability engineering. Academic Press 1993;
9. Thyvalikakath TP, Monaco V, Thambuganipalle H, Schleyer T. Comparative study of heuristic evaluation and usability testing methods. Stud Health Technol Inform. 2009;:143: 322–7.
10. Yen PY, Bakken S. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol: An example from a web-based communication tool for nurse scheduling. AMIA Annu Symp Proc. 2009;:2009: 714–8.
11. Poll R. Evaluating the library website: Statistics and quality measures. World Library and Information Congress: 73rd IFLA General Conference and Council. IFLA 2007;
12. Scholtz J. Usability evaluation: National institute of standards and technology [Internet]. 2004 [cited: 1 Dec 2020].
13. Chan AJ, Islam MK, Rosewall T, Jaffray DA, Easty AC, Cafazzo JA. Applying usability heuristics to radiotherapy systems. Radiother Oncol. 2012;102(1):142–7.
14. Choi J, Bakken S. Web-based education for low-literate parents in neonatal intensive care unit: Development of a website and heuristic evaluation and usability testing. Int J Med Inform. 2010;79(8):565–75.
15. Graham MJ, Kubose TK, Jordan D, Zhang J, Johnson TR, Patel VL. Heuristic evaluation of infusion pumps: Implications for patient safety in intensive care units. Int J Med Inform. 2004;73(11-12):771–9.
16. Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res. 2009;11(4):e47.
17. Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 2003;36(1-2):23–30.
18. Beuscart-Zephir MC, Watbled L, Carpentier AM, Degroisse M, Alao O. A rapid usability assessment methodology to support the choice of clinical information systems: A case study. Proc AMIA Symp. :2002; 1: 46–50.
19. Joshi A, Lichenstein R, King J, Arora M, Khan S. Evaluation of a computer-based patient education and motivation tool on knowledge, attitudes and practice towards influenza vaccination. International Electronic Journal of Health Education. 2009;:12: 1–15.
20. Khajouei R, Azizi A, Atashi A. Usability evaluation of an emergency information system: A heuristic evaluation. Journal of Health Administration. 2013;16(52):61–72.
21. Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of laboratory and radiology information systems integrated into a hospital information system. J Med Syst. 2014;38(4):35. PMID: 24682671 DOI: 10.1007/s10916.
22. Atashi A, Khajouei R, Azizi A, Dadashi A. User interface problems of a nationwide inpatient information system: A heuristic evaluation. Appl Clin Inform. 2016;7(1):89–100.
23. Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Stud Health Technol Inform. 2009;:150: 532–6.
24. Khajouei R, Jaspers MW. The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: A systematic review. Methods Inf Med. 2010;49(1):3–19.
25. Agharezaei Z, Khajouei R, Ahmadian L, Agharezaei L. Usability evaluation of a laboratory information system. Health Information Management. 2013;10(2):213–24.

This display is generated from Gostaresh Afzar Hamara JATS XML.

Refbacks

  • There are currently no refbacks.