Email this article 
Health Information Systems Evaluation Criteria: Overview of Systematic Reviews
Views
Abstract
Introduction: Health information systems play an important role in improving the quality of patient care and patient safety. to ensure their effectiveness and efficiency, they need to be evaluated. Although HIS evaluation has been investigated in many studies, there is no consensus on which aspects of HIS to evaluate. The aim of this study is to identify the indicators for the evaluation of health information systems and to provide an overview of the criteria devised and studies conducted.
Methods: An umbrella review was performed exploring databases PubMed, Science direct, Web of Science, Science, and IEEE while following the PRISMA protocol. Articles were reviewed by two authors independently using the covidence tool to check the inclusion criteria and to extract the data items. Risk of bias was assessed using ROBIS and AMSTAR.
Result: All included studies showed a high risk of bias according to ROBIS criteria. The extracted evaluation criteria were classified into 13 categories. Most of the studies believe that a more reliable and standardized tool is needed for the evaluation of health information systems. Two studies mentioned that surveys and questionnaires were the most commonly used method for evaluation of the systems. Summative evaluation was the most used method in two studies and the least used method in another one study.
Conclusion: All the included studies had high risk of bias. Accordingly, further research and evidence is needed in this field. Most of the studies highlighted the need for more reliable and standardized tools for evaluation of health information systems.
Keywords
References
Lippeveld T, Sauerborn R, Bodart C. Design and implementation of health information systems. World Health Organization. 2000.
Haux R. Health information systems: Past, present, future. Int J Med Inform. 2006; 75(3-4): 268-81. PMID: 16169771 DOI: 10.1016/j.ijmedinf.2005.08.002
Azubuike MC, Ehiri JE. Health information systems in developing countries: Benefits, problems, and prospects. J R Soc Promot Health. 1999; 119(3): 180-4. PMID: 10518358 DOI: 10.1177/146642409911900309
Ahmadian L, Salehi Nejad S, Khajouei R. Evaluation methods used on health information systems (HISs) in Iran and the effects of HISs on Iranian healthcare: A systematic review. Int J Med Inform. 2015; 84(6): 444-53. PMID: 25746766 DOI: 10.1016/j.ijmedinf.2015.02.002
Shahmoradi L, Habibi-Koolaee M. Integration of health information systems to promote health. Iran J Public Health. 2016; 45(8): 1096-7. PMID: 27928542
Yusof MM, Paul RJ, Stergioulas LK. Towards a framework for health information systems evaluation. International Conference on System Sciences. IEEE ; 2006.
Ahmadi M, Shahmoradi L, Barabadi M, Hoseini A. A survey of usability of hospital information systems from the perspective of nurses, department secretaries, and paraclinic users in selected hospitals: 2009. Journal of Health Administration. 2011; 14(44): 11-20.
Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J, et al. Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform. 2004; 73(6): 479-91. PMID: 15171977 DOI: 10.1016/j.ijmedinf.2004.04.004
Ammenwerth E, Shaw N. Bad health informatics can kill–is evaluation the answer? Methods Inf Med. 2005; 44(1): 1-3. PMID: 15778787
Shahmoradi L, Ahmadi M, Haghani H. Determining the most important evaluation indicators of healthcare information systems (HCIS) in Iran. Health Inf Manag. 2007; 36(1): 13-22. PMID: 18195393 DOI: 10.1177/183335830703600103
Ammenwerth E, Gräber S, Herrmann G, Bürkle T, König J. Evaluation of health information systems—problems and challenges. Int J Med Inform. 2003; 71(2-3): 125-35. PMID: 14519405 DOI: 10.1016/s1386-5056(03)00131-x
Shahmoradi L, Ahmadi M, Haghani H. Defining evaluation indicators of health information systems and a model presentation. Journal of Health Administration. 2007; 10(28): 15-24.
Andargoli AE, Scheepers H, Rajendran D, Sohal A. Health information systems evaluation frameworks: A systematic review. Int J Med Inform. 2017; 97: 195-209. PMID: 27919378 DOI: 10.1016/j.ijmedinf.2016.10.008
Odhiambo-Otieno GW. Evaluation criteria for district health management information systems: Lessons from the ministry of health, Kenya. Int J Med Inform. 2005; 74(1): 31-8. PMID: 15626634 DOI: 10.1016/j.ijmedinf.2004.09.003
Palvia P, Jacks T, Brown W. Critical issues in EHR implementation: Provider and vendor perspectives. Communications of the Association for Information Systems. 2015; 36(1): 36.
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Ann Intern Med. 2009; 151(4): 264-9. PMID: 19622511 DOI: 10.7326/0003-4819-151-4-200908180-00135
Babineau J. Product review: Covidence (systematic review software). Journal of the Canadian Health Libraries Association. 2014; 35(2): 68-71.
Whiting P, Savović J, Higgins JP, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016; 69: 225-34. PMID: 26092286 DOI: 10.1016/j.jclinepi.2015.06.005
Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007; 7:10. PMID: 17302989 DOI: 10.1186/1471-2288-7-10
Yusif S, Hafeez-Baig A, Soar J. e-Health readiness assessment factors and measuring tools: A systematic review. Int J Med Inform. 2017; 107: 56-64. PMID: 29029692 DOI: 10.1016/j.ijmedinf.2017.08.006
Ellsworth MA, Dziadzko M, O'Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. J Am Med Inform Assoc. 2017; 24(1): 218-26. PMID: 27107451 DOI: 10.1093/jamia/ocw046
Genitsaridi I, Kondylakis H, Koumakis L, Marias K, Tsiknakis M. Evaluation of personal health record systems through the lenses of EC research projects. Comput Biol Med. 2015; 59: 175-85. PMID: 24315661 DOI: 10.1016/j.compbiomed.2013.11.004
Mohammadzadeh Z, Ghazisaeedi M, Nahvijou A, Kalhori SRN, Davoodi S, Zendehdel K. Systematic review of hospital based cancer registries (HBCRs): Necessary tool to improve quality of care in cancer patients. Asian Pac J Cancer Prev. 2017; 18(8): 2027-33. PMID: 28843217 DOI: 10.22034/APJCP.2017.18.8.2027
Sousa VE, Lopez KD. Towards usable e-health: A systematic review of usability questionnaires. Appl Clin Inform. 2017; 8(2): 470-90. PMID: 28487932 DOI: 10.4338/ACI-2016-10-R-0170
Lee Y, Jung MY, Shin GW, Bahn S, Park T, Cho I, et al. Safety and usability guidelines of clinical information systems integrating clinical workflow: A systematic review. Healthc Inform Res. 2018; 24(3): 157-69. PMID: 30109149 DOI: 10.4258/hir.2018.24.3.157
Mohammed SA, Yusof MM. Towards an evaluation framework for information quality management (IQM) practices for health information systems: Evaluation criteria for effective IQM practices. J Eval Clin Pract. 2013; 19(2): 379-87. PMID: 22502634 DOI: 10.1111/j.1365-2753.2012.01839.x
Lau F, Kuziemsky C, Price M, Gardner J. A review on systematic reviews of health information system studies. J Am Med Inform Assoc. 2010; 17(6): 637-45. PMID: 20962125 DOI: 10.1136/jamia.2010.004838
Shafiei Nikabadi M, Naghipour N. A model for assessing hospital information systems. Journal of Health Administration. 2015; 18(60): 50-66.
Chen H, Hailey D, Wang N, Yu P. A review of data quality assessment methods for public health information systems. Int J Environ Res Public Health. 2014; 11(5): 5170-207. PMID: 24830450 DOI: 10.3390/ijerph110505170
Sligo J, Gauld R, Roberts V, Villa L. A literature review for large-scale health information system project planning, implementation and evaluation. Int J Med Inform. 2017; 97: 86-97. PMID: 27919399 DOI: 10.1016/j.ijmedinf.2016.09.007
DOI: https://doi.org/10.30699/fhi.v11i1.376
Refbacks
- There are currently no refbacks.
Frontiers in Health Informatics (Official Journal of Iranian Association of Medical Informatics) ISSN-Online: 2676-7104
No. 4, Rashid Yasami Av., Vali-e-Asr St., Tehran-Iran; SMS: (+98) 9383621232 email: info@ijmi.ir
This work is licensed under a Creative Commons Attribution 4.0 International License which allows users to read, copy, distribute and make derivative works from the material, as long as the author of the original work is cited properly.
The theme is developed by: Gostaresh Afzar Hamara