Designing and Evaluating of a Guideline-Based Recommender-Interpreter System to Inform Diabetic Patients about the Status of Periodic Lab Tests
One of the most important issues in managing diabetes is the periodic checkups and tests to prevent the secondary complications of the disease. Low level of literacy in patients with diabetes, and the widespread use of abbreviations and numbers in the lab test results, makes it difficult for the patient to understand and interpret her health status. The purpose of this study is to design an expert system based on clinical guidelines in order to interpret the laboratory test results to patients and provide relevant recommendations in a textual report.
Material and Methods:
The study consists of two phases: the design and the evaluation. Design phase consists of 4 stages. In the first step, based on a Delphi study, the biological and laboratory tests, periodically measured for diabetic patients, were identified. In the second phase, according to the American Diabetes Association guideline, the rules for the interpretation of tests were extracted. In the third stage, an observational study was conducted to identify the elements of explanations that were provided by the physician about the results of patients' tests. In the fourth stage, the template messages were designed. In the evaluation phase, 12 diabetic patients assessed the usability of the generated report in two aspects of the visual design and the content. Five indices of apparent attractiveness, ease of comprehension, applicability, description adequacy, and novelty of content was evaluated with a 5-point Likert scale checklist.
The results of the Delphi study revealed that routine tests for diabetic patients included three profiles (e.g. blood glucose, blood lipids, and kidney status), with two examinations (e.g. blood pressure and weight). The structure of the report was designed according to the patient physician communication at visit sessions. Each section of the report includes three types of feedback: descriptive, comparative, and conclusive statements. The average age of participants was 56.4 years with 72.1% women. Patients believed that the report was attractive with an average score of 9.3, and evaluated the report's comprehensiveness with an average score of 9.4. The usability (8.3), the information adequacy (8.7) and the novelty (8.2) were also perceived acceptable by patients.
The results showed that the report was acceptable from the perspective of diabetic patients, and patients would like to get more information about their health status. The findings of this study can be used as guidance to design of the next phase of the study, e.g. evaluation of intervention effectiveness
Diabetes mellitus is a chronic metabolic disease with a rising prevalence in Iran and in the world, requiring continuous care throughout the patient's life. Failure to control diabetes can lead to serious complications such as cardiovascular disease, blindness, renal failure, and lower limb amputation. Diabetes is the fifth cause of death in the world, and 8.3 million people die each year from the disease or its complications .
Diabetes also imposes costly economic burdens on society and the health system. In the United States, in 2003, average health care costs for each diabetic person were 6 times higher than for non-diabetic patients. In our country, 10 billion Rials a year (direct medical expenses) are allocated to this disease . Therefore, prevention and control of diabetes has been considered.
In the management of chronic diseases, such as diabetes, the patient has a key role in the treatment process. Contrary to the fact that patients were not allowed access to medical information, modern treatment systems require patients to seek information to manage their illness. Having enough knowledge of the disease is a necessary foundation for the effective participation of diabetic patients in self-care activities .
Several studies have investigated the effect of patient's knowledge on his health status and the extent he participates in therapeutic decision making. The results of these studies have identified health literacy as an important factor in improving self-management behaviors. In this regard, the study by Wiemann found that the main reason for patients' non-compliance with self-care activities was unintentional issues such as lack of knowledge and practical barriers . The patient's knowledge and beliefs about the process of illness, treatment, self-care needs, risk factors, etc., affects their adherence to treatment, and reduces the outcomes of the disease, such as mortality and disability .
Patients often follow the recommendations that they were already informed about their consequences. Despite the importance of knowledge and awareness among diabetic patients in managing self-care activities, various studies have reported a low level of knowledge in these patients [6-9]. The results of a study about the level of knowledge of diabetic patients showed more than half of the participants (76%) had low-to-moderate knowledge about self-care .
According to Braun Only 35% of diabetics are advised about their medications, which are mostly presented verbally . Heisler Concluded in a study that despite the patient's highly acceptance the recommendations that are provided in routine visit sessions, restrictions such as the doctor's limited time, lack of access to the doctor, the use of jargons, and patient's forgetfulness caused this approach to be inadequate to educate patients .
Considering the key role of diabetic patient in the treatment process and the importance of his knowledge in managing self-care activities, any method that improves the quality of information communication will be valuable. Various studies have shown that most people want their recommendations to be printed on paper so that they can carry it home and read it in a stress-free environment appropriate to their learning speed and ability .
Another issue that increases the effectiveness of education is to tailor information in accordance with the patient's needs and health condition [14, 15]. When people find the information relevant to their needs and health condition, they read it more carefully, and are more likely to change behavior. Based on the Elaboration Likelihood model, when individuals perceive information to be personally relevant, they process it more thoroughly .
As a result, personalization can play an important role in persuading people to change behavior.An important part of the management of diabetes is the regular checkups and periodic tests to monitor the effectiveness of treatment and timely prevention of secondary complications of the disease. Awareness of the patient about the status and trends of their lab tests can help them improve their participation in self-care activities. The low level of patient literacy on the one hand, and the widespread use of abbreviations and numbers in the lab test results, on the other hand, made it difficult for the patient to understand and interpret the results of these tests.
The purpose of this study is to design a recommender-interpreter system based on a clinical guideline to inform diabetic patients about the status of their routine lab tests.
Material and methods
The study consists of two phases: the design and the evaluation. Design phase consists of 4 stages. In the first stage, lab tests that are commonly requested for diabetic patients are identified. A preliminary list of tests was formed from the literature and the available medical records in five diabetes clinics. Then, in a Delphi study, five diabetic physicians (two endocrinologists and three general practitioners) decided on the necessity of each test. Items selected by two thirds of experts were considered for inclusion in the study.
In the second phase, according to the American Diabetes Association guideline, the rules for the interpretation of tests were extracted and confirmed by experts. The set of rules for determining the patient's condition for each test was made in the conditional expressions "if-then" based on the reference values specified in the guideline. In this way, the possible values for the patient’s condition are: desirable and undesirable.
Comparing the patient's lab test status with previous session, reveals the changes over time as improved, worsened, and no change. Each cell in Table 1 represents a conditional expression that is enabled if the specified states in the row and column are enabled.
Possible states of change in lab tests results
In the third stage, the structure of the report was determined from a field observation study. Doctor's explanation about the test results was noted and analyzed. This study was conducted in the period from March 96 to January 1997, with the participation of three diabetes specialists. During this period, the researcher observed an average of 50 hours of information exchanged between patients and the physician in visit sessions for 102 patients. The researcher did not attend the patient's visit, in case of his dissatisfaction. During the meetings, the researcher only observed and recorded the data without interfering in the physician-patient communication process. Three diabetes doctors (an endocrinologist and two general practitioners) participated in the study. During the session, the patient's questions and the physician's response were explicitly recorded by the researcher and then coded using content analysis. The result of this phase is the identification and extraction of topics that can be presented in the report.
In the fourth phase, a library of templates and text components was designed. Library messages are tailored to the topics identified in the previous step. Messages include all of the situations that a patient may have regarding his lab tests. The length of each message can be up to three sentences and can be identified with a unique code. Library messages are divided into two main categories: feedback messages and knowledge provision. The method of collecting, validating and organizing the message set is explained below.
The purpose of the feedback messages is to inform the patient about his health and performance over time. These messages are formed comparing the current status with a reference range. In this study, two common types of feedbacks, are used: descriptive and comparative. In the descriptive feedback, the basis of comparison is reference values from the clinical practice guidelines, and in comparative feedback, the comparison basis is the previous status. In terms of structure, feedback messages may be designed as a template or full text.
In this research, the design of the feedback messages was developed based on the principles introduced in a Guideline for designing health messages  that provide effective principles and factors in designing and formatting an effective message, such as the writing style, the choice of vocabulary, and the proper tone . To design the feedback messages similar to the way that doctors provide information, notes collected from the observation study were analyzed in terms of the length of the message, the format, and the tone of the statement.
Knowledge raising messages include proven facts and self-care recommendations that are presented in the framework of the issues identified in step 3 of the study. These messages were extracted in a two-step approach, from valid patient education resources. In this process, the information was extracted in the form of text snippets, and each piece was included in the list of messages related to one of the topics.
In the final stage, according to the patient's lab tests status, relevant messages are selected from the library and are provided to the patient in the form of a printed text report. Report generation is done automatically using text generating techniques.In the evaluation phase of the generated report, 12 diabetic patients assessed the usability of the generated report in two aspects of the visual design and the content. Five indices of apparent attractiveness, ease of comprehension, applicability, description adequacy, and novelty of content was evaluated with a 5-point Likert scale checklist.
The usual tests for diabetics include blood glucose profiles, blood lipids profiles, and kidney status, with two parameters of blood pressure and weight. The list of commonly used tests for diabetics and their rules for interpreting is based on the Guideline in Table 2.
The reporting structure was designed based on observing the physician's performance in providing information at the visit, and was approved by a team of field experts. Each section of the report includes three types of feedbacks: descriptive, comparative, and conclusive.
List of commonly used tests for diabetics and their rules for interpreting based on the Guideline
Diabetic patients are divided into six categories in terms of comparison feedback and standard status feedback. Table 3 shows six possible cases for these feedbacks and the format of messages associated with it.
Following each of the six feedback feeds specified in Table 3, an associated information message is provided. Information topics fall into five main categories: The role and importance of the measured parameter in the body, the symptoms of parameter disturbance, the causes of the disorder, the consequences of the disorder, and the ways to improve the disorder. In order to identify the most relevant issue for each of the 6 test feedback modes, a checklist was provided by 3 diabetes specialists to determine, for each case, the proportion of information items by assigning a score of 1 to 5. By extracting messages related to the status of each test from the Message Library, they will be provided with the final report to be provided to the patient.
In the evaluation phase of the test, 12 diabetic patients are registered in the system and each interpretative report is produced. The quality of these reports is then evaluated from the perspective of diabetic patients. The characteristics of the patients participating in the study are shown in Table 4. The mean age of patients was 56.4 years with 72.1% being women.
Possible states for patient test status
Specifications of participants in the study
|Under 30||3 (23)|
|Education||Upper bachelor||2 (11)|
|Diploma and bachelor||3 (22)|
The evaluation of the report from the perspective of usability consists two aspects of visual design and content with five aspects of apparent attractiveness, ease of comprehension, applicability, description adequacy, and novelty of the content was evaluated by a valid checklist based on the Likert scale of 10. The design of the report with a mean score of 9.3 was attractive for patients. Patients evaluated the comprehension of the presented materials with an average score of 9.4. Also, the usability (8.3), the information adequacy (8.7) and the novelty (8.2) were scored acceptable by patients.
This study, for the first time, uses medical record to produce written patient specific information in Persian for diabetic patients. This system can be categorized as a data-to-text. Babytalk is a nursing summary synthesis system developed in 2012 in the Neonatal Intensive Care Unit of the British Royal Clinic . To evaluate this system, the nurses rated the section of the three aspects of the level of comprehension, accuracy, and usefulness of the text generated by the system. The results showed 90% perceptibility, 70% accuracy, and 59% contributing. The evaluation of the present system in this paper is based on indicators similar to those of this study.
The work done in the field of automated reporting can be divided into two general categories based on the type of attitude and expertise of the research group. The first category includes works done by engineering researchers that focus on artificial intelligence techniques and algorithms, machine learning methods, and automated reasoning, and less attention to real-world issues such as the implementation platform and the characteristics of the audience. In another group, personalized information studies in the field of medicine and health are generally focused on assessing the impact of the system on health-related outcomes. The Move More for Life  system developed a personalized educational pamphlet to improve the physical activity of breast cancer survivors, and assessed its impact on 330 participants. The results showed that the individuals in the intervention group were three times more likely to perform resistance exercises than the control group. One of the weaknesses in these studies is the failure to provide sufficient explanation about the characteristics and manner in which the personal documents manufacturer system works. In the present article, with a detailed description of the system design process, it has been tried to compensate for this vacuum and facilitate the production of similar systems in the future.
One of the features of the report produced in this study from the patient's point of view, which is mentioned in the results section, is the ability to understand the patient need. The paper showed in this regard that the better information content is closer to the level of patient perception; better information is absorbed by the reader and more likely to be applied .
The only study of information personalization for diabetic patients is a study that classifies educational products based on the level of health literacy and learning style of diabetic patients and measures its impact on increasing the knowledge of 160 patients by RCT method. The results showed that the knowledge of patients who received personalized educational products was significantly more than the control group.
Despite the many advances that have been made in the field of natural language processing and text production in recent years, Persian language has not progressed in line with other languages due to limitations due to the lack of textual contexts and basic tools. According to the sources, so far, no research has been done on the production of text based data in Persian in the field of health. The number of participants and the limited time of the study can be considered as one of the limitations of this study, which has an impact on the reliability of the results. Adding self-care materials and taking into account behavioral factors in the report can be expressed as suggestions for future work. Also, in later studies, the long-term effectiveness of the system's impact on outcomes such as increased knowledge and self-awareness of the patient can be assessed.
Overall, the results of this study showed that the interpretive report from the viewpoint of diabetic patients is desirable and patients are willing to receive more information about their health status. The findings of this study can be used as guidance for the design of the next phase of the study, it means the study of the effectiveness of the system on patients.