Skip to main content
  • Research article
  • Open access
  • Published:

Merits, features, and desiderata to be considered when developing electronic health records with embedded clinical decision support systems in Palestinian hospitals: a consensus study

Abstract

Background

Electronic health records (EHRs) with embedded clinical decision support systems (CDSSs) have the potential to improve healthcare delivery. This study was conducted to explore merits, features, and desiderata to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs.

Methods

A mixed-method combining the Delphi technique and Analytic Hierarchy Process was used. Potentially important items were collected after a thorough search of the literature and from interviews with key contact experts (n = 19). Opinions and views of the 76 panelists on the use of EHRs were also explored. Iterative Delphi rounds were conducted to achieve consensus on 122 potentially important items by a panel of 76 participants. Items on which consensus was achieved were ranked in the order of their importance using the Analytic Hierarchy Process.

Results

Of the 122 potentially important items presented to the panelists in the Delphi rounds, consensus was achieved on 110 (90.2%) items. Of these, 16 (14.5%) items were related to the demographic characteristics of the patient, 16 (14.5%) were related to prescribing medications, 16 (14.5%) were related to checking prescriptions and alerts, 14 (12.7%) items were related to the patient’s identity, 13 (11.8%) items were related to patient assessment, 12 (10.9%) items were related to the quality of alerts, 11 (10%) items were related to admission and discharge of the patient, 9 (8.2%) items were general features, and 3 (2.7%) items were related to diseases and making diagnosis.

Conclusions

In this study, merits, features, and desiderata to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs were explored. Considering items on which consensus was achieved might promote congruence and safe use of EHRs. Further studies are still needed to determine if these recommendations can improve patient safety and outcomes in Palestinian hospitals.

Peer Review reports

Background

Improving patient safety has become a top priority in healthcare systems around the globe [1]. Recent digital innovations and advancements in information technology (IT) have radically transformed many aspects of our world today, including healthcare delivery. Electronic health records (EHRs), also known as computerized patient records have largely replaced traditional paper-based health records in many hospitals and other healthcare delivery centers around the world [2]. These EHRs allow systematic collection and compilation of patient information such as medical history, vital signs, laboratory results, medical imaging reports, prescription and non-prescription medications, and other notes made by the healthcare providers (physicians, hospital pharmacists, and nurses) [2, 3]. Many of these EHRs include clinical decision support systems (CDSSs) that can offer medical and clinical information, warnings, alerts, reminders, and/or other support services that might help the healthcare providers make decisions and improve healthcare delivery [4]. Previous studies have reported that EHRs reduced documentation times, increased adherence of the healthcare providers to guidelines, improved communication between the healthcare providers, reduced medication errors, reduced adverse drug events, improved healthcare delivery, and promoted safer healthcare practices [5,6,7]. Currently, there is consensus that EHRs have the potential to improve healthcare delivery and therefore their full adoption and implementation have been recommended and incentivized [6, 8].

Despite the well-established benefits of adopting and successfully implementing well-designed EHR and CDSS systems, the literature has also reported many cases in which healthcare providers were dissatisfied with many features of EHR and CDSS systems [9,10,11]. It has been argued that the healthcare providers spend more time interacting with computers compared to time spent interacting with their patients face-to-face [12]. This might result in compromising the quality and safety of healthcare delivery, worsening patient-healthcare provider relationship, and fatigue of healthcare providers [9].

Additionally, poor design of EHRs and CDSS systems can adversely affect patient safety and consequently might compromise the patient health outcomes [13,14,15,16]. In many cases, system failures have led to incorrect identification of patients, dose calculation errors, and inability to access essential patient clinical data [13]. Previous studies have shown that EHRs and vendors were reported as sources of harm to patients [13, 14]. Howe et al. analyzed reports voluntarily entered by nurses into the Pennsylvania Patient Safety Authority database in the period from 2013 through 2016 [14]. The study reported that 557 safety events related to usability of EHRs resulted in patient harm. Of those, 468 required monitoring to prevent harm, 80 caused temporary harm, 7 cause permanent harm, and 2 allegedly resulted in death of patients.

EHR and CDSS systems differ significantly in terms of the informational items that the healthcare providers need to collect and enter as well as the support services that such systems can provide. Khalifa reported 10 success strategies and requirements for CDSSs [17]. Roshanov et al. reported features of effective CDSSs through a meta-regression of 162 randomized trials [18]. Fung et al. compared three commercial knowledge bases used in CDSSs to detect drug-drug interactions [19]. In another study, Ratwani et al. analyzed user-centered design processes in EHRs from 11 vendors [20]. The analysis provided insights on how to improve these EHRs. In today’s healthcare system, hospitals and other healthcare centers might either purchase a commercial EHR system with or without a CDSS or opt to develop a homegrown (in-house) system. In either case, there has been many calls to improve commercial as well as homegrown EHR and CDSS systems since their inception [19, 21]. Scheife et al. convened experts and developed consensus recommendations for systematic evaluation of drug-drug interactions for CDSSs [21].

Currently, evidence for the benefits and desirable features of EHRs and CDSS is limited and contradictor. Similarly, little was narrated on which important safety features to consider when designing, developing, and/or upgrading EHRs with embedded CDSSs. As many hospitals and healthcare centers design, develop, and/or upgrade their own homegrown EHRs and CDSSs, healthcare providers and IT/programming experts are left wondering what safety features are important to consider in the absence of real guidelines helping developers and implementers of EHRs and CDSSs.

This study was conducted to develop a core list of important safety features and desiderata to consider when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs and explore the opinions and views of a panel of experts in EHRs compared to handwritten patient records with regards to patient safety, cost, record keeping, and workflow. The core list of important safety features and desiderata might be used to guide decision makers in healthcare and IT/programming sectors when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs.

Methods

Study context

In Palestine, healthcare is delivered through three main sectors: the government, the United Nations Relief and Works Agency for Palestine Refugees (UNRWA), and the private sector. Many hospitals in Palestine currently adopt and use some sort of EHRs. Recently, the Palestinian Ministry of Health has applied an electronic health information system (HIS) to connect all governmental hospitals operating in the West Bank of Palestine using AviCenna Health Information Medical System [22]. In 2009, the UNRWA started developing an EHR system for its 143 health centers in Jordan, Syria, Lebanon, the West Bank, and Gaza Strip. In 2017, the in-house built EHR system that was developed to improve services for common illnesses, maternal and child health, non-communicable diseases, laboratory and pharmacy was implemented in 121 health centers of which 100 were paperless [23].

Because different EHR systems are used in Palestinian hospitals, it is not always possible to efficiently transfer and/or share patient information between different healthcare delivery establishments. EHRs used in governmental hospitals contain features that are missing from EHRs used in the other sectors, and vice versa. Currently, there are many design, logistic, technical, and compatibility issues that need to be addressed before efficient transfer of patient information between different providers of healthcare services would become possible. Probably, the first step in addressing these issues is to achieve consensus on the important features that need to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with embedded CDSSs.

Study design

In the current study, a mixed method combining the Delphi technique and the Analytic Hierarchy Process was used. The Delphi technique was used as a formal consensus technique to develop and achieve consensus on a core list of important safety features to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. The Analytic Hierarchy Process was used to rank these features (which will be referred to as items from now and on) in the order of their importance. Figure 1 shows a flowchart illustrating the different phases of the study.

Fig. 1
figure 1

Flowchart illustrating the different phases of the study

Collecting potentially important items

Literature search

A thorough literature search was conducted to identify potentially important items to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs (Fig. 1). Although a systematic review of the literature was not performed in this study, whenever possible, PRISMA statement was followed [24]. Adherence to PRISMA statement is shown in Additional file 1.

The databases MEDLINE, EMBASE, COCHRANE, CInAHL, SCOPUS, and the search engine Google Scholar were used. The search strategy combined key terms like “computerized”, “electronic”, “patient”, “inpatient”, “medical”, “health”, “hospital”, “record”, “profile”, “chart”, “prescribing”, “e-prescribing”, “e-health”, “e-prescription”, “medicine”, “medication”, “clinical decision support”, “embedded”, “integrated”, “incorporated”, “coupled”, “features”, “entry”, “information”, “evaluation”, “safety”, “assessment”, “review”, and “quality”. The terms used were combined using “AND” and “OR” options. The search was supplemented with a manual search within the references of the articles retrieved from the search. Together, the manual search within the references of the articles retrieved and Google Scholar allowed inclusion of grey literature. The final literature search was performed on March 12th, 2019. Titles and abstracts of articles retrieved from the search were screened manually before deciding on which articles would be subjected to a full text review [1, 2, 4,5,6,7, 9, 10, 17,18,19,20,21, 25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62]. Original studies, reviews, and reports were included if they reported standard and advanced features of EHRs and CDSSs. Papers on prevalence and incidence of medical errors in health records were not included. As the main desired outcome of this search was to identify and obtain potentially important items to be considered, articles with mention of safety, features, and/or quality of EHRs with embedded CDSSs were given a priority for a full text review. Articles were reviewed if they were in English language. No restrictions to study design, year of publication, and publication status were applied.

Titles and abstracts of the retrieved papers and reports were screened for eligibility after removing duplicate results. The literature search was conducted by the main investigator (PhD). To ensure reproducibility, the screening process was repeated three times. The literature search was verified by another researcher (PhD) who had previous experience in conducting literature search within the databases used. The second researcher reviewed all the decisions to include and/or exclude titles and noted any discrepancies. Conflicting results were resolved by discussion and consensus.

Extraction of items

The full text of each paper and report included was reviewed by the main author (PhD). A data collection form was created using an Excel spreadsheet (Microsoft Inc.) which was used to collect the items extracted by the main investigator (PhD) from the literature. To ensure reproducibility, the process was repeated three times. Another researcher (PhD) independently reviewed the highlighted text and the items extracted into the data collection form. Conflicts were resolved by discussion and consensus. A list of items was constructed after removing duplicates.

The first Delphi round: interviews with key contact experts

Key contact experts in the field were interviewed to collect more potential items that were not found in the literature with the objective of supplementing the list of items extracted from the literature. Whenever possible, consolidated criteria for reporting qualitative research (COREQ) were followed [63]. Adherence to COREQ checklist is shown in Additional file 2.

Potential participants were identified through personal contacts in the field. A purposive sampling method was used to invite and recruit the study participants. An email was sent to 25 potential participants inviting them to take part in the study. The email invitation included details of the design and objectives of the study. The sample size was informed by previous studies in which interviews were conducted to supplement items to be used in the subsequent iterative Delphi rounds [8, 22, 64,65,66,67,68]. Those who did not respond receive 3 reminders, each reminder was 1 week a part. Those who were willing to participate replied with an email expressing their interest in participation in the study. Prior to interviews, the interviewees were informed that the study was being conducted as a research study with the far aim of improving EHRs and CDSSs in Palestinian hospitals. The interviewees were also informed that investigators had no personal interest to influence the final outcomes of the project. The interviews were conducted by the main investigator (PhD) who was male in gender and had previous experience in conducting interviews and studies using the Delphi technique. The interviewer held an academic associate professor position at the time of the interviews. The key contacts were extensively involved either in planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs for a period of not less than 5 years. The participants were interviewed one-on-one in a calm side of their workplaces. After obtaining their sociodemographic and practice/experience details, the interviewees were asked to mention all items that were, in their opinion, important to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. The interviews were open and on occasions, the interviewer mentioned a feature that was provided by another interviewee. The objective of providing such prompts was to provide the interviewee an opportunity to “hitch-hike” with other interviewees and mention more potential items. The interviews were audio-recorded and were not repeated. On occasions, the interviewer made notes of items mentioned by the interviewees during the interviews. The main investigator listened to each recording three times and the interviews were analyzed for their content. Items were interpreted from the recordings and extracted directly into the data collection form. Analysis was repeated again and continued till all items were grouped. Grouping was derived from the data. Items obtained from each interview were returned to the interviewee for corrections and if they wished to add more items. Participants were recruited from 5 different Palestinian hospitals and IT/programming companies.

Together, items collected from the literature and those mentioned by the interviewees were summarized and formulated into statements. These statements were compiled into an initial list of potentially important items which was then used as input in the subsequent iterative Delphi rounds.

The questionnaire

A questionnaire was designed to contain 3 sections. The first section was destined to collect the sociodemographic, academic, and professional variables of the participants. In this section, the participants (panelists) needed to disclose their gender, age, academic qualifications, specialty/rank/hierarchy, and employer. The second part was destined to explore the views and opinions of the panelists on their perceived impact of EHRs with embedded CDSSs compared to hand-written paper-based patient records. In this section, the panelists were supposed to express the level of their disagreement/agreement on 12 items related to patient’s safety, costs, record keeping, and workflow using a Likert-scale of 1–5 (1 indicated strong disagreement and 5 indicated strong agreement). The 5-point Likert-scale was extensively used in measuring the level of disagreement/agreement on items presented to participants [69]. A space was left for comments and the panelists were encouraged to include written qualitative comments to supplement their scores. The third section contained the initial list of potentially important items to be presented in the Delphi iterative rounds. The list contained 122 items that were collected from the literature and/or provided by the interviewees during the interviews. The panelists had to express the level of their disagreement/agreement with each potentially important item using a Likert-scale of 1–9 (1 indicated strong disagreement, i.e. in the participant’s view, the item was not important and 9 indicated strong agreement, i.e. in the panelist’s view, the item was important). Although many consensus indices have been used in Delphi technique [70], the 9-point Likert-scale has emerged as one of the most commonly employed scale in panelists’ extent of disagreement/agreement with items in the Delphi technique [8, 22, 64,65,66,67]. Below each item, a space was provided and the panelists were encouraged to write comments to qualify/justify their scores. As a high response rate was desired in this study, the panelists received paper-based questionnaire by hand.

Reviewing and piloting the questionnaire

The questionnaire was reviewed and piloted by 5 physicians (2 residents and 3 interns), 2 nurses, 1 hospital pharmacist, and 3 IT/programming specialists (Fig. 1). The aim of the review and pilot was to test the questionnaire for readability and comprehensibility. Based on comments of the participants, some items were rephrased for clarity and some medical and programming specific terms (jargon) were replaced by simpler terms.

Panel of healthcare providers and IT/programming specialists

A panel of healthcare providers (physicians, nurses, and hospital pharmacists) and IT/programming specialists was recruited using a purposive sampling technique (Fig. 1). The potential panelists were identified and approached using personal contacts in the field. Selection of the panelists is one of the most critical steps in the Delphi technique. The panelists need to be knowledgeable of the subject being investigated [71]. In this study, the panelists selected were rich in information and had appreciable experience in planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. The panelists were selected, approached, invited, and included based on their existing experience and involvement in planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. The selection process ensured diversity of the panelists in terms of gender, age group, profession, academic qualifications, length of working experience, rank/hierarchy, and type of employer. Upon invitation, the design and objectives of the study were explain to potential panelists. The panelists needed to provide their informed consent to qualify for this study. The inclusion criteria were as follows: 1) possessing a qualifying academic degree in one of the healthcare or IT/programming professions, 2) having a license to practice (for healthcare providers), 3) at least 5 years of experience in using or involvement in planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs, 4) employment in a public or private hospital, or IT/programming firm involved in planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs, and 5) willingness to participate and providing informed consent. Physicians who were specialists/consultants, residents, and interns were recruited to the panel. Senior and staff hospital pharmacists and nurses were also recruited. Recruitment of hospital pharmacists and nurses was intentional to ensure representation of other healthcare provision team. IT/programming specialists were also represented in the panel. The panel included 76 panelists. The number of panelists recruited for this study was within the range of number of panelists used in previous studies in which the Delphi technique was used to develop and achieve formal consensus on issues lacking consensus in healthcare [22, 41, 64,65,66,67,68, 72,73,74,75,76,77,78,79,80,81]. It is noteworthy mentioning that currently there is no formal consensus on the number of panelists to be included in a study using the Delphi technique. Previous studies used panel sizes in the range of 10 to 1000 [78]. No financial incentives were offered to the panelists in exchange of their participation in this study.

The iterative Delphi process

All panelists received paper-based copies of the questionnaire and an iterative Delphi process with controlled feedback was initiated (Fig. 1). In the second Delphi round, the panelists provided their sociodemographic, academic, and professional details, expressed their level of their disagreement/agreement on the 12 statements, and voted on each of the 122 items. The Delphi technique was conducted in accordance with the conducting and reporting of Delphi studies (CREDES) guidelines [81]. Adherence to CREDES guidelines is shown in Additional file 3.

Analysis of votes

Data were extracted from the questionnaires completed in the second Delphi round and were entered into an Excel Sheet (Microsoft Inc.). Basic descriptive statistics like the 1st quartile (Q1), median (Q2), 3rd quartile (Q3), and the interquartile range (IQR). The IQR was computed by subtracting (Q3 - Q1). The descriptive statistics were computed for each item separately. The qualitative comments were analyzed for their content, summarized, and categorized by the main investigator (PhD). Summaries were returned to the participants for feedback and corrections.

Definition of consensus

The definition of consensus used in this study was informed by previous studies in which the Delphi technique was used to achieve formal consensus on issues in healthcare [8, 22, 64,65,66,67,68, 74, 82]. Briefly, when a panelist scored a vote of 1–3 on an item, this meant that in the opinion of the panelist, the item was not important and should be excluded from the final core list. When a panelist scored a vote of 7–9 on an item, this meant that in the opinion of the panelist, the item was important and should be included in the final core list. When a panelist scored a vote of 4–6 on an item, this meant that the panelist was indecisive if the item was important or not (partial agreement). Criteria for consensus were defined a priori as: 1) when the median of scores on an item was within the range of 1–3 and the IQR was ≤2 of at least 75% of the panelists, the item was considered as unimportant and was excluded from the final core list, 2) when the median of scores on an item was within the range of 7–9 and the IQR was ≤2 of at least 75% of the panelists, the item was considered as important and was included in the final core list, and 3) when the median of scores on an item was within the range 4–5 and/or the IQR was > 2, the item was considered equivocal. It was decided a priori that all equivocal items in the second Delphi round would be subjected to a third Delphi round.

The revised questionnaire containing equivocal items and the third iterative Delphi round

All items that remained equivocal in the second Delphi round were included into a revised questionnaire (Fig. 1). For each equivocal item, the panelists were provided with: 1) a reminder of their own score, 2) the median score, 3) the IQR, and 4) summary of the qualitative comments that were made by other panelists to qualify/justify their scores at the time of voting. The panelists were given the chance to reconsider or maintain their scores in view of the scores and comments of the other panelists. Anonymity of the panelists was maintained during the Delphi rounds. Scores obtained in the third Delphi round were analyzed using the same definitions used in the second Delphi round.

Informed by previous studies in which the Delphi technique was used to achieve consensus on issues in healthcare and based on the votes and comments of the panelists in the third Delphi round, consensus on the remaining equivocal items was unlikely. Therefore, it was decided not to conduct a fourth Delphi round.

Items on which consensus was achieved to be included into the final core list were grouped by the main investigator (PhD) into 9 categories. Groups were derived from the items on which consensus was achieved to be included in the final core list.

Ranking of items in order of their importance

In this study, it was decided to rank the items that were included in the final core list in order of their importance using the Analytic Hierarchy Process (Fig. 1). The Analytic Hierarchy Process is a powerful tool that has been extensively used in multi-criteria decision analysis, notably, in healthcare [83,84,85]. The Analytic Hierarchy Process allows addressing and facilitating making decisions on complex problems with multi-criteria. The method is especially appropriate for small group settings [83, 85]. Using this method, participants can make pairwise comparisons in order to hierarchize items in the order of their perceived importance.

Ten participants who already participated in the Delphi rounds were purposively approached and invited to take part in this step. Participants who provided more qualitative comments on the items during the Delphi rounds to qualify/justify their scores were given a priority. The participants were asked to make pairwise comparisons of items within the same category using a scale of 9-points. When a higher score was given to an item, this indicated the higher importance of that item compared to the other items in the same category. Scores from each participant were used to create matrices in Excel Spreadsheets. Mathematical formulas originally developed by Saaty were used to calculate importance weights (%) with their consistency ratios [86]. Scores were considered when their consistency ratios were less than 0.1. For each category, importance weights (%) were calculated for each item within all categories. Importance weights (%) of all items within the same category sum to 100%.

Statistical analysis

Scores on each item were entered into GraphPad Prism 6.0 for Windows (GraphPad Software). One-way analysis of variance (ANOVA) with Bonferroni post-hoc tests were used to compare differences in importance weights (%) [85]. Statistical significance was considered * when the p-value was < 0.05, ** when the p-value was < 0.01, *** when the p-value was < 0.001, and **** when the p-value was < 0.0001.

Ethical considerations

This study was approved by the Institutional Review Board (IRB) of An-Najah National University. The panelists understood that the Delphi technique was a semi-anonymous technique in which the identity of the panelist is known to the investigator while the panelist remains anonymous to the rest of the panel members. Scores of the panelists weighed equally in the analysis. All participants provided written informed consent.

Results

Search results

The literature search yielded a total of 4023 records. Duplicate records were removed. The rest of records were screened for ineligibility based on title and abstract, data were extracted from 51 records.

Study participants

Interviews

Interview invitations were sent to 25 key contact experts. Of those, 21 (84.0%) expressed interest in participation and 4 (16.0%) were not available at that time. Interviews could not be scheduled with 2 (8.0% of those originally invited) who later declined due to personal reasons and time constrains. Interviews were conducted with 19 (76.0% of those originally invited) key contacts in the field who were 3 specialist physicians, 4 senior residents, 3 interns, 2 hospital pharmacists, 4 nurses, and 3 IT/programming specialists. Respondents belonged to all specialties that were purposively invited in the study. The detailed sociodemographic, academic, and professional details of the panelists who participated in this stage of the study are shown in Table 1. The median duration of the interviews was 37 with an IQR of 16 min.

Table 1 Sociodemographic, academic, and professional details of the panelists who participated in the study (n = 76)

The panelists

In this study, a total of 76 panelists who were invited voted in the second and third iterative Delphi rounds, giving a response rate of 100%. Both genders were represented in the panel, females represented 40.8% of the panel members. The panel members belonged to different age groups (55.3% were 40 years old and above). Of the participants, the largest groups were physicians (42.0%) and IT/programming specialists (25.0%). However, nurses and hospital pharmacists were also represented in the panel. The panelists were employed by private and public hospital, as well as, IT/programming firms. The detailed sociodemographic, academic, and professional details of the panelists who participated in this stage of the study are shown in Table 1.

The analytic hierarchy process

A total of 10 participants made pairwise comparisons of the items in the Analytic Hierarchy Process. All 10 participants who were invited to take part in this step made pairwise comparisons, giving a response rate of 100%. The participants were physicians, pharmacists, nurses, and IT/programming specialists. Of all participants, 50.0% were physicians by profession. The detailed sociodemographic, academic, and professional details of the panelists who participated in this stage of the study are shown in Table 1.

Merits of EHRs with embedded CDSSs compared to traditional paper-based handwritten patient records

Merits related to patient’s safety

The vast majority (93.4%) of the panelists agreed (either strongly agreed or agreed) that EHRs with embedded CDSSs can reduce medication errors compared to traditional paper-based handwritten patient records (Table 2). However, those who voted neutral or with disagreement commented that EHRs with embedded CDSSs were associated with other types of errors. Of all panelists, 77.6% agreed that EHRs with embedded CDSSs can reduce adverse medication events. Those who disagreed or were neutral cited conflicting findings reported in the literature. The vast majority of the panelists (89.4%) agreed that EHRs with embedded CDSSs can improve communication between healthcare providers. Those who disagreed or were neutral stated that more interactions were needed to resolve issues when inaccurate information was entered. About 80% of the panelists agreed that EHRs with embedded CDSSs can improve the overall patient care experience. Those who did not agree cited poor design of systems, drop-down menus, screen, and automatic filling functions might increase likelihood of errors and may negatively impact the overall patient care experience.

Table 2 Views and opinions of the panelists on EHRs with embedded CDSSs

Merits related to costs

About 68% of the panelists agreed that EHRs with embedded CDSSs can improve the prescriber’s ability prescribe more cost-effective medications (Table 2). The panelists who did not agree stated that this might depend on the list of product options available through the system for the prescriber to choose from. About 74% of the panelists agreed that EHRs with embedded CDSSs can increase the prescriber’s likelihood to discontinue unnecessary and/or ineffective medications. Those who did not agree stated that the literature was inconclusive. About 88% of the panelists agreed that EHRs with embedded CDSSs can decrease the costs associated with adverse medication events and medication errors. Those who did not agree stated that there were other costs associated with the other types of errors encountered in EHRs with embedded CDSSs.

Merits related to record keeping

The vast majority of the panelists (97.3%) agreed that EHRs with embedded CDSSs can improve storage of patient information and prescription records (Table 2). Those who did not agree cited efforts needed to keep and maintain copies of these EHRs. Again, the vast majority (about 96%) of the panelists agreed that EHRs with embedded CDSSs can improve prescriber’s ability to trace patient’s prescribing information. Those who did not agree cited efforts needed to improve search and retrieval through such systems. 75% of the panelists agreed that EHRs with embedded CDSSs can improve prescriber’s ability to monitor medications and evaluate clinical outcomes. Those who did not agree cited efforts needed to improve reminders that should pop-up only when necessary to avoid prescriber’s desensitization.

Merits related to workflow

About 86% of the panelists agreed that EHRs with embedded CDSSs can reduce wait times needed to call prescribers to clarify illegible or ambiguous medication orders (Table 2). Those who did not agree mentioned that more wait times were needed when technology failed and systems malfunctioned. However, only about 47% of the panelists agreed that EHRs with embedded CDSSs can improve workflow of prescribers and other healthcare team by enabling copying and editing of prescribing information. However, more than half of the panelists did not agree and cited challenges to workflow, more personnel, training, and maintenance efforts needed.

The iterative Delphi rounds

Items included in the core list with their weights of importance

Of the 122 items presented to the panelists in the second Delphi round, consensus was achieved to include 77 (63.1%) items. The remaining 45 items were included into the revised questionnaire and were subjected to a third Delphi round. In the third Delphi round, consensus was achieved on further 33 (27.1%) items. Following the two iterative Delphi rounds, consensus was achieved on 110 (90.2%) items. These items were grouped under 9 categories. Of these, 16 (14.5%) items were related to the demographic characteristics of the patient, 16 (14.5%) were related to prescribing medications, 16 (14.5%) were related to checking prescriptions and alerts, 14 (12.7%) items were related to the patient’s identity, 13 (11.8%) items were related to patient assessment, 12 (10.9%) items were related to the quality of alerts, 11 (10%) items were related to admission and discharge of the patient, 9 (8.2%) items were general features, and 3 (2.7%) items were related to diseases and making diagnosis. All categories and items ranked in the order of their importance weights are listed in Table 3. The Additional file 4:Tables S1-S9 show multiple comparisons of importance weights items in each category.

Table 3 Important features of EHRs with embedded CDSSs on which consensus was achieved in this study

Items related to the demographic characteristics of the patient

The 16 items related to demographic characteristics of the patient on which consensus was achieve included body weight, height, surface area, and body mass index. Items in this category ranked by their importance weights are shown in Table 3. The item related to the body weight of the patient received significantly (p-value < 0.01) higher weight scores compared to body surface area and body mass index. Multiple comparisons of importance weights of items in the demographic characteristics of the patient category are shown in Additional file 4: Table S1.

Items related to prescribing medications

Consensus was also achieved on 16 items related to prompts with regards to the name of the medication, dose, frequency, route, formulation, units, duration, date, and who prescribed the medication. Prompts to specify the dose of the medication prescribed received significantly (p-value < 0.01) higher weight scores compared to specifying the formulation and other items. Multiple comparisons of importance weights of items in the prescribing medications category are shown in Additional file 4: Table S2.

Items related to checking prescriptions and alerts

Consensus was achieved on 16 items related to checking the suitability of the prescription in relation to the patient’s clinical and pathological conditions, discontinuing a medication, warnings of drug interactions, and justifying deviations from evidence-based prescribing. Ability to assess the suitability of the dose in relation to patient’s clinical and pathological conditions received significantly (p-value < 0.01) higher weight scores compared to warnings of potential contraindications. Multiple comparisons of importance weights of items in the checking prescriptions and alerts category are shown in Additional file 4: Table S3.

Items related to patient’s identity

Consensus was achieved on 14 items related to patient’s name, identification number, gender, date of birth, age, telephone number and address. Name of the patient received significantly (p-value < 0.05) higher weight scores compared to gender. Multiple comparisons of importance weights of items in the patient’s identity category are shown in Additional file 4: Table S4.

Items related to patient assessment

Consensus was achieved on 13 items related to prompts to enter clinical and pathological conditions of the patient, laboratory results, and patient history. Prompts to enter presenting symptoms of the patient received significantly (p-value < 0.05) higher weight scores compared to ability to enter and/or automatically import results of medical images ordered for the patient. Multiple comparisons of importance weights of items in the patient assessment category are shown in Additional file 4: Table S5.

Items related to quality of alerts

Consensus was achieved on 12 items related to providing evidence-based suggestions and alerts, ability to distinguish between serious and minor risks, and guiding the prescriber to best practices. Providing evidence-based suggestions and alerts received significantly (p-value < 0.01) higher weight scores compared to ability to distinguish serious and minor allergy. Multiple comparisons of importance weights of items in the quality of alerts category are shown in Additional file 4: Table S6.

Items related to admission and discharge of the patient

Consensus was achieved on 11 items related to details of the admission and discharge of the patient. Items related to the hospital’s admission number assigned to the patient at the time of admission received significantly (p-value < 0.05) higher weight scores compared to ability to name(s) of the ward(s) to which the patient was (were) admitted. Multiple comparisons of importance weights of items in the admission and discharge of the patient category are shown in Additional file 4: Table S7.

Items related to general features

Consensus was achieved on 9 items related to general features. The item related to a user friendly system received significantly (p-value < 0.05) higher weight scores compared to the item related to specifying why alerts are displayed. Multiple comparisons of importance weights of items in the general features category are shown in Additional file 4: Table S8.

Items related to diseases and making diagnosis

Consensus was achieved on 3 items related to diseases and making diagnosis. The item related to entering diagnosis received significantly (p-value < 0.05) higher weight scores compared to ability to access to offline, online, and searchable databases and references related to diseases and differential diagnosis. Multiple comparisons of importance weights of items in the diseases and making diagnosis category are shown in Additional file 4: Table S9.

Items on which consensus was not achieved

Following the third Delphi round, a total of 12 (9.8%) items remained equivocal (Table 4). Consensus was not achieved on items related to email address of parent(s)/guardian(s) of a pediatric patient, registration numbers of physicians, indicating why the medication was prescribed, ability to provide the prescriber with dosing recommendations, ability to check compatibility between medications and diluents, alerting prescribers of unlicensed medications for pediatric patients, verification of each medication administration by two persons, allowing search for all medications available in the country, collecting adverse reactions and prescribing errors, providing warnings to medications requiring monitoring and high alert medications.

Table 4 Features of EHRs with embedded CDSSs on which consensus was not achieved and remained as optional in the opinions of the panelists who participated in this study

Discussion

In the present study, consensus was sought on a core list of important safety features to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. Currently, little guidance is available on what safety features are important to consider when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. This study presents for the first time a comprehensive consensus core list of 110 important safety features categorized into 9 categories that could be useful in guiding decision makers in hospitals, IT/programming industry, clinicians, and other healthcare provision team to design and improve EHRs and CDSSs. Keeping in mind that it can be difficult to consider all of these items, decision makers might need to select some items considering their relative importance weights.

This study was conducted in different stages using a mixed method (Fig. 1). The initial list of items that were collected after an extensive literature search and review followed by interviews with key contact experts in the field. The decision to conduct a thorough search instead of a systematic review was made after carefully considering the following issues: 1) objectives of the current study, 2) nature of the research question, 3) problem/population, intervention, comparison, outcome (PICO), 4) scope, and 5) number and nature of papers/materials to be included in the study [87]. Moreover, all panelists invited took part in the second Delphi round. This high response rate adds strength and validity to the design and findings of this study [22, 64,65,66,67,68, 74, 81]. The panel size was within the range of sizes used in previous studies involving developing concepts and achieving consensus on issues in healthcare [22, 41, 64,65,66,67,68, 72,73,74,75,76,77,78,79,80,81]. Despite the small size of the panel, both genders, different age groups, specialties/ranks/hierarchies, and type of employer were represented all represented (Table 1). Such diversity can add to the validity and suitability of using the consensus core items when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs.

In general, the majority of the panelists agreed that EHRs with embedded CDSSs can improve patient’s safety, costs, record keeping, and workflow compared to paper-based handwritten patient records (Table 2). The panelists in this study, generally seemed to agree with findings reported in the literature as EHRs with embedded CDSSs improved the quality of healthcare delivery, increased time efficiency, improved adherence to guidelines, reduced medication errors and adverse medication events [5, 40]. However, some panelists were neutral or even disagreed with some statements and cited negative effects of EHRs with embedded CDSSs in relation to patient’s safety, costs, record keeping, and workflow in agreement with those reported in the literature [29, 44, 52].

Gold standards in guiding the design and development of systems EHRs with embedded CDSSs do not exist. When gold standards are absent, stakeholders are left wondering what items are important to consider when designing such systems. In this case, consensus achieving approaches might be useful in reducing bias, increase transparency, and adding strength to judgmental approaches [88]. It has been argued that professionals tend to adhere to guidelines when they agree with them compared to guidelines they do not agree with. Therefore in this case, it is believed that decision makers in both health provision team and IT/programming are expected to consider such consensus items when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs.

In this study, consensus was achieved that the system should be able to record and keep admission and discharge information of the patient (Table 3). Such information might be used to answer why, when, where, and how questions, like when the patient was admitted to the hospital, why the patient was admitted, where the patient was admitted (ward and bed number), who admitted and who discharged the patient. Such information can be indispensable in tracking the patient’s history and retrieving the patient’s information whenever needed. Consensus was also achieved on items related to the patient’s identify and body characteristics. Body weight and surface area might be used by the prescribers to calculate the dose for example, especially in pediatric patients [89, 90]. The panelists also agreed that features need to include prompts and abilities to make entries related to patient assessment like presenting symptoms, vital signs, laboratory and imaging reports, co-morbidities, prescription and non-prescription medications, allergies, and adverse drug reactions. Such features might improve diagnostic accuracy [35, 42, 48, 51], effectiveness of medication review and patient care [36,37,38, 46], improve screening for allergies to medications [34], reduced adverse drug reactions [33], drug-drug, and drug-food interactions [4, 49, 50]. The panelists also agreed that the system should provide prompts to specifying doses of the medications prescribed, frequencies, routes of administration, dosage forms, dosing units, alternative medications, and duration of therapy. Such features were shown to reduce prescribing errors and administration errors [39, 43, 45]. Previous studies have shown that medication errors increased the length of hospital stays, costs of therapy and often resulted in death of the patient [91]. Consensus was also achieved on items related to alerts provided by the system after checking prescriptions. Previous studies have shown usefulness of EHRs with embedded CDSSs in assessing appropriateness and safety of doses in relation to renal function [32], drug allergies, contraindications, adverse drug reactions, drug-drug, and drug-food interactions [19, 33, 50]. It is noteworthy mentioning that in order to improve diagnostic accuracy and patient care, systems should provide the healthcare provider with up-to-date information and evidence [21]. In this study, consensus was achieved on items related to the quality of alerts and warnings provided by the system. The panelists agreed on items related to the ease of using the system (Table 3).

In the present study, consensus was not achieved to include items like the email address of the parent(s)/guardian(s) of the pediatric patient, registration number of the prescriber, indication for which the medication was prescribed, dosing recommendations, compatibility with diluents, unlicensed use of a medication, all medications available in the country, collecting adverse drug reactions, using high alert medications, and reporting medication errors. The decision to either consider these items or not is left to the decision makers when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. The decision to whether consider these items or not might be shaped by the individual needs of healthcare providers and decision makers in hospitals.

This study is not without limitations. First, a systematic review of the literature was not conducted in this study. Compared to qualitative reviews, systematic reviews are more robust and allow reproducibility of results. Second, data extraction was conducted by one investigator. Although another researcher verified the results independently, the risk of bias could have been reduced if two or more researchers independently conducted the data extraction step. One investigator summarized, extracted, and grouped the qualitative data. Items were directly entered into the data collection form as interpreted from the recordings. Although the participants had the opportunity to read and correct the summaries, risk of bias could have been reduced if two or more researchers independently summarized, extracted, and grouped the qualitative data or if the exact words of the interviewees were transcribed. Third, patients were not included as panelists in this study. Including patients in the panel should have allowed to explore patient’s views with regards to EHRs with CDSSs. In modern healthcare systems, patients are increasingly involved in designing their care plans. Fourth, the panel of experts recruited for this study was relatively small. However, there is no consensus on the number of panelists that should be included in a Delphi study [78]. Previous studies in healthcare used panel sizes in the range of 10–50 [64,65,66,67,68, 74]. In this study, care was exerted and a diverse panel was composed to include physicians, nurses, hospital pharmacists, and IT/programming specialists. Fifth, the panelists were identified by key contacts in the field and were recruited using a purposive sampling technique. Such sampling technique has long been criticized as biased [64,65,66,67,68, 74]. However, prior knowledge of the subject being investigated is a pre-requisite for potential participants to be selected into a panel. Finally, this study could have been conducted in a conference or large scientific meeting in which experts with interest in EHRs with embedded CDSSs would gather.

Conclusions

In this study, merits, features, and desiderata to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs were explored. Consensus was achieved on items related to safety features to be considered when planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. Considering items on which consensus was achieved might promote congruence and safety while planning for, designing, developing, implementing, piloting, evaluating, maintaining, upgrading, and/or using EHRs with CDSSs. Further studies are still needed to determine if these recommendations can improve patient safety and outcomes in Palestinian hospitals.

Availability of data and materials

Data related to this study are either presented in the results section or in the additional files. Raw data can be obtained from the corresponding author on reasonable request.

Abbreviations

%A:

Percentage of panelists who voted 7–9 on the item

ANOVA:

Analysis of variance

B:

Both (literature and interviews)

BSc:

Bachelor of Science

CDSSs:

Clinical decision support systems

COREQ:

Consolidated criteria for reporting qualitative research

CREDES:

Conducting and reporting of Delphi studies

EHRs:

Electronic health records

HIS:

Health information system

I:

Interviews

IQR:

Interquartile range

IRB:

Institutional Review Board

IT:

Information technology

L:

Literature

M:

Median

MD:

Doctor of Medicine

MSc:

Master of Science

NA:

Not applicable

Pharm.D:

Doctor of Pharmacy

PhD:

Doctor of Philosophy

PICO:

Problem/population, intervention, comparison, outcome

Q1:

1st quartile

Q2:

2nd quartile

Q3:

3rd quartile

SD:

Standard deviation

UNRWA:

United Nations Relief and Works Agency for Palestine Refugees

References

  1. Singh H, Sittig DF. Measuring and improving patient safety through health information technology: the health IT safety framework. BMJ Qual Saf. 2016. https://0-doi-org.brum.beds.ac.uk/10.1136/bmjqs-2015-004486.

    Article  Google Scholar 

  2. Adler-Milstein J, DesRoches CM, Kralovec P, Foster G, Worzala C, Charles D, Searcy T, Jha AK. Electronic health record adoption in US hospitals: Progress continues, but challenges persist. Health Aff (Millwood). 2015. https://0-doi-org.brum.beds.ac.uk/10.1377/hlthaff.2015.0992.

    Article  PubMed  Google Scholar 

  3. Jamoom E, Beatty P, Bercovitz A, Woodwell D, Palso K, Rechtsteiner E. Physician adoption of electronic health record systems: United States, 2011. NCHS Data Brief. 2012;(98):1–8.

  4. Amkreutz J, Koch A, Buendgens L, Trautwein C, Eisert A. Clinical decision support systems differ in their ability to identify clinically relevant drug interactions of immunosuppressants in kidney transplant patients. J Clin Pharm Ther. 2017. https://0-doi-org.brum.beds.ac.uk/10.1111/jcpt.12508.

    Article  CAS  PubMed  Google Scholar 

  5. Campanella P, Lovato E, Marone C, Fallacara L, Mancuso A, Ricciardi W, Specchia ML. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. Eur J Pub Health. 2016. https://0-doi-org.brum.beds.ac.uk/10.1093/eurpub/ckv122.

    Article  PubMed  Google Scholar 

  6. Palabindala V, Pamarthy A, Jonnalagadda NR. Adoption of electronic health records and barriers. J Community Hosp Intern Med Perspect. 2016. https://0-doi-org.brum.beds.ac.uk/10.3402/jchimp.v6.32643.

    Article  PubMed  Google Scholar 

  7. DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A, Kaushal R, Levy DE, Rosenbaum S, Shields AE, et al. Electronic health records in ambulatory care--a national survey of physicians. N Engl J Med. 2008. https://0-doi-org.brum.beds.ac.uk/10.1056/NEJMsa0802005.

    Article  CAS  PubMed  Google Scholar 

  8. Shawahna R. Development of key performance indicators to capture in measuring the impact of pharmacists in caring for patients with epilepsy in primary healthcare: a Delphi consensual study. Epilepsy Behav. 2019. https://0-doi-org.brum.beds.ac.uk/10.1016/j.yebeh.2019.07.034.

    Article  Google Scholar 

  9. Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann Intern Med. 2018. https://0-doi-org.brum.beds.ac.uk/10.7326/M18-0139.

    Article  PubMed  Google Scholar 

  10. Meigs SL, Solomon M. Electronic Health Record Use a Bitter Pill for Many Physicians. Perspect Health Inf Manag. 2016;1:13.

    Google Scholar 

  11. Nelson R. Nurses' dissatisfaction with electronic health records remains high. Am J Nurs. 2016. https://0-doi-org.brum.beds.ac.uk/10.1097/01.NAJ.0000505578.35140.cb.

    Article  PubMed  Google Scholar 

  12. Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, Westbrook J, Tutty M, Blike G. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med. 2016. https://0-doi-org.brum.beds.ac.uk/10.7326/M16-0961.

    Article  PubMed  Google Scholar 

  13. Virginio LA Jr, Ricarte IL. Identification of patient safety risks associated with electronic health records: a software quality perspective. Stud Health Technol Inform. 2015. https://0-doi-org.brum.beds.ac.uk/10.3233/978-1-61499-564-7-55.

  14. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA. 2018. https://0-doi-org.brum.beds.ac.uk/10.1001/jama.2018.1171.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Aldosari B. Patients' safety in the era of EMR/EHR automation. Inform Med Unlocked. 2017. https://0-doi-org.brum.beds.ac.uk/10.1016/j.imu.2017.10.001.

    Article  Google Scholar 

  16. Colicchio TK, Cimino JJ, Del Fiol G. Unintended consequences of Nationwide electronic health record adoption: challenges and opportunities in the post-meaningful use era. J Med Internet Res. 2019. https://0-doi-org.brum.beds.ac.uk/10.2196/13313.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Khalifa M, Alswailem O. Clinical decision support knowledge management: strategies for success. Stud Health Technol Inform. 2015. https://0-doi-org.brum.beds.ac.uk/10.3233/978-1-61499-538-8-67.

  18. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, Nieuwlaat R, Souza NM, Beyene J, Van Spall HG, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013. https://0-doi-org.brum.beds.ac.uk/10.1136/bmj.f657.

    Article  PubMed  Google Scholar 

  19. Fung KW, Kapusnik-Uner J, Cunningham J, Higby-Baker S, Bodenreider O. Comparison of three commercial knowledge bases for detection of drug-drug interactions in clinical decision support. J Am Med Inform Assoc. 2017. https://0-doi-org.brum.beds.ac.uk/10.1093/jamia/ocx010.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc. 2015. https://0-doi-org.brum.beds.ac.uk/10.1093/jamia/ocv050.

    Article  PubMed  Google Scholar 

  21. Scheife RT, Hines LE, Boyce RD, Chung SP, Momper JD, Sommer CD, Abernethy DR, Horn JR, Sklar SJ, Wong SK, et al. Consensus recommendations for systematic evaluation of drug-drug interaction evidence for clinical decision support. Drug Saf. 2015. https://0-doi-org.brum.beds.ac.uk/10.1007/s40264-014-0262-8.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  22. Shawahna R, Al-Atrash M. What do primary healthcare providers and complementary and alternative medicine practitioners in Palestine need to know about exercise for Cancer patients and survivors: a consensual study using the Delphi technique. Evid Based Complement Alternat Med. 2019. https://0-doi-org.brum.beds.ac.uk/10.1155/2019/7695818.

    Article  Google Scholar 

  23. Ballout G, Al-Shorbaji N, Abu-Kishk N, Turki Y, Zeidan W, Seita A. UNRWA’s innovative e-health for 5 million Palestine refugees in the near east. BMJ Innovations. 2018. https://0-doi-org.brum.beds.ac.uk/10.1136/bmjinnov-2017-000262.

    Article  Google Scholar 

  24. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2009.06.005.

    Article  PubMed  Google Scholar 

  25. El Mahalli A. Adoption and barriers to adoption of electronic health records by nurses in three governmental hospitals in Eastern Province, Saudi Arabia. Perspect Health Inf Manag. 2015;12:1f

  26. Chiang MF, Boland MV, Margolis JW, Lum F, Abramoff MD, Hildebrand PL, American Academy of Ophthalmology Medical Information Technology C. Adoption and perceptions of electronic health record systems by ophthalmologists: an American Academy of Ophthalmology survey. Ophthalmology. 2008. https://0-doi-org.brum.beds.ac.uk/10.1016/j.ophtha.2008.03.024.

    Article  Google Scholar 

  27. Wang T, Biedermann S. Adoption and utilization of electronic health record systems by long-term care facilities in Texas. Perspect Health Inf Manag. 2012;9:1g.

  28. Jamoom EW, Yang N, Hing E. Adoption of Certified Electronic Health Record Systems and Electronic Information Sharing in Physician Offices: United States, 2013 and 2014. NCHS Data Brief. 2016;(236):1–8.

  29. Palojoki S, Makela M, Lehtonen L, Saranto K. An analysis of electronic health record-related patient safety incidents. Health Informatics J. 2017. https://0-doi-org.brum.beds.ac.uk/10.1177/1460458216631072.

    Article  PubMed  Google Scholar 

  30. Vishwanath A, Scamurra SD. Barriers to the adoption of electronic health records: using concept mapping to develop a comprehensive empirical model. Health Informatics J. 2007. https://0-doi-org.brum.beds.ac.uk/10.1177/1460458207076468.

    Article  PubMed  Google Scholar 

  31. Takian A, Petrakaki D, Cornford T, Sheikh A, Barber N, National NHSCRSET. Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems. BMC Health Serv Res. 2012. https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6963-12-105.

  32. Heringa M, Floor-Schreudering A, De Smet P, Bouvy ML. Clinical decision support and optional point of care testing of renal function for safe use of antibiotics in elderly patients: a retrospective study in community pharmacy practice. Drugs Aging. 2017. https://0-doi-org.brum.beds.ac.uk/10.1007/s40266-017-0497-z.

    Article  Google Scholar 

  33. Kane-Gill SL, Achanta A, Kellum JA, Handler SM. Clinical decision support for drug related events: moving towards better prevention. World J Crit Care Med. 2016. https://0-doi-org.brum.beds.ac.uk/10.5492/wjccm.v5.i4.204.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Legat L, Van Laere S, Nyssen M, Steurbaut S, Dupont AG, Cornu P. Clinical decision support Systems for Drug Allergy Checking: systematic review. J Med Internet Res. 2018. https://0-doi-org.brum.beds.ac.uk/10.2196/jmir.8206.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Castaneda C, Nalley K, Mannion C, Bhattacharyya P, Blake P, Pecora A, Goy A, Suh KS. Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. J Clin Bioinforma. 2015. https://0-doi-org.brum.beds.ac.uk/10.1186/s13336-015-0019-3.

  36. Blum D, Raj SX, Oberholzer R, Riphagen II, Strasser F, Kaasa S, Euro Impact EIMPCRT. Computer-Based Clinical Decision Support Systems and Patient-Reported Outcomes: A Systematic Review. Patient. 2015. https://0-doi-org.brum.beds.ac.uk/10.1007/s40271-014-0100-1.

    Article  Google Scholar 

  37. Meulendijk MC, Spruit MR, Drenth-van Maanen AC, Numans ME, Brinkkemper S, Jansen PA, Knol W. Computerized decision support improves medication review effectiveness: an experiment evaluating the STRIP Assistant's usability. Drugs Aging. 2015. https://0-doi-org.brum.beds.ac.uk/10.1007/s40266-015-0270-0.

    Article  Google Scholar 

  38. Prasert V, Shono A, Chanjaruporn F, Ploylearmsang C, Boonnan K, Khampetdee A, Akazawa M. Effect of a computerized decision support system on potentially inappropriate medication prescriptions for elderly patients in Thailand. J Eval Clin Pract. 2018. https://0-doi-org.brum.beds.ac.uk/10.1111/jep.13065.

    Article  PubMed  Google Scholar 

  39. Jia P, Zhang L, Chen J, Zhao P, Zhang M. The effects of clinical decision support systems on medication safety: an overview. PLoS One. 2016. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pone.0167683.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  40. Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003. https://0-doi-org.brum.beds.ac.uk/10.1001/archinte.163.12.1409.

    Article  PubMed  Google Scholar 

  41. Shawahna R, Rahman NU, Ahmad M, Debray M, Yliperttula M, Decleves X. Electronic prescribing reduces prescribing error in public hospitals. J Clin Nurs. 2011. https://0-doi-org.brum.beds.ac.uk/10.1111/j.1365-2702.2011.03714.x.

    Article  PubMed  Google Scholar 

  42. Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc. 2017. https://0-doi-org.brum.beds.ac.uk/10.1093/jamia/ocw056.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Pontefract SK, Hodson J, Slee A, Shah S, Girling AJ, Williams R, Sheikh A, Coleman JJ. Impact of a commercial order entry system on prescribing errors amenable to computerised decision support in the hospital setting: a prospective pre-post study. BMJ Qual Saf. 2018. https://0-doi-org.brum.beds.ac.uk/10.1136/bmjqs-2017-007135.

    Article  Google Scholar 

  44. Clarke A, Adamson J, Watt I, Sheard L, Cairns P, Wright J. The impact of electronic records on patient safety: a qualitative study. BMC Med Inform Decis Mak. 2016. https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-016-0299-y.

  45. Reis WC, Bonetti AF, Bottacin WE, Reis AS Jr, Souza TT, Pontarolo R, Correr CJ, Fernandez-Llimos F. Impact on process results of clinical decision support systems (CDSSs) applied to medication use: overview of systematic reviews. Pharm Pract (Granada). 2017. https://0-doi-org.brum.beds.ac.uk/10.18549/PharmPract.2017.04.1036.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Moja L, Passardi A, Capobussi M, Banzi R, Ruggiero F, Kwag K, Liberati EG, Mangia M, Kunnamo I, Cinquini M, et al. Implementing an evidence-based computerized decision support system linked to electronic health records to improve care for cancer patients: the ONCO-CODES study protocol for a randomized controlled trial. Implement Sci. 2016. https://0-doi-org.brum.beds.ac.uk/10.1186/s13012-016-0514-3.

  47. Ghitza UE, Sparenborg S, Tai B. Improving drug abuse treatment delivery through adoption of harmonized electronic health record systems. Subst Abus Rehabil. 2011. https://0-doi-org.brum.beds.ac.uk/10.2147/SAR.S23030.

  48. Belard A, Buchman T, Forsberg J, Potter BK, Dente CJ, Kirk A, Elster E. Precision diagnosis: a view of the clinical decision support systems (CDSS) landscape through the lens of critical care. J Clin Monit Comput. 2017. https://0-doi-org.brum.beds.ac.uk/10.1007/s10877-016-9849-1.

    Article  PubMed  Google Scholar 

  49. Tilson H, Hines LE, McEvoy G, Weinstein DM, Hansten PD, Matuszewski K, le Comte M, Higby-Baker S, Hanlon JT, Pezzullo L, et al. Recommendations for selecting drug-drug interactions for clinical decision support. Am J Health Syst Pharm. 2016. https://0-doi-org.brum.beds.ac.uk/10.2146/ajhp150565.

    Article  PubMed  Google Scholar 

  50. Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL, Chaffee BW, Hartman C, Tamis V, Galbreth B, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc. 2015. https://0-doi-org.brum.beds.ac.uk/10.1093/jamia/ocv011.

    Article  PubMed  Google Scholar 

  51. Nurek M, Kostopoulou O, Delaney BC, Esmail A. Reducing diagnostic errors in primary care. A systematic meta-review of computerized diagnostic decision support systems by the LINNEAUS collaboration on patient safety in primary care. Eur J Gen Pract. 2015. https://0-doi-org.brum.beds.ac.uk/10.3109/13814788.2015.1043123.

    Article  PubMed  Google Scholar 

  52. Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records Interface design: a review of recent literature and guideline formulation. Hum Factors. 2015. https://0-doi-org.brum.beds.ac.uk/10.1177/0018720815576827.

    Article  Google Scholar 

  53. Pearson SA, Moxey A, Robertson J, Hains I, Williamson M, Reeve J, Newby D. Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990-2007). BMC Health Serv Res. 2009. https://0-doi-org.brum.beds.ac.uk/10.1186/1472-6963-9-154.

  54. Avery AJ, Savelyich BS, Sheikh A, Cantrill J, Morris CJ, Fernando B, Bainbridge M, Horsfield P, Teasdale S. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study. Inform Prim Care. 2005. https://0-doi-org.brum.beds.ac.uk/10.14236/jhi.v13i1.575.

    Article  Google Scholar 

  55. Lee S. Features of computerized clinical decision support systems supportive of nursing practice: a literature review. Comput Inform Nurs. 2013. https://0-doi-org.brum.beds.ac.uk/10.1097/01.NCN.0000432127.99644.25.

    PubMed  Google Scholar 

  56. Lau G, Ho J, Lin S, Yeoh K, Wan T, Hodgkinson M. Patient and clinician perspectives of an integrated electronic medication prescribing and dispensing system: a qualitative study at a multisite Australian hospital network. Health Inf Manag. 2019. https://0-doi-org.brum.beds.ac.uk/10.1177/1833358317720601.

    Article  PubMed  Google Scholar 

  57. Wright A, Ai A, Ash J, Wiesen JF, Hickman TT, Aaron S, McEvoy D, Borkowsky S, Dissanayake PI, Embi P, et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc. 2018. https://0-doi-org.brum.beds.ac.uk/10.1093/jamia/ocx106.

    Article  PubMed Central  Google Scholar 

  58. Cassar Flores A, Marshall S, Cordina M. Use of the Delphi technique to determine safety features to be included in a neonatal and paediatric prescription chart. Int J Clin Pharm. 2014. https://0-doi-org.brum.beds.ac.uk/10.1007/s11096-014-0014-y.

    Article  CAS  PubMed  Google Scholar 

  59. International Organisation for Standardisation. 18308: 2011 Health Informatics—requirements for an electronic health record architecture. https://www.iso.org/standard/52823.html.

  60. Jha AK, DesRoches CM, Kralovec PD, Joshi MS. A progress report on electronic health records in US hospitals. Health affairs. 2010;29(10):1951-7.

    Article  PubMed  Google Scholar 

  61. Sittig DF, Longhurst CA, Russo E, Singh H. Electronic Health Record Features, Functions, and Privileges That Clinicians Need to Provide Safe and Effective Care for Adults and Children. In: Weaver CA, Ball MJ, Kim GR, Kiel JM, editors. Healthcare Information Management Systems: Cases, Strategies, and Solutions. Cham: Springer International Publishing; 2016. p. 21–38.

    Chapter  Google Scholar 

  62. Spyropoulos AC, Viscusi A, Singhal N, Gilleylen J, Kouides P, Howard M, Rudd K, Ansell J, Triller DM. Features of electronic health records necessary for the delivery of optimized anticoagulant therapy: consensus of the EHR task force of the New York state anticoagulation coalition. Ann Pharmacother. 2015. https://0-doi-org.brum.beds.ac.uk/10.1177/1060028014555176.

    Article  PubMed  Google Scholar 

  63. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007. https://0-doi-org.brum.beds.ac.uk/10.1093/intqhc/mzm042.

    Article  PubMed  Google Scholar 

  64. Shawahna R, Qiblawi S, Ghanayem H. Which benefits and harms of using fenugreek as a Galactogogue need to be discussed during clinical consultations? A Delphi study among breastfeeding women, gynecologists, pediatricians, family physicians, lactation consultants, and pharmacists. Evid Based Complement Alternat Med. 2018. https://0-doi-org.brum.beds.ac.uk/10.1155/2018/2418673.

    Google Scholar 

  65. Shawahna R. Which information on women's issues in epilepsy does a community pharmacist need to know? A Delphi consensus study. Epilepsy Behav. 2017. https://0-doi-org.brum.beds.ac.uk/10.1016/j.yebeh.2017.09.026.

    Article  Google Scholar 

  66. Shawahna R, Haddad A, Khawaja B, Raie R, Zaneen S, Edais T. Medication dispensing errors in Palestinian community pharmacy practice: a formal consensus using the Delphi technique. Int J Clin Pharm. 2016. https://0-doi-org.brum.beds.ac.uk/10.1007/s11096-016-0338-x.

    Article  PubMed  Google Scholar 

  67. Shawahna R, Masri D, Al-Gharabeh R, Deek R, Al-Thayba L, Halaweh M. Medication administration errors from a nursing viewpoint: a formal consensus of definition and scenarios using a Delphi technique. J Clin Nurs. 2016. https://0-doi-org.brum.beds.ac.uk/10.1111/jocn.13062.

    Article  PubMed  Google Scholar 

  68. Shawahna R, Odeh M, Jawabreh M. Factors promoting clinical inertia in caring for patients with dyslipidemia: a consensual study among clinicians who provide healthcare to patients with dyslipidemia. J Natl Med Assoc. 2019. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jnma.2018.04.002.

    Article  PubMed  Google Scholar 

  69. Bishop PA, Herron RL. Use and misuse of the Likert item responses and other ordinal measures. Int J Exerc Sci. 2015;8(3):297-302.

  70. Birko S, Dove ES, Özdemir V. Evaluation of nine consensus indices in Delphi foresight research and their dependency on Delphi survey characteristics: a simulation study and debate on Delphi design and interpretation. PLoS One. 2015. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pone.0135162.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  71. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf Manag. 2004.

  72. Duffield C. The Delphi technique: a comparison of results obtained using two expert panels. Int J Nurs Stud. 1993;30(3):227-37.

    Article  CAS  PubMed  Google Scholar 

  73. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000. https://0-doi-org.brum.beds.ac.uk/10.1046/j.1365-2648.2000.t01-1-01567.x.

    CAS  PubMed  Google Scholar 

  74. Shawahna R, Taha A. Which potential harms and benefits of using ginger in the management of nausea and vomiting of pregnancy should be addressed? A consensual study among pregnant women and gynecologists. BMC Complement Altern Med. 2017. https://0-doi-org.brum.beds.ac.uk/10.1186/s12906-017-1717-0.

  75. Shawahna R, Zyoud A, Dwikat J, El-Helo M, Yacoub B, Hilal H. Breast Milk Lead levels in 3 major regions of the West Bank of Palestine. J Hum Lact. 2016. https://0-doi-org.brum.beds.ac.uk/10.1177/0890334416646566.

    Article  PubMed  Google Scholar 

  76. Ghaleb MA, Barber N, Dean Franklin B, Wong IC. What constitutes a prescribing error in paediatrics? Qual Saf Health Care. 2005. https://0-doi-org.brum.beds.ac.uk/10.1136/qshc.2005.013797.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  77. Dean B, Barber N, Schachter M. What is a prescribing error? Qual health care; 2000;9(4):232-7.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  78. Page A, Potter K, Clifford R, McLachlan A, Etherton-Beer C. Prescribing for Australians living with dementia: study protocol using the Delphi technique. BMJ Open. 2015. https://0-doi-org.brum.beds.ac.uk/10.1136/bmjopen-2015-008048.

    Article  PubMed  PubMed Central  Google Scholar 

  79. McKenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs. 1994. https://0-doi-org.brum.beds.ac.uk/10.1046/j.1365-2648.2000.t01-1-01567.x.

    CAS  PubMed  Google Scholar 

  80. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pone.0020476.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  81. Junger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017. https://0-doi-org.brum.beds.ac.uk/10.1177/0269216317690685.

    Article  PubMed  Google Scholar 

  82. De Meyer D, Kottner J, Beele H, Schmitt J, Lange T, Van Hecke A, Verhaeghe S, Beeckman D. Delphi procedure in Core outcome set development: rating scale and consensus criteria determined outcome selection. J Clin Epidemiol. 2019. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jclinepi.2019.03.011.

    Article  PubMed  Google Scholar 

  83. Maruthur NM, Joy SM, Dolan JG, Shihab HM, Singh S. Use of the analytic hierarchy process for medication decision-making in type 2 diabetes. PLoS One. 2015. https://0-doi-org.brum.beds.ac.uk/10.1371/journal.pone.0126625.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  84. Hummel MJ, Volz F, van Manen JG, Danner M, Dintsios CM, Ijzerman MJ, Gerber A. Using the analytic hierarchy process to elicit patient preferences: prioritizing multiple outcome measures of antidepressant drug treatment. Patient. 2012. https://0-doi-org.brum.beds.ac.uk/10.2165/11635240-000000000-00000.

    Article  Google Scholar 

  85. Shawahna R. Combining and using the Utrecht method and the analytic hierarchy process to facilitate professional and ethical deliberation and decision making in complementary and alternative medicine: a case study among a panel of stakeholders. Evid Based Complement Alternat Med. 2018. https://0-doi-org.brum.beds.ac.uk/10.1155/2018/2315938.

    Google Scholar 

  86. Saaty TL: The analytic hierarchy process: planning, priority setting, resource allocation: McGraw-hill international book company; 1980.

  87. Robinson P, Lowe J. Literature reviews vs systematic reviews. Aust N Z J Public Health. 2015. https://0-doi-org.brum.beds.ac.uk/10.1111/1753-6405.12393.

    Article  PubMed  Google Scholar 

  88. Ferguson ND, Davis AM, Slutsky AS, Stewart TE. Development of a clinical definition for acute respiratory distress syndrome using the Delphi technique. J Crit Care. 2005. https://0-doi-org.brum.beds.ac.uk/10.1016/j.jcrc.2005.03.001.

    Article  PubMed  Google Scholar 

  89. Lack JA, Stuart-Taylor ME. Calculation of drug dosage and body surface area of children. Br J Anaesth. 1997. https://0-doi-org.brum.beds.ac.uk/10.1093/bja/78.5.601.

    Article  CAS  PubMed  Google Scholar 

  90. Bernius M, Thibodeau B, Jones A, Clothier B, Witting M. Prevention of pediatric drug calculation errors by prehospital care providers. Prehosp Emerg Care. 2008. https://0-doi-org.brum.beds.ac.uk/10.1080/10903120802290752.

    Article  PubMed  Google Scholar 

  91. Rishoej RM, Almarsdottir AB, Thybo Christesen H, Hallas J, Juel KL. Identifying and assessing potential harm of medication errors and potentially unsafe medication practices in paediatric hospital settings: a field study. Ther Adv Drug Saf. 2018. https://0-doi-org.brum.beds.ac.uk/10.1177/2042098618781521.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

All participants are thanked for their participation is the different stages of the study. An-Najah National University is acknowledged for making this study possible.

Funding

Not applicable.

Author information

Authors and Affiliations

Authors

Contributions

RS conceived the study. RS collected the data, performed data analysis and wrote the manuscript. The author read and approved the final manuscript.

Corresponding author

Correspondence to Ramzi Shawahna.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Institutional Review Board (IRB) of An-Najah National University. The panelists understood that the Delphi technique was a semi-anonymous technique in which the identity of the panelist is known to the investigator while the panelist remains anonymous to the rest of the panel members. Scores of the panelists weighed equally in the analysis. All participants provided written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Adherence to PRISMA guidelines.

Additional file 2.

Adherence to COnsolidated criteria for REporting Qualitative research (COREQ) Checklist.

Additional file 3.

Adherence to Conducting and REporting of DElphi Studies (CREDES) guidelines.

Additional file 4: Tables S1-S9.

show multiple comparisons of importance weights items in each category.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shawahna, R. Merits, features, and desiderata to be considered when developing electronic health records with embedded clinical decision support systems in Palestinian hospitals: a consensus study. BMC Med Inform Decis Mak 19, 216 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-019-0928-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-019-0928-3

Keywords