Testing SOAR Tools in Use

Modern security operation centers (SOCs) rely on operators and a tapestry of logging and alerting tools with large scale collection and query abilities. SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information and then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a new technology that promise to collect, filter, and display needed data; automate common tasks that require SOC analysts' time; facilitate SOC collaboration; and, improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and understand them in use. In this paper, we design and administer the first hands-on user study of SOAR tools, involving 24 participants and 6 commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools and a methodology for testing them. We describe configuration of the test environment in a cyber range, including network, user, and threat emulation; a full SOC tool suite; and creation of artifacts allowing multiple representative investigation scenarios to permit testing. We present the first research results on SOAR tools. We found that SOAR configuration is critical, as it involves creative design for data display and automation. We found that SOAR tools increased efficiency and reduced context switching during investigations, although ticket accuracy and completeness (indicating investigation quality) decreased with SOAR use. Our findings indicated that user preferences are slightly negatively correlated with their performance with the tool; overautomation was a concern of senior analysts, and SOAR tools that balanced automation with assisting a user to make decisions were preferred

The Effect of Automated Alerts on Provider Ordering Behavior in an Outpatient Setting

BACKGROUND: Computerized order entry systems have the potential to prevent medication errors and decrease adverse drug events with the use of clinical-decision support systems presenting alerts to providers. Despite the large volume of medications prescribed in the outpatient setting, few studies have assessed the impact of automated alerts on medication errors related to drug–laboratory interactions in an outpatient primary-care setting. METHODS AND FINDINGS: A primary-care clinic in an integrated safety net institution was the setting for the study. In collaboration with commercial information technology vendors, rules were developed to address a set of drug–laboratory interactions. All patients seen in the clinic during the study period were eligible for the intervention. As providers ordered medications on a computer, an alert was displayed if a relevant drug–laboratory interaction existed. Comparisons were made between baseline and postintervention time periods. Provider ordering behavior was monitored focusing on the number of medication orders not completed and the number of rule-associated laboratory test orders initiated after alert display. Adverse drug events were assessed by doing a random sample of chart reviews using the Naranjo scoring scale. The rule processed 16,291 times during the study period on all possible medication orders: 7,017 during the pre-intervention period and 9,274 during the postintervention period. During the postintervention period, an alert was displayed for 11.8% (1,093 out of 9,274) of the times the rule processed, with 5.6% for only “missing laboratory values,” 6.0% for only “abnormal laboratory values,” and 0.2% for both types of alerts. Focusing on 18 high-volume and high-risk medications revealed a significant increase in the percentage of time the provider stopped the ordering process and did not complete the medication order when an alert for an abnormal rule-associated laboratory result was displayed (5.6% vs. 10.9%, p = 0.03, Generalized Estimating Equations test). The provider also increased ordering of the rule-associated laboratory test when an alert was displayed (39% at baseline vs. 51% during post intervention, p < 0.001). There was a non-statistically significant difference towards less “definite” or “probable” adverse drug events defined by Naranjo scoring (10.3% at baseline vs. 4.3% during postintervention, p = 0.23). CONCLUSION: Providers will adhere to alerts and will use this information to improve patient care. Specifically, in response to drug–laboratory interaction alerts, providers will significantly increase the ordering of appropriate laboratory tests. There may be a concomitant change in adverse drug events that would require a larger study to confirm. Implementation of rules technology to prevent medication errors could be an effective tool for reducing medication errors in an outpatient setting

Chronology of prescribing error during the hospital stay and prediction of pharmacist's alerts overriding: a prospective analysis

<p>Abstract</p> <p>Background</p> <p>Drug prescribing errors are frequent in the hospital setting and pharmacists play an important role in detection of these errors. The objectives of this study are (1) to describe the drug prescribing errors rate during the patient's stay, (2) to find which characteristics for a prescribing error are the most predictive of their reproduction the next day despite pharmacist's alert (<it>i.e</it>. override the alert).</p> <p>Methods</p> <p>We prospectively collected all medication order lines and prescribing errors during 18 days in 7 medical wards' using computerized physician order entry. We described and modelled the errors rate according to the chronology of hospital stay. We performed a classification and regression tree analysis to find which characteristics of alerts were predictive of their overriding (<it>i.e</it>. prescribing error repeated).</p> <p>Results</p> <p>12 533 order lines were reviewed, 117 errors (errors rate 0.9%) were observed and 51% of these errors occurred on the first day of the hospital stay. The risk of a prescribing error decreased over time. 52% of the alerts were overridden (<it>i.e </it>error uncorrected by prescribers on the following day. Drug omissions were the most frequently taken into account by prescribers. The classification and regression tree analysis showed that overriding pharmacist's alerts is first related to the ward of the prescriber and then to either Anatomical Therapeutic Chemical class of the drug or the type of error.</p> <p>Conclusions</p> <p>Since 51% of prescribing errors occurred on the first day of stay, pharmacist should concentrate his analysis of drug prescriptions on this day. The difference of overriding behavior between wards and according drug Anatomical Therapeutic Chemical class or type of error could also guide the validation tasks and programming of electronic alerts.</p

Case Report: Activity Diagrams for Integrating Electronic Prescribing Tools into Clinical Workflow

To facilitate the future implementation of an electronic prescribing system, this case study modeled prescription management processes in various primary care settings. The Vanderbilt e-prescribing design team conducted initial interviews with clinic managers, physicians and nurses, and then represented the sequences of steps carried out to complete prescriptions in activity diagrams. The diagrams covered outpatient prescribing for patients during a clinic visit and between clinic visits. Practice size, practice setting, and practice specialty type influenced the prescribing processes used. The model developed may be useful to others engaged in building or tailoring an e-prescribing system to meet the specific workflows of various clinic settings