This was my capstone project for my Master’s degree back in 2019.
Abstract
Always Be Careful Insurance (ABC Ins.) identified the need to implement cybersecurity tools and initiatives to better align with common security standards and to lower mean-time-to-detection (MTTD) and mean-time-to-recovery (MTTR) while controlling costs. The security team needed to find ways to leverage the tools employed by attackers and leverage automation to close gaps, communicate forensics data, and recover from attacks.
Extending a Security Information and Event Management (SIEM) platform, concepts of Security Orchestration, Automation, and Response (SOAR) were used to improve security detection and response. AT&T Cybersecurity’s AlienVault USM Anywhere, which leverages Alien Labs, Open Threat Exchange (OTX), and other security tools were used to implement these concepts.
The plan was be broken into 3 phases, over 30 days. The first phase ran for ten days as an overall planning process to review security standards in the NIST Cybersecurity Framework (CSF), in SP 800-53. The product of this phase was used to guide a unified plan and ensure tools aligned with security controls. Phase two consisted of five Agile sprints, lasting one week each. The sprints enumerated and collected information about data sources. The security team developed and communicated steps for the curation and identification of actionable data to improve collaboration efforts within the organization. Each sprint’s goal was to create a more unified security ecosystem by closing identified gaps for each task with automation or orchestration mechanisms.
The project’s goals were to improve mapping to security frameworks and controls, reducing the mean-time-to-detection (MTTD) of threats and mean-time-to-recovery (MTTR) of incidents controlled costs. By applying automation to security information and event management, the organization was better able to handle the growth in volume and of the sophistication of cyber threats now and in the future.
Table of Contents
Organization of the Capstone Report 12
Detailed and Functional Requirements 17
Quality Assurance Approach. 32
Strategy for the Implementation. 34
Quantitative and Qualitative Risks 37
Post Implementation Support and Issues 42
Post Implementation Support 42
Post Implementation Support Resources 43
Conclusion, Outcomes, and Reflection. 43
Appendix A: Ansible Playbook. 49
Appendix B: Alarm Configuration. 50
Appendix C: Response Action Rule. 51
Appendix D: Automation Deliverables Document: Task Audit 52
Appendix E: Cost Analysis Detail 56
Appendix F: Task Requirements 57
Appendix G: Controls Mapping. 59
Appendix G: Dependency Breakdown. 60
Appendix H: Post-Implementation Support 63
Introduction
The sophistication of attacks on information systems has continued to evolve at an ever-increasing pace. With this comes the need to go beyond traditional security practices and add the processing intelligence and speed of automation and machine learning. Institutional pushback hinders organizations seeking to implement security frameworks while threat actors are pushing the boundaries of automated attacks by leveraging machine learning and automation to find and exploit targets. Rob Joyce, a former White House Cybersecurity Coordinator, suggested that attackers will often “know networks better than the people who designed and run them.” (Thomson, 2016) Security teams must leverage the same tools to have any hope of stemming this tide of threats. “Incident Response” must be able to leverage crowdsourced threat databases, intelligent automation, and sophisticated orchestration methods to close gaps, ingest forensics data and recover promptly from attacks.
This study lasted for 30 days. It began with a review of how the security event management and automation mapped to the NIST cybersecurity framework and controls. It contained a review of the implementation of a Security Information and Event Management, or SIEM, system which utilized concepts initially suggested by Gartner known as Security Orchestration, Automation, and Response, or SOAR. (Lawson, C et al., 2017) Implementation of AT&T Cybersecurity’s USM Anywhere as a SIEM, which leverages Alien Labs, Open Threat Exchange (OTX), and other security tools, was used to apply these concepts. Mean-time-to-detection (MTTD) improved by identifying and configuring processes to provide automatic enumeration and collection of data streams into the security platform. It then curated the data to provide actionable information and communicated this to improve collaboration efforts in the organization. A unified ecosystem closed gaps in spaces between existing security tools. Mean-time-to-recovery (MTTR) decreased through the creation of playbooks and the ability to update security control points to act on identified threats in real-time.
The overall project measured four key factors. 1) how well the solution mapped to the NIST Cybersecurity framework and controls. 2) reduction in the Mean-Time-To-Detection for vulnerabilities and exploitations. 3) reduction in the Mean-Time-To-Recovery for remediating vulnerabilities and acting on exploitation detection, and 4) the cost to implement and operate automation and orchestration in a SIEM system was lower than hiring additional security staff.
Project scope
This project reviewed the business cost and operational effectiveness of implementing a SIEM that is augmented by automation and orchestration capabilities, through both built-in tools and through additional automation solutions. The security team determined that not all processes would be effective use cases for automation. The project team met and developed a list of five use-cases to narrow the project scope to ensure effective results in a timely fashion.
The teams detailed each of these five tasks into procedures called playbooks. A financial analysis reviewed the cost of hiring additional analysists or engineers to perform these tasks, which would be automated and compared it to the amount for the existing security team to receive adequate training to implement the SIEM and then to configure, monitor and respond to the automation augmentation.
Defense of the Solution
Response to security events has grown to a level that automated processing must be introduced to handle the quantity and to respond rapidly. Sources predict that there will be 64 billion IoT devices by 2025. That is an increase from 10 billion in 2018. (Newman 2019). IoT is only one source for security professionals to monitor. Others include infrastructure devices, security devices, server logs, web servers, authentication servers, hypervisors, dockers/containers, SAN infrastructure, applications, and client workstations. With each source generating many events, the volume of data is beyond a human’s ability to analyze. (Skeen 2014) Also, that data must be filtered, normalized, cross-linked with a matrix of events from multiple sources before narrowing down to credible security concerns. And even when the events have been isolated, generating response actions that are quick enough to stop an attack is difficult, especially since attackers are already utilizing automated mechanisms to identify and exploit targets. Essential business requirements that drive the need for automation and orchestration include control of costs, mitigation of risk, need to develop more Agile business units.
Control of Costs
An increase in threats and their associated complexities drives a need to increase security management. This increase unavoidably incurs business costs, regardless of the solution. As the costs of failing to protect an organization’s information systems increase, cybersecurity budgets quickly increase. Organizations need endpoint protection, mobile protection, network security, data storage protection, control access management, proper vulnerability, and patch management, security awareness training, support for security audits, and disaster recovery plans. According to an Altimeter survey, business leaders consider cybersecurity to be second only to cloud as an investment priority. And that IT security has outpaced general IT spending. (CSPI 2019) As these costs inevitably swell, the organization should consider any measures which can control the costs while improving the efficiency and success of security programs. Deploying SOAR mechanisms into security operations has the potential to avoid the high costs of increased staff by allowing existing team to manage more extensive portfolios more effectively
Mitigation of Risk
Risk is the measure of the likely occurrence and the severity of failure for cyber incidents. Risk assessments and decision-making drive organizations. Improving the metrics of mean time to detection and recovery decreases the likelihood of compromise and minimizes the severity of incidents through automation mechanisms shorting the response gap.
Business Agility
Organizations are under pressure to improve operating costs and income potential. Improving the efficiency and effectiveness of existing staff and organizational architecture requires an examination of how processes can be improved. Agile project management is one example. Allowing teams to be created in response to business needs and carry through projects on short iterative cycles supports a more flexible business structure. Security needs to support this model. Security orchestration, automation, and recovery mechanics reduce the risks in both areas. On the first hand, automation can scan for vulnerabilities continuously and process event logs to detect ongoing malicious activities and react by putting barriers and controls in place. This lowers the expected rate of occurrence by reducing vulnerabilities and the expected impact of incidents by more quickly responding to and initiating recovery activities. According to Technologent, SOAR has three technology focuses; Threat and Vulnerability Management, Security Incident Response, and Security Operations Automation. (Technologent 2019) Vulnerability management and incident response allow security teams to handle monitoring and updates more quickly. The real impact on business agility comes from operations automation. The orchestration of workflows, like server deployments for development teams using Ansible Tower, with security configurations integrated into playbooks allows Agile teams to form and deploy development, non-productions, and production environments more quickly.
Methodology Justification
Security staff had limited time and resources to handle security events. Automating event management broadened the security team’s “field of vision” across expanding information systems and improved their capacity to ingest more massive streams of logs. Automated tools correlated events and filtered out unimportant ones allowing key events to surface for review by the security analyst. Responses to events and alarms were pre-coded into ‘playbooks,’ which ran automatically in response to the chosen five tasks; blocking phishing attacks, remediating unquoted registry keys paths, reporting on unusual logins, automatically creating services tickets for found indicators of compromise and isolating endpoints when needed. These types of automation solutions answered the business needs for a more effective security program. The following methods were options for addressing the need to close the gap between expanding threats and an organization’s ability to manage these threats. Ultimately, a chosen method needed to improve the reduction of MTTD, reduction of MTTR, and costs to implement.
The first possible method was to continue to hire additional security staff to monitor, analyze, patch, and respond to security vulnerabilities, either through additional internal team members or expanding to use services like SOC-as-a-service. This approach had two significant challenges. In the first place, security positions are in high demand. According to estimates, there will be 3.5 million unfilled positions in cybersecurity in 2021. (NeSmith, 2018). As a result, finding new talent was increasingly difficult. Also, new hires are often underqualified and lack experience.
The second method was to alter information system designs to reduce threat exposure to a point manageable by existing security staff. However, competition drives business objectives and a need to grow market share. There was considerable pressure to move into new and immerging technology like cloud computing, “everything as a service,” 5G, artificial intelligence, and others. Each introduced new challenges for security staff that required training and time. These factors not only hindered landscape reduction, but they encouraged continued expansion.
Third, leverage automation and orchestration along with security event management to use existing security staff more effectively. These tools could handle everyday tasks, like event monitoring and analysis. The team could focus efforts on more complicated issues. This method used the new and immerging technologies as part of the solution; automation, artificial intelligence, cloud computing and storage with a hosted SIEM, and segmentation orchestration for networks and mobile devices moving to higher 5G speed reliance.
Organization of the Capstone Report
The following consists of the results of the 30-day project. It began with brainstorming a list of all security tasks and choosing a specific set that would provide clear and concise procedures that could be automated and limit the scope of this project to keep results attainable in a reasonable time frame. The team expanded each task into a process for manual execution, timed each one, and then outlined requirements to automate them. This section contained end-user needs. It identified the gaps that existed between the identified problem (resource scarcity and time to resolution) and the end-result of task automation.
The next section reviewed the project plan and included details on the design, development, testing, and implementation of the automation. The project teams identified all critical stages from initial audit through to post-implementation support, which included the methodology and approach used at each stage. Justifications were given for each aspect of the solution, contrasting alternative solution strategies.
The Agile teams reviewed automation and orchestration implementation with specific attention given to the hardware, software and other technical details. The team provided expected outputs for testing and quality assurance and described how the five task automation implementation would be verified, and effectiveness tested.
To wrap-up the project review, risk factors were identified, which may hinder success or adversely affect project implementation costs. Then details were provided for mitigation strategies for each risk factor. This led to a review of how support was planned for post-project maintenance and continued viability.
And finally, project results were reviewed for post-implementation considerations. Project teams examined each stage of the project and determined how successful each step had been and gave recommendations for additional future task automation and orchestration projects.
Systems and Process Audit
The first stage of the project began with evaluating the existing systems and business processes to identify gaps and solution requirements.
Audit Details
The audit contained two layers – business and technology. The business section reviews the processes employed by the security team for reaching improved mean-time-to-detection and mean-time-to-recovery for vulnerabilities and compromise. This section examined the policies, procedures, guidelines, and best practices followed. The second section focused on the technology tools and details of each stage of the processes.
Business Layer – the ABC Insurance security team, audited each of the five tasks to determine how it affected its related metric (MTTD or MTTR) in the manual process. The findings from the audit are included in the ‘Automation Deliverables Document: Task Audit’ (see appendix D)
Problem Statement
There is a significant delay due to waiting times for analysts to focus on detection activities and for service desk technicians, and security engineers, to focus on remediation activities. This delay allows threats to persist, which increases the likelihood of compromise.
Problem Causes
The following table outlines the key factors and causes of poor MTTD and MTTR for each task.
| Task | Contributing Problem Factor |
| A. Blocking Phishing Attacks | 1. The time it takes for the analyst to remediate the threat of one phishing email can be up to 55 minutes. 2. Manual access for multiple administrative interfaces to update blacklists and remove offending email is cumbersome. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | 1. Vendor patches often reintroduce unquoted paths. 2. The remediation action is simple but must be done frequently. |
| C. Reporting Logins from Unusual Devices | 1. The asset inventory is static and disconnected from security tools 2. The process of checking for unusual logins is time-consuming. 3. Manual processes introduce the possibility of human error. |
| D. Creating Service Ticket for Indicators of Compromise | 1. The process of identifying an IoC and then opening a ticket requires manual intervention, which is time-consuming. 2. This process is additionally delated by the meantime to detect an IoC and be monitored by the security team |
| E. Isolating Compromised Endpoints | 1. The service desk technician workload determines how quickly they can respond to the security team request. 2. Locating and manually disconnected the LAN cable is time-consuming. |
Business Impacts
ABC Insurance’s primary business activity is the issuing and management of insurance policies. Insurance operations include marketing, underwriting, administration, actuarial analysis and investments, reinsurance, legal and regulatory issues, claims adjusting, and management. IT operations support all these functions, of which security is a component. Policy information, claims data, legal and regulatory data, and administration and services data are processed and stored by information systems. These activities are impacted by the cost in resources to monitor and manage threat risks, and then by the consequences of a failure to catch and remediate vulnerabilities in the information systems.
A failure to close these caps can lead to quantitative financial penalties for non-compliance and from payouts to cybercriminals for compromises like ransomware, or from qualitative costs in loss of customer confidence and sales as a result. Some organizations fail ever to recover. According to Inc.com, 60 percent of small businesses fail within six months of experiencing a cyber incident. (Galvin 2018)
All five identified tasks result in increased time investment to handle security actives but result in poor effectiveness in time it takes to detect and then to remediate threats. Therefore, the sensitive data is subject to more threats and for an extended time before being handled.
A choice to implement automation as a strategy over staff expansion will impact business cooperation in time and resources to select, deploy, and properly configure the orchestration tools, or as an expansion of an existing SIEM. Standard IT operations and security tasks will divert focus while implementing the solutions and mechanisms, and while current security staff is training.
Cost Analysis
Risk Analysis
Automation and orchestration introduce several risk factors.
- Misconfiguration can include the incorrect deployment of SIEM sensors or of log ingestion parameters, wrong orchestration rules, or zero-day threats that would not trigger an alert or action. This risk is considered ‘Low.’ (Low expected likelihood. Low expected impact if it occurs due to management oversight.)
- False positives can trigger response actions that could shut down client workstations and network segments or block external service access without cause, causing loss of business during the downtime. False negatives can allow for malicious actions. This risk is considered ‘Medium. (There is a high expected occurrence in the beginning as the automation and orchestration are configured and fine-tuned. The expected impact is Low as security staff will monitor and ensure a proper response.)
- SOAR implementation cost. This risk is considered ‘Low.’ (Initial examination shows the costs to expand existing tools is low. Additional, costs are evaluated as stages are implemented, allowing management oversight before charges are committed to.)
Detailed and Functional Requirements
Introducing SOAR principles to the chosen five tasks involved essential functional (end-user) and exact requirements. During the “envision” and “speculate” segments of team cycles, each task was reviewed, and functional and technical requirements established. Then teams reviewed the gaps for each task. These findings were incorporated into a ‘Task Requirements’ document. (See Appendix F)
Project Design
There was a first “compliance and task identification” phase. This foundational effort reviewed security processed for specific tasks that could be automated and that filled compliance framework requirement. After this phase, the security team implemented automation and orchestration mechanisms along an Agile project management process, with two Agile teams working on 1-week cycles each. The cycles consisted of the five segments; Envision, Speculate, Explore, Adapt, and Close.
Scope
Five tasks were identified for automation. Efforts were focused on how each step could be replaced with an automation mechanism, or if the entire task could be replaced with another process utilized machine learning and automation. Notably, the automation needed to shorten the time to detect a malicious event or recover from an incident while retaining or decreasing costs. Attention was given to ensure tangential tasks that did not specifically affect the automation objective of each task were separated from the project to avoid “scope creep”.
Assumptions
- Team Availability – Agile team members, will have sufficient time to devote to project objectives during the weekly cycles.
- Subject Matter Experts (SME) – the security and IT operations team have subject matter experts in the technical details needed to evaluate assigned automation tasks and implemented the technical requirements. Or they will have access to SMEs outside the Agile automation team.
- Tasks are capable of automation – the chosen five tasks can be augmented or replaced by automation or orchestration mechanisms
Project Phases
The initial phase spanned ten days. The security and IT operations team members reviewed all security processes and filtered it down to five specific tasks. Then each task was mapped to specific NIST controls found in SP 800-53 rev4 to ensure the automation efforts supported security framework guidance. This phase ended with deliverables that could be used in the Envision phase of each Agile team cycle. (See Appendix G for controls mapping.)
The following segments were followed during each of the 1-week sprints.
Envision – the assigned task was reviewed, and the goal of automating the playbook was established — additionally, stakeholders and participants in the functions where identified.
Speculate – this segment developed a time frame for the cycle, identified risks to the existing processes and future automation mechanisms, formalized objectives, and any costs where estimated.
Explore – each step was audited for technical detail, requirements gathered, and tasks assigned for daily work needed to achieve objectives. The technical plan was implemented. Additionally, quality assurance metrics were identified.
Adapt – objectives achieve in the Explore stage were reviewed for the level of success using the previously identified quality assurance metrics. Gaps in implementation work and the targets were identified and prepared for future iterations, which could cycle back to speculate and explore stages. If no gaps are found, then the process moved on to the Close segment.
Close – once objectives and the final goal of the Agile cycle is achieved; the team met to review each step and develop feedback on the work. Future “post-implementation” support needs were identified, and brainstorming was performed to identify future additional automation and orchestration tasks.
Timelines
Phase one involved task selection and control mapping. Phase two began with two Agile teams on 1-week Agile sprints following 5 segments each. Team 1 began week 2 with “Blocking Phishing Attacks” then moved to “Reporting Unusual Devices” in week 3. Team 2 began with “Remediating Unquoted Paths” in week 2, then moved to Creating Service Tickets” in week 3. A final team was created from select members of Team 1 to complete the last automation task in the 4th week, “Isolating Endpoints”. Week 5 was used to meet to review automation effectiveness and provide feedback from all team members.
| Week 1 | Week 2 (Cycle 1) | Week 3 (Cycle 2) | Week 4 (Cycle 3) | Week 5 | |
| Phase 1 – Compliance and Task Identification | Security Team meets with Project Manager to choose tasks to automate and map them to NIST Controls Milestone = Completion of “Controls Mapping” document. | ||||
| Phase 2 – Two Agile Teams are formed. Each team is assigned tasks to automate each week. Once the first four tasks are complete, Team 2 is discontinued, and Team 1 completes task five. Milestones = Each cycle most produce “Automation Deliverables Document: Task Audit”, “Cost Analysis Detail”, “Task Requirements”, “Dependency Breakdown”, and associated documentation. | Team 1 (A. Blocking Phishing Attacks) Team 2 (B. Remediating Reoccurring Unquoted Path Vulnerability) | ||||
| Team 1 (C. Reporting Logins from Unusual Devices) Team 2 (D. Creating Service Ticket for Indicators of Compromise) | |||||
| Team 1 (E. Isolating Compromised Endpoints) | |||||
| Phase 3 -Project Closure | Security Team, Service Desk Staff and IT Operations staff meet with project manager to review all project details and complete documentation. Milestone = Project is concluded when “Post-Implementation Support” document is complete. |
Dependencies
Most automation tasks are independent. The only common dependency requirement is staffing. The security and IT operation teams only have enough staff available to support two Agile teams running in a given week. As such, a given Agile team must reach objectives and properly closeout the project before moving on to another task automation effort.
During task review, it was realized that the 5th task ‘Isolating endpoints with high-risk compromise’ could not begin before the 4th task ‘Creating service tickets for found indicators of compromise’ had been completed successfully.
The Agile teams developed an outline of each task, breaking down the processes into smaller steps. This allowed Agile teams to identify necessary deliverable dependencies in the automation process required to move to the next task step. (See Appendix H for dependency breakdown).
Resource Requirements
There are general requirements for the SIEM/SOAR automation, which touches on all task automation project objectives.
- Hosting platform for SIEM application capable of ingesting needed log volume (approx. 350GB/month)
- IT staff (two Agile teams of a minimum of 4 persons each) able to devote four weeks of time to the automation project
Risk Factors
Each project cycle carries a possible risk. The following outlines each task risk and mitigation strategies for those risks. It is to be understood that ABC Insurance has taken a stance to err on the side of more secure. This provides an expectation of false positives, but this was determined by organization leadership to be preferred over false negatives leading to undesired vulnerability. In every case, an invalid positive notification mechanism and recovery path are provided.
Important Milestones
The project contains three phases. The NIST control mapping document serves as the milestone to close phase 1 and initiate the Agile team cycles. The second phase includes five Agile cycles. Each cycle is summarized below, including the deliverable, if any are produced.
Envision – The team gathers and is assigned a task to review.
Speculate – The team produces a document that audits existing processes, establishes timelines, and details the objective.
Explore and Adapt – The team develops the technical requirements and begins daily implementation tasks. Along the way, details recorded, which tie to quality assurance metrics.
Close – Automation is complete and functioning. Teams meet to review the work and produce a closure document.
Deliverables
Phase 1 milestones (deliverables): a) Chose Five Tasks for Automation, b) Task to Security Framework Mapping
Phase 2 – Agile Team milestones (deliverables) for each of the five cycles: a) Manual Task Playbook is written, b) Audit including problem identification, costs, time, gaps, and objectives c) Task Automation including requirements, steps, dependencies, quality assurance metrics, d) Cycle Review Document including staff feedback and summary.
Methodology
The project is approached from key strategies; 1) Automation is used to answer security effectiveness deficiencies, and 2) Begin with compliance mapping then move to Agile cycles for each automation task.
Instead of automation, the team could have taken an approach to expand existing staff or to out-source security tasks to a managed service provider. An alternative to Agile cycles, the team could have kept with waterfall project methodology.
Approach Explanation
The security team reviewed plausible methods for improving efficiencies. Security operations can be boiled down to how well an organization is protected against threats and how well it can recover from an incident. Additionally, the costs to mitigate risks cannot exceed the cost of the incident itself, often refeed to as “return on investment.” Automation and orchestration mechanisms leverage the speed and continuous operation of computers. One area that computer systems far exceed human capability is repetitive tasks. Machine learning can perform data analysis up to 100 times faster and more accurately than humans (Rayome, 2017). If appropriately configured to receive the correct data streams, automation systems can process thousands of events a second and produce actionable alerts. These alerts can then trigger automation actions and orchestration across multiple systems to quickly shut the door on would-be attackers. Considering the attacks are often automated systems themselves, defense with similar automation is required.
The project management strategy uses Agile methodologies. The process of automating manual tasks is new to security and IT operations teams. As such, a project management approach needs to be reactive to the needs of the process, and well equipped to discover new issues and now to deal with this effectively. Agile methods allow the implementation teams to draw on cross-functional team members and to perform specific tasks quickly.
Approach Defense
The possible approaches are 1) hire additional staff or 2) utilize automation.
The first method was found to have the largest cost. According to the Bureau of Labor Statistics, an Information Security Analyst’s average annual wage is $98,350. (Bureau of Labor Statistics 2018). Joe Hadzima of MIT estimates that wages should be multiplied by 1.25 to 1.4 to take into consideration taxes, benefits, etc. For this study, an average of this is made at 1.325. (Quintana 2018) The cost of an additional staff member is, therefore, estimated at $130,313.75 annually.
The following table breaks down standard security tools along with cost difference to enable automation capabilities. If the base cost includes automation features, then no fee is listed.
| Security Tool | Purpose | SOAR Capability | Cost To Add Automation and Orchestration |
| AT&T USM Anywhere | Security Event and Information Monitoring | Event correlation driving response actions | $7,440 |
| Cisco Umbrella | Content Filtering | Enforcement API | $0 |
| Office 365 | Cloud Productivity | Event Forwarding | $0 |
| Digital Guardian | Data Loss Prevention | Event Forwarding | $0 |
| Jira | Ticketing System | Automatic Ticket Creation (notification) | $0 |
| Cylance | Endpoint Protection | Event Forwarding | $0 |
According to the AT&T Cybersecurity website, there are three tiers of subscriptions for USM Anywhere SIEM tool; Essentials, Standard, and Premium. The Essentials subscription does not support orchestration. Standard and Premium only differ in that Premium support longer log storage time for PCI compliance. PCI compliance is not a requirement. The difference between Essentials and Standard is $620/mo, billed yearly for $7,440. (AT&T Cybersecurity, 2019) The base fees for the other cyber tools include SOAR capabilities needed. Existing staff was used to automate the five chosen tasks using these tools and the SOAR mechanisms available in the higher AlienVault subscription.
The expected cost of hiring another security professional is close to $130,000 (or $62.5/hour), whereas the price to add SOAR capabilities is about $7,500. The final methodology provided a lower-cost solution that met the needs to reduce MTTD and MTTR.
Project Development
The security team produced deliverables during process and systems audits which included a detailed listing of hardware, software, architectural details, tech stacks for each automation task, what resources are used in the setup and the operation of each task, and finally the output of each project cycle and associated automated task. Each section identifies the requirements for each phase.
Hardware
| Task | Hardware |
| A. Blocking Phishing Attacks | – User workstation – Cloud resources for SaaS email administration platforms |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Ansible Server (virtual machine, 64-bit support, min 2 CPUs, 4GB RAM, 20GB of Dedicated HD space) – Cloud resources for SaaS AlienVault USM Anywhere |
| C. Reporting Logins from Unusual Devices | – Cloud resources for SaaS AlienVault USM Anywhere |
| D. Creating Service Ticket for Indicators of Compromise | – Cloud resources for SaaS AlienVault USM Anywhere |
| E. Isolating Compromised Endpoints | – User workstation – Cloud resources for SaaS AlienVault USM Anywhere |
Software / Tech Stack
| Task | Software |
| A. Blocking Phishing Attacks | – AlienVault USM Anywhere upgrade to “Standard” |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Ansible version 2.2 & Ansible Tower (orchestration engine) [running on Red Hat 8.0] – Remediation Playbook (YAML) – Unquoted Path PowerShell Script |
| C. Reporting Logins from Unusual Devices | – AlienVault USM Anywhere upgrade to “Standard” – AlienVault Agent (endpoint software) |
| D. Creating Service Ticket for Indicators of Compromise | – AlienVault USM Anywhere upgrade to “Standard” – Exchange Online (SMTP server) – Jira Service Desk |
| E. Isolating Compromised Endpoints | – AlienVault USM Anywhere upgrade to “Standard” – AlienVault Agent (endpoint software) |
Tech Stack
| Task | Stacks |
| A. Blocking Phishing Attacks | – MS Outlook (email client) – Mimecast (email cloud service) – AlienVault USM Anywhere – Cisco Umbrella (DNS proxy) Enforcement API – Cylance (endpoint anti-virus) – Exchange Online (email server) |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Ansible version 2.2 & Ansible Tower (orchestration engine) [running on Red Hat 8.0] – PostgreSQL version 9.6.x – PowerShell ISE (script writing IDE) – AlienVault (vulnerability scanner) |
| C. Reporting Logins from Unusual Devices | – AlienVault USM Anywhere – AlienVault Agent (endpoint software) |
| D. Creating Service Ticket for Indicators of Compromise | – AlienVault USM Anywhere – Exchange Online (SMTP server) – Jira Service Desk |
| E. Isolating Compromised Endpoints | – AlienVault USM Anywhere – AlienVault Agent (endpoint software) |
Architecture Details
| Task | Details |
| A. Blocking Phishing Attacks | – Outlook client had the Mimecast Phishing reporting toolbar addon installed using a GPO. This configuration emails Mimecast security and the ABC Ins. Security team. – AlienVault contains a Mimecast Email Security plugin for ingestion and normalization of Syslog data. Rules were configured to trigger alarms on events reporting phishing email verification using Mimecast documentation (Mimecast 2019) – The same AlienVault alarm triggers the Forensics and Response integration to automatically gather forensic data from the reporting endpoint. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Ansible Tower (together with Ansible Engine) was implemented on-premise. State persistence was enforced using a playbook designed to check and then update unquoted paths on all endpoints and servers running windows OS daily. See appendix A for the playbook and PowerShell script details (Appendix A). – AlienVault vulnerability scanning is configured to run on all assets twice a week. |
| C. Reporting Logins from Unusual Devices | – AlienVault agent was deployed to all workstations by the Service Desk staff. This agent reports changes to IP address and network interfaces. – AlienVault was additionally configured to trigger ‘Environment Awareness’ alarms on any ‘Asset Discovered’ event. (Appendix B) This event triggered emails to the Security Team and the Service Desk. Assets with agents did not trigger this type of event. Therefore it is significant to investigate when AlienVault is alerted to a new asset found. |
| D. Creating Service Ticket for Indicators of Compromise | – AlienVault is configured to send all alarm alerts as email notifications to user accounts. The Jira Service desk system email was set as a ‘Read Only’ account in AlienVault and therefore received all alarms notifications, which in turn converts to service tickets. |
| E. Isolating Compromised Endpoints | – AlienVault was configured to trigger the Forensic and Response integration to disable the networking interfaces on the source asset, which reported an Indicator of Compromise in the “System Compromised” intent category. (See Appendix C) |
Resources Used
| Task | Required Resource(s) |
| A. Blocking Phishing Attacks | – Email client ‘phishing’ reporting button – Email security platform capable of scanning links and attachments. – Email security platform capable of triggering SIEM platform action – Six user workstations from a minimum of 3 different departments for initial testing (during the Explore and Adapt phase). This is likely to last for 2 to 3 days and must be able to handle system downtime if implementation interferes with daily tasks. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Automation engine capable of: – running daily – executing PowerShell script on all windows endpoints – Vulnerability scanning tool capable of: – running daily – reviewing scans for published CVE data |
| C. Reporting Logins from Unusual Devices | – Network devices (physical or virtual) capable of forwarding logs in Syslog format to a SIEM. – Security software capable of forwarding event logs to the SIEM – Plugin availability in the SIEM for ingesting and normalizing the Syslog data – Six user workstations from a minimum of 3 different departments for initial testing (during the Explore and Adapt phase). This is likely to last for 2 to 3 days and must be able to handle system downtime if implementation interferes with daily tasks. – Two test workstations to ‘act’ as unusual devices to test asset discovery without SIEM agent. This will be needed for a minimum of the entire Agile cycle (1 week). |
| D. Creating Service Ticket for Indicators of Compromise | – Ticket platform capability of integration with SIEM platform |
| E. Isolating Compromised Endpoints | – Test malware to execute on the endpoint to mimic malicious actions that can trigger SIEM high-risk events, but not pose any real risk to the network. – Two test workstations to ‘act’ as compromised devices to test. This will be needed for a minimum of the entire Agile cycle (1 week). |
Final Output
| Task | Output |
| A. Blocking Phishing Attacks | – Phishing email blocking and removal from inboxes will be automated. MTTD is expected to remain the same as still required the user to report. Reduction of MTTR from 55 min per email per day to Zero. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Vulnerability will be remediated daily in an automated process. MTTD and MTTR reduce to zero. – AlienVault vulnerability scan reports for tracking process success. |
| C. Reporting Logins from Unusual Devices | – AlienVault alarms generate automated service desk tickets, which includes alerts for assets found. |
| D. Creating Service Ticket for Indicators of Compromise | – AlienVault alarms generate automated service desk tickets. Alarms are triggers by events for indicators of compromise. |
| E. Isolating Compromised Endpoints | – AlienVault alarms generate automated service desk tickets, which includes alerts for forensic and response integration (which triggers the network card shutdown) |
Quality Assurance
The security team worked from the beginning of the project and through the Agile process and to the conclusion to ensure they created an environment where quality requirements were met, according to two standards; the NIST Cybersecurity Framework and the measurements of MTTD, MTTR and cost control. An on-going testing and evaluation process provided quality control during the Agile lifecycle.
Quality Assurance Approach
Ensuring each phase of the project met quality standards, the first meeting reviewed compliance and controls for cybersecurity frameworks. The NIST CSF was chosen as the standard to which each process and procedure was compared. Specific controls were chosen and mapped to each automation task. These standards provide the basis for method development.
Additionally, a baseline was established for each task to measure the current manual metrics for mean-time-to-detection, mean-time-to-recovery, and cost in working hours. Agile team members reviewed processed during development for a reduction in these metrics.
At the close of each phase, team members reviewed the cybersecurity controls and the automation mechanisms to ensure compliance. As task automation held to these quality standards, the next Agile cycle was allowed to begin.
Solution Testing
A test case was developed for each task to review automation actions, verify expected automation output, and confirm testing criteria acceptance.
| Task | Test Case | Acceptance Criteria |
| A. Blocking Phishing Attacks | – An independent phishing testing service, Infosec IQ, generated two waves of fake messages to send to the organization. Staff watched for phishing reports and recorded the time it took from the report to a DNS blacklist update and exchange email removal. | – Reduction of MTTR from 55 min per email per day to less than 5 minutes. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Two Windows endpoints (one Windows server 2012 and one Windows 10) had unquoted paths manually reintroduced four times over two days. | – AlienVault report shows no instances of target vulnerability when scanned at the end of each business day. |
| C. Reporting Logins from Unusual Devices | – Service Desk staff introduced three new endpoints to the network which did not have the AlienVault agent installed. – Two workstation laptops had new IP addresses assigned after AlienVault agent was installed. | – AlienVault triggers a ‘Suspicious Behavior Potential Unauthorized Device Connection’ alarm when any of the three devices are connected to the network. – Alarm sends an email to the Service Desk which is converted to a service ticket. – When IP address changes, that information is automatically updated in AlienVault asset list. |
| D. Creating Service Ticket for Indicators of Compromise | – Download KnowBe4 RanSim software for testing various types of ransomware activities. Trigger on two workstations can view alarms in AlienVault and watch for ticket creation. | – AlienVault alarms generate automated service desk tickets. Alarms are triggers by events for indicators of compromise. |
| E. Isolating Compromised Endpoints | – Manual Alarm triggered for ‘System Compromised’ severity. Monitor test workstation for forensic response action to shut off the network interface card. | – AlienVault alarms generate automated service desk tickets, which includes alerts for forensic and response integration (which triggers the network card shutdown) |
Implementation Plan
During the second phase of the project, each Agile team implemented the automation or orchestration mechanisms needed for the assigned task. The following describes the plan and resources for each Agile segment; envision, speculate, explore, adapt, and close.
Strategy for the Implementation
The Agile approach was used as opposed to a tradition waterfall method to keep the team tasks targeted and capable of reacting to unforeseen changes in implementation. Work to augment detection and recovery with automation and orchestration is a new process for the security team. They expected some level of unknown variables to impact implementation.
Additionally, the tasks were broken out to different implementation teams to provide sufficient closure in the 30 days’ time constraint. This approach allows the teams to be comprised of subject matter experts for the various elements of the “tech stack” to focus on specific details of the assigned automation task.
Phases of the Rollout
The implementation sequence is detailed in the following list.
Phase 1 – Requirements and controls. The security team met to chose specific tasks to automation and limit scope. NIST cybersecurity controls mapped to each task. Acceptance requirements were established. Team members assigned and time frames set.
Phase 2 – Implementation, testing, and acceptance. Using Agile segments envision, speculate, explore, adapt, and close, the Agile team cycle ran. Envision & speculate allowed teams to ingest assigned tasks and detail acceptance criteria. The explore and adapt phases included both implementation and testing work. As acceptance criteria are met, the team moved to the final Agile segment.
Closing – After acceptance, team members evaluated each stage of the automation process. Regression testing was performed to validate that additional automation or orchestration tasks did not change or interfere with previous automation. Finally, after all cycles complete, the entire project was evaluated, and requirements for post-implementation support established.
Details of the Go-Live
During the close phase of the entire project, each section was reviewed to verify that acceptance criteria had been met. Additionally, team members documented what additional work or updates to the automation or orchestration would be needed in the next 3 months and 1 year (if any was identified.)
The production of a final acceptance document, including future support requirements, signaled full-implementation.
Dependencies
Phase 1 needed to be completed before phase 2 could begin to ensure teamwork was on task and completion requirements understood. It was could that tasks 1, 2, 3, and 4 needed to be completed before task five could begin. Email notification was required to be verified first because task five’s scope did now include work to ensure email notification to the service desk was functional.
Deliverables
Each team was required to produce expected output during the acceptance phase. The following list reviews the required output deliverables expected from each automation task.
| Task | Output |
| A. Blocking Phishing Attacks | – Phishing email blocking and removal from inboxes will be automated. MTTD is expected to remain the same as still required the user to report. Reduction of MTTR from 55 min per email per day to Zero. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – Vulnerability will be remediated daily in an automated process. MTTD and MTTR reduce to zero. – AlienVault vulnerability scan reports for tracking process success. |
| C. Reporting Logins from Unusual Devices | – AlienVault alarms generate automated service desk tickets, which includes alerts for assets found. |
| D. Creating Service Ticket for Indicators of Compromise | – AlienVault alarms generate automated service desk tickets. Alarms are triggers by events for indicators of compromise. |
| E. Isolating Compromised Endpoints | – AlienVault alarms generate automated service desk tickets, which includes alerts for forensic and response integration (which triggers the network card shutdown) |
Training Plan for Users
General Users – The only impact on general users was a slight change in the look and wording of the phishing alert button in MS Outlook. Other automation processes only involved security, IT operations, and service desk staff.
Security, Operations, and Service Desk – A part of the planning process and task notes is documentation on how the teams work through the automation processes. What code was written, where it is stored, how new software was configured and what changes to detection and recovery processes happened. After project completion, the project lead and the security team met with the IT operations (system administrators) and the service desk staff and reviewed the project results to familiarize the staff with new expectations and what their roles are for monitoring and supporting the new automation and orchestration.
Risk Assessment
The project included some measure of risk. Misconfiguration could result in failure for the automation to trigger and run correctly and timely. False positives can cause automation actions to hinder legitimate operations. Implementation costs could exceed expectations. The following are specific risks identified for each task, a cost/benefit analysis of each and mitigation strategies along with roll-back and alternative implementation plans.
Quantitative and Qualitative Risks
Risks are measured for probability on a scale of 0 to 1, and severity on a scale of 1 to 5. These measurements are multiple to each a priority level.
| Risk | Probability (0 to 1) | Severity (1 to 5) | Priority |
| Phishing false positives can block legitimate domains and delete legitimate emails. | 0.5 (Francis, 2017) | 3 | 1.5 |
| Automation engine task fails to complete. | 0.1 | 1 | 0.1 |
| AlienVault agent is not installed on endpoint. | 0.1 | 2 | 0.2 |
| Not all network segments are included in SIEM scans. | 0.5 | 4 | 2.0 |
| Ticketing system overwhelmed by high volume of false positives. | 0.5 | 2 | 1.0 |
| False-positive blocking legitimate workstation activity. | 0.1 | 4 | 0.4 |
Cost/Benefit Analysis
All of the automated tasks provide a clear benefit of reduced staff time. Blocking phishing attacks, remediating unquoted path vulnerability, and reporting unusual device logins to offer improvements in mean-time-to-detect possible malicious activity. All five automation mechanisms provide improvement in mean-time-to-recovery once a malicious activity has been detected. The cost of implementation (and possible over-runs in implementation time) in this project contain the majority of the risk. The following identifies the benefit associated with a given risk, the cost of the automation fails to operate and provide the expected benefit, and the cost if implementation takes longer than expected. The risk exposure related to a failure to protect against the security threat is minimal since the primary result of failure of automation is to fall back to previous manual processes, and related costs in working hours. (See Appendix E for specific cost references).
| Risk | Benefit | Shortfall Cost | Overrun Cost |
| Phishing false positives can block legitimate domains and delete legitimate emails. | 1. Threat remediation improving from 55 minutes to 5 minutes per email. 2. Remove the need for manual access for multiple administrative interfaces to update blacklists and remove offending email. | Blocking legitimate emails can cause the additional time to recovery. The estimated 5-minute response could expand to the possible 55-minute estimate for manual analysis and response. The potential shortfall has a max potential cost of the original manual process estimate of $1,432.50. | Security tools (AlienVault, Mimecast, Office 365, and Cisco Umbrella) have the same costs whether the work is manual or automated. As such, the only potential overrun of costs would be in man-hours for implementation. The cost savings (as outlined in Appendix E) are $600K+. For an overrun to warrant a change it would need to exceed that cost annually. If it was estimated to require that level of additional man-hours, then the automation effort would be abandoned. |
| The orchestration engine task fails to complete. | 1. Daily check and updating of unquoted paths without staff intervention, fixing the repeated vulnerability. | Scans performed daily would catch that this vulnerability is not being corrected. Security staff would need to run the PowerShell script rather than manually. This would reintroduce the 60 minutes a week cost in time. | Implementation overrun would cause additional man-hours to properly configure. Ansible installation and setup is accounted for in a different project. Specific YAML and PowerShell code development is for this orchestration mechanism. The estimated cost savings is $62.50/week. That translates to 4 hours of staff time a month. If project implementation exceeds the budget for the one-week cycle by more than 4 hours, the project will begin costing more than efficiency benefit. |
| AlienVault agent is not installed on the endpoint. | Device discovery events in AlienVault are automatically associated with asset agents. Alerts not associated with agents are tagged for investigation. | The processes for investigating an unusual device discovery is estimated at 30 minutes for both security and service desk staff. | Agent deployment is time-consuming and asset management is additionally costly in working hours. The Agile team’s goal is to provide processes and use case validation for installation of asset agents. Once that initial process is complete, additional asset management (including the agent) is included in service desk activities. This will exceed cost-benefit if managing the agent installations exceeds 50 minutes a month. |
| Not all network segments are included in SIEM scans. | Devices attached to network segments not included in scans will not trigger alarms or staff investigation. This can lead to potential undetected malicious activity with resulting significant compromises. Costs would include typical cybersecurity incident outages and possibly permanent loss of data. | ||
| Ticketing system overwhelmed by a high volume of false positives. | Automated generation of tickets provides faster notification and response times. | If it takes more than 15 minutes per ticket for the service desk technician to validate (the same time estimate for manual ticket creation) then any benefit will be void. With the potential of many more tickets being created by unexpected IoC generation, the automation of tickets may be significantly larger in staff time. | Configuration of automation includes create a new SIEM user account and ensuring that a single checkbox in AlienVault is checked. The potential for implementation of overruns is negligible. |
| False-positive blocking legitimate workstation activity. | Response time for discovered high-comprise IoCs would be improved – reducing reaction time to what could stop malware spreading. | False positives would require service desk staff to fall back to manual processes of location and reconnection (estimated at 40 min per incident). | Automated network isolation includes configuring Forensic and Response integration features. The potential for implementation overruns could result if response actions testing failed during project implementation. Testing for correct ‘positives’ to ensure legitimate workstation isolation is the more time-consuming aspect. However the costs of not stopping malware actions is significantly higher. |
Risk Mitigation
Once the cost/benefit analysis is complete, mitigation plans can be identified for each risk. Specific roll back plans must be established for recovering if the automation or orchestration efforts fail or do not meet cost/benefit expectations. And alternatives identified for ensuring that task purposes continue to be met.
| Risk | Mitigation | Rollback | Alternative |
| Phishing false positives can block legitimate domains and delete legitimate emails. | Users can notify the security team of possible false positive which could be blocking email. The security staff can investigate. If confirmed, the DNS block can be lifted. An email that is removed from users’ mailboxes automatically can be resorted to by Exchange Online administrative tools. | Email is saved to Veeam backups. This system includes a restoration feature for removed messages. Umbrella blacklists can be edited to remove legitimate domains. | Security staff manually reviewing and acting on verified phishing emails using established manual security playbooks. |
| The automation engine task fails to complete. | No mitigating plan is needed due to the nonintrusive nature of the remediation. | No action results in a lack of need to undo automation. | The security staff manually runs the PowerShell script. |
| AlienVault agent is not installed on the endpoint. | AlienVault alarms trigger alerts on any new assets connecting to the network. These alerts are reviewed by staff. | N/A | Manual installation of Agent and/or manual updating and validating of asset inventory with new connection information. |
| Not all network segments are included in SIEM scans. | The service desk is included in ticket creation when unauthorized devices are found. During follow-up investigations, missing agent installation will be identified. This can trigger audits of the device deployment and lifecycle management process. There will need to be regular reviews of what IP ranges are scanned by network architects to ensure all segments are included. | N/A | 802.1x implementation with hardware and wireless connections to disallow unexpected connections to the network. While waiting for 802.1x implementation, DHCP could be turned off, coupled with MAC address filtering. |
| Ticketing system overwhelmed by high volume of false positives. | The alarm ‘mute’ feature will be utilized in the SIEM. This pauses alarm creation for the same rule criteria for a given time frame. This will be set to 30 minutes, ensuring that only one alarm will be created and in-turn only one service desk ticket created every 30 minutes | New tickets created in the ‘Security Automation’ category in Jira can be selected and deleted in Jira administrative console. | Disable automatic ticket creation and have security staff manually create tickets for valid alarms. |
| False positive blocking legitimate workstation activity. | Notifications sent to the service desk for ticket creation for this action are especially marked for high-priority action. Additionally, Amazon SNS messaging is configured to notify the security team after hours. This additional level of response request is intended to initiate faster staff response to this high-risk event, and to handle false positives that may be blocking legitimate user activity. | Manual re-enabling the workstation network interface card. | Disable automatic Forensic and Response integration and have security staff manually trigger the integration action. |
Post Implementation Support and Issues
In the final phase of the project, the teams met for a closure discussion. A key part of this meeting was establishing post-implementation support and processes for handing continued development and fine-tuning of the automation.
Discuss post-implementation support for the solution. Provide a plan for support of solution after rollout (if applicable). This should include both fixes for issues and for ongoing maintenance.
Post Implementation Support
The project team identified the items for the continued support of each task automation or orchestration. (See Appendix H) Ideally, automation and orchestration will reduce the amount of staff support hours as part of the strategy to utilize it for improving efficiencies.
Some SaaS cloud platforms are utilized for key automation services – namely the AlienVault USM Anywhere. Cloud support services maintain these resources as part of regular subscriptions. It is established that the only support role of ABC Ins staff perform is administering the subscribed services.
Post Implementation Support Resources
On-Premise servers for Ansible Engine, Ansible Tower, GitHub Enterprise, and Jira ServiceDesk will be maintained and updated by System Administrators.
Maintenance Plan
Cloud services provide contractual support of platforms. The technical contact information is maintained by the security and IT operations teams. This includes MS Exchange Online, AlienVault USM Anywhere, and Mimecast.
On-Premise servers are included in ongoing server maintenance and support cycles. Each server is included in monthly patching cycles and annual reviews for significant version upgrades or replacements.
Conclusion, Outcomes, and Reflection
The project to augment security detection and recovery through automation and orchestration was completed on time and within expected scope. Key measurements of detection and recovery times were reduced while improving overall security costs. Encouraged by the results, the security team plans to expand to additional tasks were deemed appropriate. Each Agile team joined the security, service desk and IT operation teams to reflect on the processed, chosen approaches and the outcomes.
Project Summary
This project’s goal was to review the business cost and operational effectiveness to augment a SIEM through automation and orchestration capabilities, either through built-in tools or through additional automation solutions. The teams narrowed the scope to 5 common security tasks that were judged to respond well to automation or orchestration. The Agile project management methodology was used in implementation and testing to keep team members responsive to unforeseen variables. The project manager worked with the security team to develop a financial analysis of pre-project manual processes for each task by created manual playbooks that identified working hours for each task. This formed the bases of how the cost could be saved with automation instead of hiring additional analysists or engineers to perform the tasks. The project was given five weeks to complete (approximately 30 days). Week one was set aside for discovery and compliance mapping. In weeks two and three, two Agile teams worked on the first four tasks. The final task was completed in week four. Week five was used for closure and reflection.
Deliverables
Various phases of the project require products to verify phase completion and serve as documentation of work.
| Project Phase / Segment | Deliverable(s) |
| Phase 1 – Compliance and Task Identification | Appendix G: Controls Mapping |
| Phase 2 – Implementation | |
| Envision / Speculate | Appendix D: Automation Deliverables Document: Task Audit Appendix E: Cost Analysis Detail Appendix F: Task Requirements Appendix G: Dependency Breakdown |
| Explore / Adapt | Appendix A: Ansible Playbook Appendix B: Alarm Configuration Appendix C: Response Action Rule |
| Close | |
| Project Close | Appendix H: Post-Implementation Support |
Outcomes
Automation and orchestration are a hot-button topic in the cybersecurity field. The possibility of increased reaction times, becoming more efficient as a security team, and keeping costs under control, is highly attractive. There is a danger of expecting the SOAR mechanics to serve as a ‘silver bullet’ and solve security gaps altogether.
The project provided an opportunity to test now well specific tasks could be augmented. During the initial phases of the project conception, it was planned to be broad with the concepts. This idea gave way to a more focused approached and limit the scope to five specific tasks. Each of these tasks was automation in a real corporate environment, with a real team of security and IT professionals. Some details were altered to maintain corporate privacy. But the work was completed as outlined.
Staff reacted positively to the concepts and the organized approach. The organization had begun implementing Agile principles. This allowed both the SOAR concepts and Agile methods to be tested. Some additional insights were realized during the processes which can inform future automation and orchestration efforts.
- Handling phishing email reports is complex. Although the automation assisted in improving reaction times and freeing up security staff’s time, it remains dependent on user judgment. An additional improvement to this security step is to use a security product (or feature set) which screens all incoming emails for an added layer of automated review and sandbox testing of attachments; all before the potentially malicious email reaches the user inbox. This improvement has been identified and there are plans to implement it.
- The Ansible automation works well for the state-enforcement strategy. This could be easily expanded to include other remediation efforts that include system state. It is important to realize that adding the overhead and cost of Ansible and associated administration would not have been cost-effective for the single automation task “Remediating Reoccurring Unquoted Search Path Vulnerability.” The Operations team was already committed to Ansible implementation for other network and server administration tasks.
- The work to automating the reporting of unusual devices underwent the most significant change from initial concept to final implementation. Initially, it was planned to have lists of authorized endpoints added to a ‘correlation list’ in AlienVault. This would be compared against to determine if a workstation was supposed to be connecting to the network. However, before the initial planning was complete, it was realized that installing agent software would serve this purpose with no additional work. This included improved workstation event visibility and monitoring.
- For the service desk ticket creation, the concept proved successful. It functions as planned. After the project conclusion, it was realized that service desk tickets were getting created for alarms that did not need to be sent to the service desk. This prompted additional retooling of the alarm triggers reduce volume of notifications. Additionally, it was suggested that service desk users received email notification and the could, in turn, create tickets if needed. Although different than the original automation plan, it proved more practical for staff.
- The system isolation task was tested well just after project completion. The organization had an internal penetration test right after automation was complete. During the test, the tester’s computer was blocked by the isolation response action – which disabled the network access as planned during an ‘insider threat’ test. It was a great proof of concept for the value of orchestration
Reflection
The opportunity to put SOAR concepts into proactive proved valuable. The project helped flushed out all the ideas, plans, and work that needs to be completed to realize task integration in a complex environment. Many of the concepts and skills developed over the previous two years working on degree coursework have come together to provide a valuable improvement to a real-world organization.
Future orchestration possibilities have been highlighted. The work learning how to write Ansible playbooks from manual process outlines has led to plans for full patching automation! The process is currently manual and requires dedicated attention from security staff for most of a day to complete. This interrupts business operations and keeps security staff from provided needed care to other efforts. The work on formalizing a change process outlined in this project is leading the change management process for this new project.
Finally, it is essential to understand that while orchestration and automation improved mean-time-to-detect and mean-time-to-recovery from incidents, and proved to be cost-effective, it does not replace the human security professional. The changing landscape of threats can easily make automated tasks invalid. It is the experience of a security professional who must understand the need for updates and how to implement them to keep the SOAR tools sharp and effective.
References
Thomson, I (2016) NSA’s top hacking boss explains how to protect your network from his attack squads. Retrieved from https://www.theregister.co.uk/2016/01/28/nsas_top_hacking_boss_explains_how_to_protect_your_network_from_his_minions/
Lawson, C & Neiva, C & Bussa, T & Sadowski, G (30 November 2017) Innovation Insight for Security Orchestration, Automation and Response. ID: G00339719.
Newman, Peter (January 2019) IoT Report: How Internet of Things technology growth is reaching mainstream companies and consumers. Retrieved from https://www.businessinsider.com/internet-of-things-report?IR=T
Skeen, Dale (2014) The Next Big Thing in the Internet of Things: Real-time Big Data Analytics. Retrieved from https://www.vitria.com/pdf/Forrester-Vitria-IOT-Dale-Slides.pdf
CSPI (March 2019) A Look at Cyber-Security Spending in 2019: Where Budgets are Increasing and Why. Retrieved from https://securityboulevard.com/2019/03/a-look-at-cyber-security-spending-in-2019-where-budgets-are-increasing-and-why/
NeSmith, Brian (August 2018) The Cybersecurity Talent Gap In An Industry Crisis. Retrieved from https://www.forbes.com/sites/forbestechcouncil/2018/08/09/the-cybersecurity-talent-gap-is-an-industry-crisis/#297e7d38a6b3
Galvin, Joe (May 2018) 60 Percent of Small Businesses Fold Within 6 Months of a Cyber Attack. Here’s How to Protect Yourself. Retrieved from https://www.inc.com/joe-galvin/60-percent-of-small-businesses-fold-within-6-months-of-a-cyber-attack-heres-how-to-protect-yourself.html
Rayome, Alison DeNisco (December 2017) MIT’s automated machine learning works 100x faster than human data scientists. Retrieved from https://www.techrepublic.com/article/mits-automated-machine-learning-works-100x-faster-than-human-data-scientists/
Bureau of Labor Statistics (May 2018) Occupational Employment and Wages, May 2019. 15-1122 Information Security Analysts. Retrieve from https://www.bls.gov/oes/current/oes151122.htm
Quintana, Carlos (November 2018) The true cost of employees: How much does an employee *really* cost? Retrieved from https://beebole.com/blog/how-to-calculate-the-real-cost-of-an-employee/
Francis, Ryan (May, 2017) False positives still cause threat alert fatigue. Retrieved from https://www.csoonline.com/article/3191379/false-positives-still-cause-alert-fatigue.html
AT&T Cybersecurity (2019) Affordable pricing to fit every budget. Retrieved from https://www.alienvault.com/pricing
Williams-Shaw, Sydni (March 2018) How to build an incident response playbook. Retrieved from https://swimlane.com/blog/incident-response-playbook/
Technologent (2019) Why Gartner’s SOAR Model is the Future of IT Security. Retrieved from https://blog.technologent.com/gartner-soar-model-future-it-security
Appendix A: Ansible Playbook
For running PowerShell script to remediate Unquoted Paths.
Global Variables
anstower_ssh_user: adminuser
anstower_ssh_pass: adminpassword
anstower_ssh_port: 5985
anstower_connection: winrm
anstower_winrm_server_cert_validation: ignore
- name: Execute PowerShell Script to fix unquoted paths
hosts: test
tasks:
- name: PowerShell script to fix unquoted paths
script: scripts/Windows_Path_Enumeration.ps1
register: out
- debug: var=out
PowerShell Script = Windows_Path_Enumeration.ps1 (downloaded from https://gallery.technet.microsoft.com/scriptcenter/Windows-Unquoted-Service-190f0341)
Appendix B: Alarm Configuration
The following is a screenshot of the Alarm configuration in AlienVault for Asset Discovered events.
Appendix C: Response Action Rule
The following is a screenshot of the Response Action Rule confirmation for disabling the endpoint network card on an offending system.
Appendix D: Automation Deliverables Document: Task Audit
The following document outlines the manual steps into playbooks. Each playbook includes an audit of time resource investment. Information was completed from the two implementation teams in post-project debrief.
| Task | Time to Detection | Time to Recovery |
| A. Blocking Phishing Attacks | 20 min per email (daily) | 55 min per email (daily) |
| B. Remediating Reoccurring Unquoted Search Path Vulnerability | Up to 7 days | 60 min (weekly) |
| C. Reporting Logins from Unusual Devices | Up to 30 days | 10 min (monthly) |
| D. Creating Service Ticket for Indicators of Compromise | n/a | 15 min per IoC |
| E. Isolating Compromised Endpoints | n/a | 40 min |
A. Blocking Phishing Attacks [up to a possible 75 minutes per email] – The security team receives up to approximately 50 phishing email reports a day. It is estimated that it takes between 5 and 15 minutes per email to conclude if it is malicious or not. This is a possible 12 and a half hours a day spend verifying if reported phishing emails are malicious. According to a Verizon Data Breach Investigations report in 2015, half of the phishing emails are opened and clicked on which the first hour. (Verizon 2015). The team used this metric to estimated that 25 of the reported emails are malicious. This estimate increases the daily investigations and remediation time to possible 23 hours. The following are the manual steps and estimated time for each:
B. Remediating Reoccurring Unquoted Search Path Vulnerability [a possible 35 minutes each scan which is performed twice a week] – The security team determined that identifying a specific recurring vulnerability would be a more useful measure of automation effectiveness. General vulnerability scanning is already being performed on a scheduled basis by scanning tools that are monitored by security team members. Mitigating a reoccurring vulnerability has a significant time investment, which provides a prime use case for automation augmentation. This vulnerability is referenced in many CVE’s as early as 2000. The weakness allows a local privilege escalation to administrative “system” user. A simple PowerShell script from Microsoft can traverse the registry and correct this vulnerability. However new patches tend to reintroduce unquoted paths. Therefore, as updates and patches are applied, analysis is forced to search and reapply the correction across networked systems. The entire process takes an additional 35 minutes of the security team member’s time to complete each vulnerability scan cycle, which is twice a week.
C. Reporting Logins from Unusual Devices [approximately 50 minutes a month] – The service desk ticking system contains a manually maintained inventory of all purchased assets. The security team reviews the database of current assets monthly and compares them to logs of devices that have logged into the Active Directory. Devices which have logged in, but do not appear in the asset inventory are considered ‘unusual’ and prompt an investigation.
D. Creating Service Ticket for Indicators of Compromise [approximately 15 minutes per IoC] – During the daily routine of the security team members, various indicators of compromise are found. When determined credible, a security team member opens the service desk ticketing system and adds a new ticket for the IoC. This allows the entire IT department to be aware of the compromise and track its remediation. Approximately 23 possible IoC’s per week are identified and require ticket creation.
E. Isolating Compromised Endpoints [an average time of 30 minutes depending on the service desk ticket workload] – As high-risk compromises are detected; a security team member works to isolate the system from the network. Currently a service desk team member moves to the target system and manually disconnects the system from the network. The system is left running to preserve forensic integrity. ABC Insurance has an average of 1 comprised endpoint a month for the last 2 years.
- Security analysts locate a high risk IoC and emails the service desk
- A service desk technician receives the ticket and walks to the location and disconnects the Cat5 cable from the back of the workstation. [up to 30 min]
- Once complete the technician returns to the services desk and updates the ticket, informing the security team that the system is ready for further action. [10 min]
Appendix E: Cost Analysis Detail
The following provides cost analysis for each task in working hours multiplied by an estimated $62.50 for skilled security time.
| Task | Cost for Detection | Cost for Recovery | Total Annual cost |
| A. Blocking Phishing Attacks | |||
| B. Remediating Reoccurring Unquoted Path Vulnerability | |||
| C. Reporting Logins from Unusual Devices | |||
| D. Creating Service Ticket for Indicators of Compromise | n/a | ||
| E. Isolating Compromised Endpoints | n/a |
Appendix F: Task Requirements
| Task | Requirements | Gaps | |
| Functional (end-user) | Technical, Operational, Standards, or Restrictions | ||
| A. Blocking Phishing Attacks | – User reporting via the integrated button in email client | – Automate the analysis of reported email – Log data sent to SIEM must contain a) URL string containing offending source address, b) reporting user – DNS Proxy must include the capability of enforcement API call to automate adding of an offending domain – SIEM/SOAR tools must include the ability to automatically gather forensic data on reporting user endpoint. | – User reporting requires forwarding the email to the security team – The analysis is done manually with checks to VirusTotal.com and on-premise sandboxed workstation. – The volume of reported phishing emails is beyond the capacity of current security staff. As automation is introduced, more complete protection will be provided. |
| B. Remediating Reoccurring Unquoted Path Vulnerability | – The process should be automated, requiring no user interaction | – The automation engine should regularly check and update any endpoints with unsigned paths, without waiting for vulnerability detection or require user initiation of scripts. | – Currently, this process is a manual two-step process. Step 1 is for detecting the reoccurrence of the issue, and step 2 is for running the PowerShell script to remediate. |
| C. Reporting Logins from Unusual Devices | – Receive proactive alerts when devices connect which are not on the approved list | – Move security data tracking from the purchasing asset inventory over to the security information and event management system. – Track data sensitivity level – Track MAC Addresses of allowed devices – Trigger Amazon SNS alerts to security and service desk teams | – Data is currently manually entered and updated – There is no automated monitoring of network-connected devices |
| D. Creating Service Ticket for Indicators of Compromise | – SIEM/SOAR platform must automatically send emails to the service desk ticking software which autogenerates service desk tickets | – A ‘service desk’ user needs to be created in the SIEM/SOAR platform – The ‘service desk’ user must have ‘alarm notifications’ enabled and user email be set to service desk email address – Incoming emails for auto-generation of service desk tickets should be set to security group category | – Currently, there is no auto-generated alert. The security team must log into SIEM/SOAR platform to review alarms and events. |
| E. Isolating Compromised Endpoints | – A service desk ticket should be created when a system is isolated | – Correlation rules should be created to identify events which are deemed high enough criticality to warrant endpoint isolation – Response action should be taken to disable the endpoint (NIC) network interface card. | – Presently this process is manual and requires physically disconnecting the network cable. |
Appendix G: Controls Mapping
The following is a table of the five target tasks, outcome metric for outcome analysis, and to which NIST Cybersecurity Framework categories they were mapped.
| Task to be Automated | Outcome Metric | NIST CSF Category |
| A. Blocking phishing attacks | Mean Time to Recovery | PR.DS-5, PR.IP-7, RS.MI-2 |
| B. Remediating reoccurring unquoted paths vulnerability | Mean Time to Recovery | ID.RA-1, ID.RA-3, PR.IP-7, PR.IP-12, DE.CM-8, RS.MI-2 |
| C. Reporting on logins from unusual devices | Mean Time to Detection | ID.AM-1, PR.AC-1, PR.AC-4, DE.AE-1, DE.AE-2, PR.IP-7, PR.PT-1, DE.AE-5, DE.CM-1, DE.CM-3, DE.CM-7, RS.AN-1, RS.AN-3 |
| D. Creating service tickets for found indicators of compromise | Mean Time to Detection | RS.AN-1, ID.RA-2, ID.RA-3, PR.IP-7, PR.PT-1, DE.AE-3, DE.AE-4, DE.AE-5, DE.CM-4, DE.DP-4, RS.AN-3, RS.MI-3 |
| E. Isolating endpoints with high risk compromise | Mean Time to Recovery | ID.AM-5, ID.RA-3, ID.RA-4, ID.RA-6, PR.IP-7, PR.PT-3, PR.PT-4, DE.AE-3, DE.AE-4, DE.AE-5, RS.AN-4, RS.MI-1 |
Appendix G: Dependency Breakdown
| Task to be Automated | Steps | Deliverables |
| A. Blocking phishing attacks | 1) The user identifies and reports potential phishing email | 1) A notification is sent to the security platform for review. |
| 2a) Security platform reviews email links | The malicious link must be validated. If true, 1) a message sent to DNS proxy, 2) email system notified, 3) an ‘environmental awareness’ alarm is created in SIEM platform. | |
| 2b) Security platform reviews email attachments | Malicious attachment must be validated. If true, 1) a message sent to SIEM/SOAR or AV platform, 2) email system notified, 3) an ‘environmental awareness’ alarm is created in the SIEM platform. | |
| 3a) DNS proxy receives notice of valid malicious link. DNS proxy adds link domains to blocklists. | None | |
| 3b) Email system receives notice of confirmed phishing email. The email server is searched, and messages removed from user inboxes. | None | |
| 3c) SIEM/SOAR or AV platform is notified of possible malicious attachment. It searches for signatures of malicious attachment on endpoints. If found, attachment is quarantined | A ‘delivery and attack’ alarm is created in the SIEM platform if signature is found. | |
| B. Remediating reoccurring unquoted paths vulnerability | 1) Automation Engine runs PowerShell script to remediate unquote paths on all workstations and servers daily. | None. The scanning performed regular checks for automation task effectiveness independent of remediation actions. |
| 2) Vulnerability Scanner is run automatically twice a week on each endpoint. If this vulnerability is found, it is included in scanning reports available to the security team. | Scanning reports are created, but no automation steps proceed. The follow-up actions are up to security team members. | |
| C. Reporting on logins from unusual devices | 1) SIEM agent software is installed on all authorized workstations, which receive dynamic IP addresses. Servers are assigned static IP addresses. Agent software continually updates asset inventory details including IP and MAC addresses. | This tracks known workstations and provides a baseline for automating unusual device discovery. |
| 2) SIEM platform is configured to automatically scan each network segment daily to discover assets. | If a new asset is found, then it does not already have the SIEM agent. An ‘environmental awareness’ alarm is raised. | |
| 3) The SIEM platform automating sends a notification to the security team and service desk team of a potential unauthorized device connection. | None. The remaining discovery is left to security and service desk teams to investigate. | |
| D. Creating service tickets for found indicators of compromise | 1) Event logs are sent to the SIEM from a) DNS proxy ‘Cisco Umbrella’, b) endpoint SIEM agents, b) DLP solution ‘Digital Guardian’, c) anti-virus/endpoint protection ‘Cylance’, d) firewalls ‘Dell Sonicwall’, e) Linux and Windows servers, f) cloud productivity and email platform ‘office 365’, g) dark web monitoring service ‘spycloud’, h) network routers and switching spanning ports. | This data is normalized by the SIEM ‘AlienVault USM Anywhere’. |
| 2) Normalized data is analyzed by automated ‘machine learning’ and references open-source threat exchanges ‘OTX’ to identify indicators of compromise. | If IoCs are identified, alarms are created and assigned a threat category. [NOTE: this is the sole dependency for another task. Isolation of compromised endpoints requires that this automation be configured and functioning so that high-risk IoC’s can be identified and trigger endpoints isolation.] | |
| 3) Alarms trigger automated service desk ticket creation. | Email notifications are sent to the security, IT operations, and service desk teams regarding potential malicious actions. | |
| E. Isolating endpoints with high-risk compromise | An alarm is triggered for an indicator of compromise in one of the two alarm categories considered a high-risk; Exploitation and Installation, System Compromise. | High-risk IoC has been detected, and an alarm created. |
| The SIEM/SOAR platform uses its Forensics and Response integration to attempt to disable the compromised system’s network interface card. | All alarms trigger a service desk ticket, which notifies key staff of the issue. Alarms in the high-risk categories are created, the staff investigates to validate the IoC, if the network isolation happened, and if additional action is needed. |
Appendix H: Post-Implementation Support
The following list enumerates support activities outside of these platform services.
- Blocking Phishing Attacks – The security team will continue to monitor phishing email reports for false positives, and the Cisco Umbrella action logs as apart of daily operations. System Administrators will maintain the Mimecast email services and configurations as part of daily operations. Security staff will hold on-going training for the identification of phishing emails and how to use the reporting feature.
- Remediating Reoccurring Unquoted Path Vulnerability – System Administrators will maintain the RedHat 8.0 servers and Ansible Engine/Tower installations, and a GitHub repository for YAML playbook storage and control.
- Reporting Logins from Unusual Devices – Service Desk staff will install AlienVault Agent on new endpoints as deployment activities. Security Staff will monitor and continue to fine-tune the log ingestion and IoC identification criterion.
- Creating Service Ticket for Indicators of Compromise – Service Desk staff will monitor and respond to automated ticket creation. Security Staff will be included in ticket notification for automation related issues. System Administrators will support and administer the Jira service ticket servers and platform installation.
- Isolating Compromised Endpoints – Service Desk technicians support on-site support for workstations. ‘System Compromised’ service tickets would trigger an incident response action for the service desk and security teams.