Get in Touch Close Menu

How to reduce security alert fatigue

27 July 2022

Alerting is essential to cybersecurity. 

However, alerting can also be an overwhelming aspect of cybersecurity. A never-ending set of alerts that require investigating can cause alert overload. So how do you reduce security alert fatigue?

An effective Managed Security Information and Events Management (SIEM) system, paired with the skill set of a 24/7 Security Operations Centre team, will detect and communicate ‘true-positive’ security incidents to customers, as well as provide recommendations which facilitate speedy mitigation. 

cyber security alert fatigue

Conversely, a poorly optimised SIEM may risk over-burdening customers with false-positives.

True-positives are Alarms that fire within a SIEM and represent a genuine security threat.

Conversely, false-positives are Alarms that fire within a SIEM and do not represent a genuine security threat.

‘Tuning’ is an important, proactive, and reactive action performed within the SOC to reduce the likelihood of ‘false-positive’ alarms triggering and ultimately being communicated to the customer.

Why Tuning works

Effective tuning can be implemented by an experienced SOC team that has experience in dealing with a wide variety of products and tooling. 

When conducting tuning, it is critical to strike a balance between being too aggressive, which risks losing visibility, versus not being bold enough, which risks introducing alert fatigue to both customer and analyst.

Take the following example workflow of tuning. The SOC is alerted to several non-admin-level accounts being added to a sensitive-looking Active Directory group with administrative-level privileges. 

too many alerts

As per MITRE ATT&CK ID T1098 and on a variety of other factors, this may be something we wish to notify our customers of. Upon raising this to a customer, they ultimately confirm this activity is benign. 

Mitre informed us that what appeared to be an administrative group does not possess administrative-level privileges of concern to the customer. 

The SOC can then act on this information by excluding this group name from producing further alarms in the future.

reduce alert fatigue

In a global study performed by Dimensional Research in 2020 into the ‘state of security operations’, customers are experiencing false-positives and a lack of tuning. These include:

  1. The top issue reported with existing SIEM solutions is the high number of alerts
  2. 99% report high volumes of alerts cause problems for IT security teams
  3. 83% of security staff reportedly experienced ‘alert fatigue due to an unmanageable volume of alarms
  4. 70% of security staff reported a doubling in the volume of security alerts they receive over the last five-year period, caused namely by:
  • New and evolving threats
  • Additional security monitoring and controls were applied (often as a result of auditing)
  • Growth of business apps and services
  • Growth in user endpoints
  • Expanding attack surfaces

How does Sapphire fix these issues?

As a Managed SIEM provider, Sapphire works to address these highlighted problems internally to ensure our customers do not experience these same pitfalls.

To ensure our customers only ever receive a manageable and, most importantly, valuable quantity of alerts in a method which suits their staffing levels, Sapphire’s SOC team emphasise the following key areas:

1. Tuning

A common misconception is that a SIEM is a ‘tick-box’ purchase with a ‘set it and forget it mentality.

Sapphire’s SOC team understands the dangers of this approach. In all but a few instances, customer networks are in a constant state of flux.

This may be due to firewall changes, onboarding new servers and hosts, merging companies, or automated server rebuilds. Amidst this continually changing landscape, our SOC staff must react quickly.

Specifically, this involves an efficient and coordinated response across our Tier 1, Tier 2, and Tier 3 analysts to:

a) identify potential false-positives

b) be mindful of the ramifications of potential tuning

c) perform granular ‘tuning’ to prevent these events from landing in our customers’ inboxes

2. Tiered Cases

When Sapphire detects a security violation, our SOC analysts document this in a ‘Case’ along with supporting evidence. The violations are communicated to the customer. To help our customers prioritise the cases they receive, our analysts assign cases a ‘Priority’ level based on the level of risk posed to the customer’s network.

Cases with a higher risk should be treated more urgently than those with a lower Priority.

3. Custom Notification Policies

We understand that each customer is different and that their quantity of IT staff and experience can vary greatly. As such, we collaborate with our customers to ensure we communicate cases most effectively. For example, customers with a more robust IT team may choose to receive cases of all priority levels immediately.

Conversely, customers with limited resources prefer only critical cases to be sent directly, with all others sent weekly or monthly reports. This strategy can provide a more manageable workload for resource-strapped IT departments.

4. MITRE ATT&CK Enterprise Framework Mapping

MITRE’s Enterprise Framework maps the techniques used by threat actors, beginning with the ‘Reconnaissance’ phase through to the ‘Execution’ phase.

All of Sapphire’s detections are tightly integrated with this framework. The framework greatly assists with the tuning process, as it allows us to be aware of causing potential ‘blind spots.

For example, under MITRE’s ‘Defence Evasion’ technique, the SOC has numerous detection rules in place, such as: ‘BITS Job Detection’, ‘Defence Impairment Attempt’, and ‘Indicator Removal on Host’. Should one rule begin triggering false positives within a customer’s network, we are confident that our tuning will not significantly reduce visibility as we have additional Rules acting as a ‘catch-all’. Akin to the ‘Defence-in-Depth’ approach, we call this the ‘Detection-in-Depth’ approach.

5. Building Custom Detection Rules

We also understand that our customers may have unique monitoring requirements for their network that may require us to create custom rules. For example, regulatory or compliance standards may dictate some non-standard monitoring requirements.

By collaborating closely with our customers, Sapphire can create highly focused rules that will alert only when a threshold is breached, per a customer’s requirements. This rule is another example of how we aim to eliminate false positives and make our customers’ lives a little easier.

Sapphire’s Cybersecurity managed services

Whether Managed SIEM, Managed Endpoint Detection & Response or Managed Threat Intelligence service, a proactive, efficient, and experienced SOC team is essential in delivering an effectively tuned and valuable security service.


Thank you to Sapphire’s SOC Analyst, Shane, for this guest blog.

Related Articles

Cyber Security Risk Management: A Detailed Guide
20 March 2023

The increased digitisation of our world means the threat of cyberattacks and data breaches continues to grow. No organisation is immune to the risks of cybersecurity threats. In fact, a recent study shows the average time to identify and contain a data breach is 277 days, at an average cost of $4.35 million. That’s why cyber […]

Find Out More
What Is UEBA? User and Entity Behaviour Analytics Guide

Traditional security measures to deal with cybersecurity threats are no longer enough to protect a company’s sensitive data and assets. Therefore, companies need a solution that can detect and respond to potential threats in real time, and that’s where user and entity behaviour analytics (UEBA) comes in. In this article, we’ll explore UEBA in more […]

Find Out More
Web Firewall Application: Securing Online Applications

Application layer attacks or DDoS (Denial of Service Attacks)are the leading cause of breaches. However, a web application firewall (WAF) prevents malicious traffic from accessing web applications. While a web application firewall is not meant to defend against all types of attacks, it is a great tool to have in your arsenal. Let’s look at […]

Find Out More