ONLY Cortex Delivers 100% Protection and Detection in MITRE Engenuity

This post is also available in: 简体中文 (Chinese (Simplified)) 繁體中文 (Chinese (Traditional)) Français (French) Deutsch (German) Italiano (Italian) 日本語 (Japanese) 한국어 (Korean) Português (Portuguese (Brazil)) Español (Spanish)

The 2023 MITRE Engenuity ATT&CK Evaluation results are in, and only Cortex XDR delivers 100% protection and 100% analytic coverage with zero configuration changes and zero delayed detections.

Listen to the 2023 MITRE ATT&CK Results

The results of the fifth round of the MITRE Engenuity ATT&CK Evaluations have been released, and while every vendor is claiming 100% of everything, the truth is in the details, and the data does not lie. This year, Cortex is the only vendor with 100% prevention and analytic detection.

In the ever-evolving landscape of cybersecurity, staying one step ahead has never been more critical. Adversaries continuously seek new and sophisticated ways to breach defenses. It's a high-stakes game where the rules constantly change, and the consequences of falling behind can be catastrophic.

This is where MITRE Engenuity enters the scene with their Enterprise ATT&CK Evaluations, offering a beacon of clarity amidst the chaos. These evaluations have become an invaluable resource for CISOs, security professionals and anyone tasked with safeguarding an organization's digital assets. They provide a litmus test or a performance review of how the industry’s endpoint security solutions hold up in the face of the most resourceful cyber saboteurs.

But, what exactly are the MITRE Engenuity ATT&CK evaluations, and why should you care about the results? These annual evaluations emulate the tactics, techniques and procedures (TTPs) employed by some of our time's most active and notorious threat groups. The MITRE Engenuity red team invites cybersecurity solution providers to defend against carefully choreographed attacks (in both detect-only and prevent modes) to provide insight into three capabilities:

1. Visibility – what can a solution see
2. Detection – what actions can a solution accurately identify as malicious
3. Protection – what malicious actions can a solution prevent

A staggering number of vendors participate in these evaluations – 29 this year – which is a tribute to their value and reflects the painstaking efforts that MITRE Engenuity pours into the assessments to ensure they provide a challenging and provocative engagement.

Round 5 (Turla)

This year marks the fifth annual evaluation, and the MITRE Engenuity red team focused on emulating the methods of Turla, a threat group our Unit 42 threat researchers have studied extensively. Turla is an extraordinarily well-funded and sophisticated Russian-based threat group that has infected victims in over 45 countries. They have targeted government agencies, military groups, diplomatic missions, as well as research and media organizations. Turla’s infamy stems from its covert exfiltration tactics, including water holing of government websites, custom rootkits, elaborate command-and-control network infrastructure and deception tactics. In speaking with defenders who participated, it is clear that MITRE Engenuity took a great leap forward in the sophistication of their attack methods this year.

The blue team deployed the Cortex XDR Pro for Endpoint agent on both Windows and Linux endpoints. No additional solutions were deployed, and Cortex XDR was configured with default settings as it would be out of the box, with the only changes enabling the quarantining of malicious files and, for Linux, enabling the option to treat grayware as malware.

Key Metrics and Evaluation Criteria

This year's evaluation was broken down into two detection-only scenarios, named Carbon and Snake, corresponding to notable tools created and used by Turla. A protection phase followed this, mirroring the techniques in the detection tests, with enough entropy injected not to look identical to the detection test. The two detection scenarios each had 10 steps, consisting of multiple substeps that map to actual techniques in the MITRE ATT&CK framework. Combined, there were 143 substeps that each vendor had the opportunity to see. For each of these substeps, the MITRE Engenuity team recorded whether each solution had a detection for the action taken.

Each of these detections was categorized based on the detection quality observed.

MITRE Engenuity Detection Categories:

• Not Applicable – Participants didn’t have visibility on the system under test. (This would be the case if they opted out of the Linux evaluation.)
• None – There was no detection. (No telemetry associated with the malicious activity)
• Telemetry – Minimally processed data collected by the capability showing that event(s) occurred. (Detections of this type are usually basic logging of activity.)
• General – The abnormal event was detected. However, an ATT&CK tactic (or equivalent context) was not specified. (Detections of this type leave the security analyst to investigate and determine what action was taken and why.)
• Tactic – An ATT&CK tactic (or equivalent context) of the abnormal event was specified. (Detections of this type assert why an action occurred, but again leave the security analyst to investigate what action or technique was taken.)
• Technique – An ATT&CK technique (or equivalent context) of the abnormal event was specified. (Detections of this caliber provide the context and details required to answer why an adversary performed an action and precisely what action they used to achieve their objective.)

The MITRE Engenuity effectively identifies two types of coverage: Telemetry Coverage and “Analytic Coverage.” Telemetry Coverage is defined as the number of substeps where a solution produces a Telemetry detection as its highest value detection. Analytic Coverage is defined as the number of substeps that contain either a General, Tactic or Technique detection.

It is also worth noting that all detections might have one of two detection modifiers:

• Configuration Changes – A configuration change modifier is noted if the detection observed was achieved on the fourth day of testing, a day when vendors are given a second chance to detect malicious activity they missed in the initial test.
• Delayed Detections – A delayed detection modifier is noted if the detection is not observed in a timely manner, meaning a significant delay occurs when the action is taken, and when the detection is observed in the vendor’s console.

For the Protections scenario, there were 129 substeps organized into 13 major steps. In the case of these substeps. Each Protection substep was either blocked or not blocked and recorded as one of the following:

• Protected – The malicious activity was blocked.
• Not Applicable (Protected) – This occurred when a prior substep in the prevention step was blocked, and thus subsequent substeps could not be executed.
• None – This means the malicious activity was not blocked.

How Did Cortex XDR Perform?

The purpose of these evaluations is to provide insight into three capabilities:

1. Visibility – what can a solution see
2. Detection – what actions can a solution accurately identify as malicious
3. Protection – what malicious actions can a solution prevent

Cortex XDR stands alone in providing 100% Protection while delivering 100% Visibility and 100% Analytic Coverage (detections) with zero configuration changes or delayed detections.

In addition, the quality of our detections is unparalleled, with 142 of 143 detections as Technique level detections – the highest-quality detection possible. The one other detection was recognized as a Tactic level detection. Every one of the 129 substeps in the Protection evaluation was blocked. All of this was accomplished with zero “configuration changes” and zero “delayed detections.” In fact, if we exclude detections resulting from a configuration change, Cortex XDR was the only vendor with no missed detections (detection type, None). In other words, Cortex XDR was the only one with 100% visibility.

The results from this year’s evaluation serve to reflect the tremendous amount of effort that Palo Alto Networks continues to pour into both adversarial research and engineering in endpoint security, putting that knowledge to work to help our customers remain safe in the face of an increasingly hostile cyber world.

A Note about 100%

If you’ve been following the ATT&CK evaluations for some time, 100% might seem commonplace to you. But don't get confused; when other solutions claim to have achieved 100% in this year's evaluation, they have at least one detection and/or prevention in all the major steps. Only Cortex provided these detections for every one of the individual malicious actions that were taken – the substeps.

100% detection and prevention is the bar we should all strive for. As Forrester’s Allie Mellen pointed out after last year’s evaluation, “detecting an attack inspired by a known threat actor should be the floor for what security products can do, not the ceiling.”1 We completely agree, and Cortex XDR is setting the bar. 

That said, Allie also rightly pointed out that perhaps 100% detection isn’t necessarily a good thing as solutions that “are detecting on every technique, they have the potential to be noisy, have a high rate of false positives, and provide excessive alerting.” We also share this perspective, and this is why we heavily leverage machine learning to understand which activities represent normal operations and which are potentially malicious actions. In the case of Cortex XDR, this is only strengthened by the fact that we integrate additional telemetry from many other key data sources that critical context clues to help determine where you should focus your investigations. 

Exploring the ATT&CK Evaluation Results

Hats off to the MITRE Engenuity, as this year was significantly more challenging than years past. In addition, we are thrilled with the improvements they have made to help defenders make informed decisions. The ATT&CK evaluation results, hosted on MITRE Engenuity’s site, now allow you to compare the results for three vendors side by side. You can choose each of the three evaluation scenarios and up to three vendors. By default, the results are shown with delayed detections and those resulting from configuration changes included. Still, you can also view the results with detections with those modifiers removed.

We’ve heard for many years that it is difficult to understand the ATT&CK evaluation results as no data visualization tool allows you to view the results across all vendors. To aid in that effort, we’ve created a tool to help you explore this year’s results, as well as all previous years. With this tool, you can select the vendors you are interested in and the particular adversaries that have been emulated. All of the resulting charts are built directly from the JSON files provided by MITRE Engenuity for each participating vendor. Explore the ATT&CK Evaluation results for yourself and let the data reveal the best choice for defending your business.

1. Forrester®, and Allie Mellen. “Principal Analyst.” MITRE ATT&CK Evals: Getting 100% Coverage Is Not As Great As Your Vendor Says It Is, Forrester®, 2022, https://www.forrester.com/blogs/mitre-attck-evals-getting-100-coverage-is-not-as-great-as-your-vendor-says-it-is/. Accessed 19 September 2023.