On April 10, 2024 Palo Alto Networks Product Security Incident Response Team (PSIRT) learned of a suspicious exfiltration attempt at a customer site from Volexity's Steven Adair. Our Palo Alto Networks Product Security Research Lead Christopher Ganas and Unit 42's Threat Research Lead Kyle Wilhoit immediately investigated the issue with Volexity's team. They quickly determined that the suspicious traffic originated from the firewall and reflected the exploitation of a likely new zero-day vulnerability with a compromised firewall.
In the next few hours, our team assembled experts from across the company and took quick, decisive action as part of the company’s established protocols and industry best practices. We performed forensic investigations to identify the root cause of the vulnerability, understand the exploited payload tactics, and determine various options to enable protections in our product. Further, we explored workarounds and threat prevention signatures and determined the exact combination of configurations that made the system vulnerable to a compromise.
Within 24 hours of confirming the vulnerability, we released tested mitigations that blocked the known attacks, enabling immediate protections for our customers. We see these mitigations have been applied on about 90% of active susceptible devices.
Palo Alto Networks responded with an all-hands-on-deck approach to provide a quick and complete remediation to prevent further attacks while ensuring a comprehensive solution.
What Was the Problem?
The intricate vulnerability stems from a combination of two bugs in PAN-OS. In the first one, the GlobalProtect service did not sufficiently validate the session ID format before storing them. This enabled the attacker to store an empty file with the attacker's chosen filename. The second bug (trusting that the files were system-generated) used the filenames as part of a command. While neither bug provides for significant system damage, the combination allows unauthenticated remote shell command execution.
How Was It Exploited?
A highly sophisticated threat actor discovered that by uniquely combining the two bugs, they could perform a two-stage attack to achieve command execution on the vulnerable device.
In stage 1, the attacker sends a carefully crafted shell command instead of a valid session ID to GlobalProtect. This results in creating an empty file on the system with an embedded command as its filename, as chosen by the attacker.
In stage 2, an unsuspecting scheduled system job that runs regularly uses the attacker-provided filename in a command. This results in the execution of the attacker-supplied command with elevated privileges.
Successful stage 1 does not necessarily mean that the attacker's command was executed. Rather, it simply means that the attacker created an empty file with a weird name that does not damage the firewall by itself.
A system compromise requires a successful exploitation of a command that does some damage to the system, such as exfiltrating sensitive configuration details or downloading malware. Volexity and Unit 42 Threat Brief have more information about the type of malware seen in these attacks and indicators of threat activity.
How Do We Block Exploitation?
A Threat Prevention signature with Threat ID 95187 (released on April 11, 2024) detects and blocks, with 100% accuracy, all known and observed suspicious patterns in session IDs. This Prevention signature was released by Palo Alto networks within a day of confirming the vulnerability. We see approximately 90% of of susceptible devices are already protected.
In the following content updates, Threat IDs 95189 and 95191 were released to optimize and extend mitigations against other theoretical attacks that have not been observed so far. We recommend that customers continually stay up to date with their threat prevention content packages to ensure the most comprehensive protections.
Given our comprehensive code review, no other known changes are planned for our threat prevention signature for this issue. It is important to ensure Threat Prevention is appropriately enabled following the guidance at https://live.paloaltonetworks.com/t5/globalprotect-articles/applying-vulnerability-protection-to-globalprotect-interfaces/ta-p/340184
What Does Disabling Telemetry Do?
Disabling Telemetry prevented the system cron job from running, preventing the execution of the command, preventing a compromise. This completely prevented both currently known and observed attempted exploits from working.
As with similar issues, as the situation evolved, Palo Alto Networks and third-party researchers quickly investigated the vulnerability and how it could theoretically be exploited. In that process, we discovered additional ways to exploit the vulnerability that did not require telemetry to be enabled on a device to achieve a successful compromise.
We advise customers not to rely only on disabling telemetry as an interim mitigation.
How Did We Fix It?
The fix effectively removes the two problems in code that enabled this vulnerability to manifest. First, the session IDs are sufficiently validated before being stored. Then, the code that enabled command injection was rewritten using defensive programming techniques.
The fixes have been well tested by Palo Alto Networks Product Security Research Team and third-party research companies, like Bishop Fox, and found to be 100% effective in preventing this vulnerability.
For details on available fixes, please refer to our security advisory.
What Else Is Palo Alto Networks Doing?
Helping Customers Stay Protected
Palo Alto is closely monitoring the uptake of fixes and mitigations and reaching out to customers who have yet to address the issue.
Since the fix was released, Palo Alto Networks improvised its telemetry collection to help capture additional necessary logs, while monitoring new telemetry data to identify any unknown threat vector. This process helps us stay one step ahead of the threat actor.
As new information surfaces, we constantly update our security advisory and Unit 42 Threat Brief to share new IoCs – IPs that customers can use to detect and block threat activity.
Helping with the Diagnosis
To facilitate the investigation of potential compromises, our support system that monitors Tech Support Files (TSF) was quickly reengineered to detect and report traces of evidence of threat activity automatically. TSFs meant for system diagnosis and troubleshooting allow customers to export logs and diagnostic data from a firewall. TSFs provide clues to either an attempted or a successful exploitation of the vulnerability.
TSF automated analysis included searching for log entries, such as those in our security advisory, which indicated potential attempted exploitation of the Stage 1 issue. TSF analysis results now indicate the level of probing or compromise seen in the analysis.
In the event that such entries were identified, TSFs were manually searched for indicators of compromise (IOC). This included successful exploitation of both stage 1 and/or stage 2. For the TSFs that had confirmed IOCs, Unit 42 and Customer Support were involved immediately for proactive outreach.
As a general practice, customers upload a TSF when they open a case with Customer Support. Palo Alto Networks proactively looked at all the TSFs submitted for unrelated cases by the customers in the past 6 months to identify any customer impact. If a customer was found to be impacted, Unit 42 proactively notified the customer.
The Security advisory for CVE-2024-3400 provides checks that can be run on the device to look for evidence of attempted exploit activity. Please refer to Unit 42 Threat Brief for understanding the observed threat activity.
Helping with the Mitigations
Palo Alto Networks is also offering a free trial of Advanced Threat Prevention to customers who do not already have a subscription, to mitigate the issue until a fix is applied. We have also offered Unit 42 incident response services to potentially impacted customers to help them investigate this vulnerability.
A Choice of Multiple Upgrade Paths
Additional hotfixes are available for other commonly deployed maintenance releases to provide the most seamless upgrade path for customers. These are for customers who do not want to upgrade to a fixed version but are looking for an incremental spot fix to address the vulnerability.
Preventing Such Issues
Per our standard product security assurance process, we are performing a Root Cause Analysis to further identify these issues, ensure that they are identified and addressed during development, and continue to enhance product design, including resistance to such attacks.