respond to the changing threat landscape at operational speed .
Why it happened ?
The issue on Friday involved a Rapid Response Content update with an undetected error .
Rapid Response Content is used to perform a variety of behavioural patternmatching operations on the sensor using a highly optimised engine . Rapid Response Content is a representation of fields and values , with associated filtering . This Rapid Response Content is stored in a proprietary binary file that contains configuration data . It is not code or a kernel driver .
Rapid Response Content is delivered as Template Instances , which are instantiations of a given Template Type . Each Template Instance maps to specific behaviours for the sensor to observe , detect or prevent . Template Instances have a set of fields that can be configured to match the desired behaviour .
In other words , Template Types represent a sensor capability that enables new telemetry and detection , and their runtime behaviour is configured dynamically by the Template Instance that is Rapid Response Content .
Rapid Response Content provides visibility and detections on the sensor without requiring sensor code changes . Threat detection engineers use this capability to gather telemetry , identify indicators of adversary behaviour and perform detections and preventions .
Rapid Response Content is behavioural heuristics , separate and distinct from CrowdStrike ’ s on-sensor AI prevention and detection capabilities . Rapid Response Content is delivered as content configuration updates to the Falcon sensor .
How it happened ?
On July 19 , 2024 , two additional IPC Template Instances were deployed . Due to a bug in the Content Validator , one of the two Template Instances passed validation despite containing problematic content data .
Based on the testing performed before the initial deployment of the Template Type , trust in the checks performed in the Content Validator , and previous successful IPC Template Instance deployments , these instances were deployed into production .
When received by the sensor and loaded into the Content Interpreter , problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception . This unexpected exception could not be gracefully handled , resulting in a Windows Operating System BSOD crash .
The channel file responsible for system crashes on Friday , July 19 , 2024 beginning at 04:09 UTC was identified and deprecated on operational systems . When deprecation occurs , a new file is deployed , but the old file can remain in the sensor ’ s directory . Out of an abundance of caution , and to prevent Windows systems from further disruption , the impacted version of the channel file was added to Falcon ’ s known-bad list in the CrowdStrike Cloud .
No sensor updates , new channel files , or code was deployed from the CrowdStrike Cloud . For operational machines , this is a hygiene action . For impacted systems with strong network connectivity , this action could also result in the automatic recovery of systems in a boot loop .
Future remediation
CrowdStrike makes changes to testing
• Local developer testing
• Content update and rollback testing
• Stress testing , fuzzing and fault injection
• Stability testing
• Content interface testing
• Add additional validation checks to Content Validator
• Enhance existing error handling in the Content Interpreter
CrowdStrike makes changes to content deployment
• Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base , starting with a canary deployment .
• Improve monitoring for both sensor and system performance , collecting feedback during Rapid Response Content deployment to guide a phased rollout .
• Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed .
Out of an abundance of caution , the impacted version of the channel file was added to Falcon ’ s known-bad list in the CrowdStrike Cloud .
INTELLIGENT TECH CHANNELS 41