According to a research report by CITIC Securities, on July 19, 2024, CrowdStrike configuration update errors caused 8.5 million Windows host systems to crash, seriously affecting the business operations of important industries worldwide (such as airlines/airports, trains, broadcasting companies, hospitals, financial institutions, government agencies, etc.), resulting in significant impacts such as flight suspensions, medical program cancellations/interruptions, and media shutdowns. At present, the relevant configuration errors have been fixed, and a comprehensive solution will take several days.
This incident may result in certain economic losses for CrowdStrike, as well as more severe reputation damage, and may cause existing and potential customers of CrowdStrike to reconsider their partnership. CrowdStrike's main competitors may benefit from this.
Event description: Endpoint security vendor CrowdStrike configuration update caused partial Windows system crashes.
At around 4:00 UTC on July 19, 2024 (around 12:00 Beijing time), endpoint security vendor CrowdStrike released an erroneous Falcon Sensor update (an agent used to monitor PC/virtual machine operating system activity, aimed at detecting and preventing potential threats), causing a large-scale shutdown of Windows systems worldwide where the agent was installed. Millions of PC/servers/virtual machines worldwide went offline due to a "blue screen of death" (BSOD) error. Due to CrowdStrike being a leading global provider of endpoint security products, and mainstream PCs worldwide being equipped with Windows systems, this erroneous update has affected banks, airlines, supermarkets, and television broadcasting companies, including aviation command systems leading to flight suspensions/forced landings, ticketing/ticket checking/settlement systems affecting various offline service scenarios, and even affecting the production processes of some factories. It is the most widespread IT incident in recent years.
Event cause: Falcon Sensor channel file update caused a logical error, resulting in the operating system crashing.
According to the technical blog on CrowdStrike's official website, on July 19, 2024 at 04:09 UTC, CrowdStrike released a Falcon Sensor configuration update to the Windows system, which is the core cause of system crashes and is not related to any network attacks. CrowdStrike states that the updated configuration file is referred to as a "channel file" and is part of Falcon Sensor's behavior protection mechanism. Updating channel files is a normal part of Falcon Sensor's operation, and CrowdStrike updates multiple times a day based on newly discovered tactics, techniques, and strategies. The affected channel file this time is 291, and its file name is& quot; C-00000291- " Start with. sys extension and end with. sys extension. Channel file 291 is used to evaluate the execution of named pipes in Windows systems (named pipes are used for inter process or inter system communication in Windows systems). This update aims to target newly observed malicious named pipes, but the configuration update caused a logical error that led to the operating system crashing.
Remedial measures: The configuration error has been fixed, but the downtime issue needs to be gradually resolved.
According to the technical blog on the official website, the configuration update that caused the Windows system to crash has been fixed at 05:27 UTC on July 19th. The company stated that this impact does not involve Linux and macOS hosts, and Windows hosts launched after 05:27 UTC are also not affected. For the affected hosts, the company provides different solutions in different situations: 1) Prioritize restarting the host in a wired network (rather than WiFi) environment to give it a chance to download the recovered channel files. 2) If the system still crashes after restarting, you need to boot Windows into safe mode or Windows recovery environment, navigate to the CrowdStrike directory of the operating system volume, find channel file 291 and delete it, and then restart the host from a shutdown state. 3) For hosts using BitLocker encryption, it is usually required to enter a recovery key when entering secure mode to ensure system security. In public cloud or virtual environments, users can achieve batch recovery through automated scripts. But for physical servers or PC devices, recovery can only be achieved through manual input by IT administrators, and the recovery cycle will be longer. Overall, we conclude that some hosts can be quickly restored, while a comprehensive solution will take several days.
Subsequent impact: CrowdStrike may face economic and reputational losses, and the landscape of endpoint security market may also change.
According to Microsoft's official website, approximately 8.5 million Windows hosts were affected this time, accounting for about 20% of CrowdStrike's server points. According to the convention of software company contract signing, it is usually not necessary to compensate customers for direct economic losses. However, service level agreements (SLAs) are generally set up in contracts, requiring service availability time, response time, and resolution time. If CrowdStrike fails to meet these requirements, they will need to provide certain compensation to customers or offer SLA points to offset future service fees. At the same time, the company also needs to increase public relations/brand/repair related expenses and bear the loss of reputation and brand image. Although downtime incidents themselves are not isolated, AWS, Azure, Atlassian (April 2022), and Datadog (March 2023) have all caused similar incidents. However, considering the extensive impact of this incident, we believe that the related damages will also be more severe. After this incident, it may cause existing and potential customers of CrowdStrike to reconsider their partnership, and CrowdStrike's main competitors may benefit from it.
Risk factors:
The rise in crude oil prices poses a risk of further uncontrolled high inflation in Europe and the United States; The risk of rapid upward trend in US bond interest rates; Continuous tightening of policy regulation for technology giants poses risks; The risk of global macroeconomic recovery falling short of expectations; Macroeconomic fluctuations lead to lower than expected IT expenditures by European and American companies, posing a risk; The evolution of security platformization falls short of expected risks; The global cloud computing market is facing unexpected risks in terms of development; Cloud computing enterprise data leakage and information security risks; The industry competition continues to intensify, and there are risks involved.
Investment Strategy:
CrowdStrike configuration update error caused 8.5 million Windows host systems to crash, seriously affecting the business operations of important industries worldwide (such as airlines/airports, trains, broadcasting companies, hospitals, financial institutions, government agencies, etc.), resulting in significant impacts such as flight suspensions, medical program cancellations/interruptions, and media shutdowns. At present, the relevant configuration errors have been fixed, and a comprehensive solution will take several days. This incident may result in certain economic losses for CrowdStrike, as well as more severe reputation damage, and may cause existing and potential customers of CrowdStrike to reconsider their partnership. CrowdStrike's main competitors may benefit from this.