Published on July 22, 2024 at "The Asian Mirror"

On 19th July 2024 a major Global IT outage occurred due to an error in a software update from CrowdStrike. This event known as the “2024 CrowdStrike incident” or “Y2K24” was a significant moment in IT history. The disruption affected 674,620 direct relationships between CrowdStrike, Microsoft and their customers and indirectly impacted over 49 million, as per data from Interos. This report provides a detailed look at what happened, investigates why it happened and discusses what the information technology community can learn from it.

Incident Timeline:

• Early Morning, July 19th: As part of its ongoing security operations, CrowdStrike rolls out a routine “sensor configuration update” for its Falcon platform on Windows systems. This update, intended to enhance endpoint protection, inadvertently triggers a chain reaction.
• Mid-Morning: Reports of system crashes and malfunctions begin to surface globally. Airlines ground flights due to issues with check-in systems and air traffic control software. Hospitals experience delays in patient care as electronic medical records become inaccessible. Financial institutions grapple with frozen transactions and disrupted online banking services.
• Afternoon: The outage reaches its peak, causing widespread disruption. Communication networks become overloaded, hindering information flow and emergency response efforts. The ripple effects are felt across industries, causing significant economic losses and public anxiety.
• Late Afternoon/Evening: CrowdStrike identifies the faulty update as the culprit. After a period of intense troubleshooting, they deploy a patch to address the issue. Recovery efforts begin across the globe, a slow and laborious process.

Unveiling the Culprit:

The root cause of the outage was a logic error embedded within the CrowdStrike update. This error resided in the configuration update for the Falcon sensor, a critical component responsible for monitoring and protecting endpoints. When the update encountered specific system configurations or malicious techniques, the logic error triggered abnormal behavior, ultimately leading to system crashes.

You Can Also Read: Most common cyber security threats to startups

Technical Breakdown:

• Affected Systems: The update specifically impacted devices running the Falcon sensor for Windows version 7.11 and above. However, only those devices online between 4:09 UTC and 5:27 UTC on July 19th were susceptible, as the faulty update was rolled out during that timeframe.
• Sensor Configuration Updates: CrowdStrike utilizes “channel files” to deliver updates to the Falcon sensor. These files, updated several times a day, provide the sensor with information on the latest threats and vulnerabilities discovered by CrowdStrike. The faulty update targeted a specific channel file (Channel File 291) associated with the platform’s behavioural protection mechanisms.
• The Logic Error: The update was designed to address newly observed malicious techniques employed by cybercriminals. These techniques involved the abuse of “named pipes,” a communication mechanism used in some Command and Control (C2) frameworks. C2 frameworks allow attackers to remotely control compromised systems. While the update aimed to safeguard against such abuse, the logic error within it created a system vulnerability when encountering these techniques.
• C2 Frameworks and Named Pipes: C2 frameworks are a critical tool in an attacker’s arsenal, allowing for remote control, data exfiltration and lateral movement within compromised networks. Named pipes are a type of inter-process communication mechanism that can be exploited within C2 frameworks. The intended purpose of the update was to enhance the Falcon sensor’s ability to detect and prevent the abuse of named pipes in C2 attacks. However, the logic error caused the sensor to malfunction when encountering these techniques, leading to system crashes.

Impact Analysis:

The outage triggered a global cascade of disruptions, causing significant financial losses, operational challenges and public anxiety.

• Financial Losses: Businesses across various sectors experienced downtime, hindering transactions, impacting productivity and incurring significant costs. Estimates suggest billions of dollars in lost revenue due to the outage.
• Operational Disruption: Critical services like airlines, hospitals and communication networks were severely disrupted. Airlines grounded flights, hospitals postponed surgeries and communication networks became overloaded, hindering emergency response efforts. These disruptions had a domino effect on various industries, causing widespread economic and social consequences.
• Public Anxiety: The outage exposed the heavy reliance on technology within modern society. The disruption of essential services like healthcare, communication and financial transactions created frustration and a sense of vulnerability amongst the public. The event highlighted the need for robust cybersecurity solutions and contingency plans to ensure the resilience of critical infrastructure.

Lessons Learned:

The CrowdStrike incident serves as a stark reminder of the importance of robust software development and testing practices in the cybersecurity industry. While CrowdStrike acted swiftly to identify the issue and deploy a patch, the outage underscores the need for heightened vigilance and proactive measures.

Thorough Testing: The importance of rigorous testing procedures cannot be overstated. CrowdStrike likely has established testing protocols, but this incident highlights the need for even more robust procedures. Employing a combination of unit testing, integration testing and scenario-based testing can help identify and eliminate logic errors before updates are deployed. Additionally, compatibility checks across various system configurations can help mitigate the risk of unforeseen issues.

Transparency and Communication: Clear and timely communication during outages is essential for minimizing disruption and restoring trust. CrowdStrike’s prompt identification of the issue and release of a patch are commendable actions. However, ongoing communication throughout the incident, keeping stakeholders informed about the situation and the progress of recovery efforts, would have further minimized anxiety and facilitated a smoother recovery process.

Vendor Reliance: The incident highlighted the potential pitfalls of overdependence on a single vendor for critical security software. Diversifying security solutions and fostering industry collaboration can help mitigate such risks. Organizations should consider employing a layered security approach, utilizing solutions from multiple vendors to address different aspects of their security needs. Additionally, collaboration between cybersecurity vendors and industry stakeholders can foster the sharing of threat intelligence and the development of more robust security solutions.

Incident Response Planning: Having a well-defined incident response plan in place is crucial for efficient recovery during outages. This plan should outline protocols for identifying the issue, deploying fixes, communicating with stakeholders and minimizing downtime. Regularly testing and updating the incident response plan ensures its effectiveness during a real-world event. Organizations should also consider conducting tabletop exercises to simulate potential outages and test the response plan.

The Evolving Threat Landscape: Cybercriminals are constantly developing new tools and techniques to exploit vulnerabilities. Security solutions need to be adaptable and capable of evolving alongside the threat landscape. CrowdStrike’s update aimed to address a newly observed technique, but the logic error within it exposed a vulnerability in the system. This underscores the need for continuous threat intelligence gathering, vulnerability assessments and security updates to ensure that defenses remain effective.

The Road Ahead:

The CrowdStrike incident serves as a valuable learning experience for the cybersecurity community. By prioritizing thorough testing, fostering transparency, diversifying security solutions and continuously adapting to the evolving threat landscape, we can build a more resilient digital infrastructure and minimize the impact of future outages.

This incident also highlights the importance of ongoing collaboration between cybersecurity vendors, industry stakeholders and governments. By sharing threat intelligence, developing best practices and promoting robust security standards, we can collectively create a safer digital environment for everyone.

Related posts:

Impacts of Video Games on Kids and Cyber Crime Issues