Skip to content

The Global Microsoft Outage Today – ft Crowdstrike 19 July 2024

On July 19, 2024, a significant IT outage disrupted Microsoft services globally, causing widespread chaos. This event, triggered by a faulty software update from CrowdStrike, resulted in operational failures across various sectors, from airlines to retail. Major companies experienced disruptions, leading to flight cancellations, retail system failures, and more. This blog post aims to provide a detailed examination of the outage, starting with an easy-to-understand overview and progressing to a more technical analysis. By understanding the causes, effects, and responses to this incident, we can better prepare for and mitigate future risks in our increasingly digital world.

Table of Contents

What exactly happened?

Summary of Events on July 19, 2024

On the morning of July 19, 2024, users worldwide began experiencing significant issues with Microsoft services. From business operations to personal communications, the outage quickly became apparent, causing frustration and confusion across different sectors. The disruption was traced back to a problematic software update from cybersecurity firm CrowdStrike, which had been deployed the previous night.

Initial Signs of the Outage

The first signs of trouble appeared as users reported issues with accessing Microsoft services, including Office 365, Azure, and Windows-based applications. Many encountered the notorious “blue screen of death” errors, rendering their systems unusable. Social media and support forums were flooded with complaints and queries, indicating a widespread problem.

Scope and Scale of the Impact

The outage was global, affecting millions of users. Critical sectors like aviation and retail were particularly hard hit. Airports experienced severe delays as check-in systems failed, leading to flight cancellations and long lines. Retailers faced disruptions in their point-of-sale systems, causing interruptions in sales and customer service. The scale of the impact underscored the deep integration of Microsoft services in modern business operations.

What is the Issue?

The root cause of the outage was identified as a faulty software update released by CrowdStrike, a leading cybersecurity firm. This update inadvertently introduced a bug into Windows systems, triggering widespread failures. The specific nature of the bug caused critical system errors, leading to the “blue screen of death” and preventing normal operations. CrowdStrike’s update, meant to enhance security, instead compromised the stability of millions of devices relying on Windows.

The combined effect of these factors led to a significant global disruption, highlighting the vulnerabilities in our interconnected digital infrastructure and the critical importance of robust testing and security measures in software deployment.

Why did it happen?

Detailed Analysis of the CrowdStrike Update Issue

The Microsoft outage on July 19, 2024, was primarily caused by a faulty software update from CrowdStrike, a prominent cybersecurity firm. This update, intended to enhance security features, inadvertently introduced a critical bug into Windows operating systems. The flaw in the update code led to system instability, triggering widespread failures.

Interaction with Microsoft Services

The CrowdStrike update had a significant impact on Microsoft services due to their deep integration with Windows OS. Microsoft services like Azure and Office 365 depend heavily on the stability and performance of the underlying Windows infrastructure. The buggy update affected core system files and processes, leading to crashes and operational failures across these services.

Technical Explanation of the Resulting Errors

The faulty update caused several technical issues:

  • Memory Corruption: The update introduced errors in the handling of system memory, causing data corruption and triggering the “blue screen of death” (BSOD).
  • System File Conflicts: Key system files were altered or corrupted, leading to conflicts and preventing essential services from running correctly.
  • Service Interruptions: Core services required for running Microsoft applications were disrupted, leading to widespread outages.

These errors cascaded through the system, affecting not only individual computers but also large-scale server environments running critical business operations. The resulting instability forced many businesses to halt operations temporarily until the issue could be resolved.

Who is Affected?

Major Companies Impacted

The outage had a profound effect on major corporations globally. Businesses that rely heavily on Microsoft services for their day-to-day operations experienced significant disruptions. This included financial institutions, healthcare providers, and tech companies, all of which faced operational hurdles due to the loss of critical services like email, cloud storage, and collaboration tools.

Airlines

The aviation sector was severely affected, with airlines such as American Airlines and Southwest Airlines encountering major issues. Their check-in systems, which rely on Microsoft services for booking, scheduling, and customer management, went down. This led to widespread flight cancellations, delays, and long lines at airports as staff struggled to manage the situation manually.

Retail Sector and Other Industries Affected

Retailers faced significant challenges as their point-of-sale systems and inventory management tools, many of which are integrated with Microsoft services, went offline. This disruption caused delays in transactions, affected customer service, and led to a temporary halt in sales. Other industries, including manufacturing and logistics, also experienced operational slowdowns as their digital infrastructure faltered.

Individuals and Businesses Relying on Microsoft Services

The outage impacted not only large organizations but also small businesses and individual users. Entrepreneurs, freelancers, and remote workers who depend on Microsoft 365 for email, document management, and communication were unable to perform their tasks. This led to missed deadlines, loss of productivity, and financial repercussions. Individual users experienced difficulties accessing personal files, emails, and other essential services, highlighting the pervasive dependence on Microsoft’s ecosystem in everyday life.

Services Affected

Microsoft Services Impacted

The outage on July 19, 2024, significantly impacted several core Microsoft services:

  • Azure: Microsoft’s cloud computing platform experienced widespread disruptions, affecting businesses that rely on it for their cloud infrastructure and services.
  • Office 365: This suite of productivity tools, including Outlook, Word, Excel, and Teams, was rendered unusable for many users, halting business communications and operations.

Specific Functionalities Disrupted

The outage caused specific disruptions in various industries:

  • Airlines: Check-in processes, flight scheduling, and customer service systems were severely affected, leading to delays and cancellations.
  • Retail Operations: Point-of-sale systems, inventory management, and online sales platforms experienced outages, causing interruptions in sales and customer service.
  • Healthcare: Electronic health records (EHR) systems and patient management tools were disrupted, affecting patient care and administrative functions.
  • Finance: Banking systems, trading platforms, and financial services relying on Azure and Office 365 experienced downtime, impacting transactions and financial operations.

The extensive reach of Microsoft’s services meant that the outage had a ripple effect across multiple sectors, highlighting the critical role of reliable IT infrastructure in modern business operations.

Immediate Effects

Operational Disruptions

The Microsoft outage led to significant operational disruptions across various sectors. Businesses reliant on Microsoft services faced immediate halts in productivity. Airlines had to cancel flights and experienced severe delays as their systems for check-in, scheduling, and customer management went offline. Retailers saw their point-of-sale systems and inventory management tools crash, causing interruptions in transactions and customer service.

Customer and User Experiences

Customers and end-users were met with frustration and inconvenience. Airline passengers dealt with long lines and flight cancellations, causing travel plans to be disrupted. Shoppers faced delays and difficulties in completing purchases, both in-store and online. For individuals and remote workers, the inability to access email, documents, and collaboration tools resulted in missed deadlines and reduced productivity.

Impact on Various Sectors

  • Aviation: The most visible impact was seen in the aviation industry, where check-in systems failures led to widespread delays and cancellations, stranding passengers and creating backlogs.
  • Retail: Retailers struggled with processing sales and managing inventory, leading to revenue losses and dissatisfied customers.
  • Healthcare: Healthcare providers encountered issues with accessing patient records and managing appointments, potentially affecting patient care.
  • Finance: Financial services faced interruptions in transactions and data processing, impacting trading and banking operations.

The immediate effects underscored the critical dependency on Microsoft’s infrastructure for the seamless operation of modern business and daily activities.

Aftermath of the Incident

Recovery Process and Timeline

The recovery from the outage began immediately after the issue was identified. Microsoft and CrowdStrike teams worked around the clock to diagnose and resolve the problem. By the end of the day, a patch was issued to correct the faulty update, and systems gradually began to restore normal operations. Full recovery took several days as businesses and individuals implemented the patch and verified their systems’ stability.

Long-term Implications for Businesses and Individuals

The incident highlighted the vulnerability of even the most robust IT infrastructures. Businesses are likely to re-evaluate their dependency on single-service providers and consider diversifying their technology stacks to mitigate risks. For individuals, the outage underscored the importance of regular system backups and contingency plans for accessing critical information and services.

Steps Taken to Mitigate Future Risks

In response to the outage, Microsoft and CrowdStrike have implemented several measures to prevent similar incidents in the future:

  1. Enhanced Testing Protocols: More rigorous testing of updates before deployment to catch potential issues earlier.
  2. Improved Monitoring Systems: Advanced monitoring tools to quickly detect and respond to anomalies.
  3. Collaboration on Security: Closer collaboration between cybersecurity firms and service providers to ensure seamless integration of updates and security measures.

These steps aim to strengthen the reliability and resilience of IT systems, ensuring better preparedness for potential future disruptions.

Official Statements by Stakeholders

Microsoft’s Response

In the wake of the outage, Microsoft swiftly acknowledged the disruption, issuing a public apology and providing continuous updates on the recovery process. They collaborated closely with CrowdStrike to identify and resolve the issue, deploying a patch to fix the problem by the end of the day. Microsoft emphasized their commitment to enhancing its testing and monitoring processes to prevent future incidents​. Microsoft has mentioned in their official statement that virtual machines running Windows clients and Windows servers, running the CrowdStrike Falcon agent were affected. As per the 10:30 UTC update by Microsoft on 19 July 2024, the successful recovery is reported by some customers by attempting multiple Virtual Machine restarts. In addition to that, Microsoft has suggested restoring the system from the backup taken before 04:09 UTC on 18 July 2024. Alternatively, customers can attempt repairs on the OS disk by following the instructions given on this link.

Once the disk is attached, customers can attempt to delete the following file:

Windows/System32/Drivers/CrowdStrike/C00000291*.sys

The disk can then be attached and re-attached to the original VM.

Awareness – Virtual Machines

We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state. 

Updated: We approximate impact started as early as 04:09 UTC on the 18th of July, when this update started rolling out.

Additional details from CrowdStrike are available here: Statement on Windows Sensor Update – crowdstrike.com

Update as of 10:30 UTC on 19 July 2024:

We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines. Customers can attempt to do so as follows:

  • Using the Azure Portal – attempting ‘Restart’ on affected VMs
  • Using the Azure CLI or Azure Shell (https://shell.azure.com)

https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az-vm-restart

We have received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.

Additional options for recovery:

We recommend customers that are able to, to restore from a backup, preferably from before 04:09 UTC on the 18th of July, when this faulty update started rolling out.

  • Customers leveraging Azure Backup can follow the following instructions:

How to restore Azure VM data in Azure portal

  • Alternatively, customers can attempt repairs on the OS disk by following these instructions: 

Troubleshoot a Windows VM by attaching the OS disk to a repair VM through the Azure portal

Once the disk is attached, customers can attempt to delete the following file:

Windows/System32/Drivers/CrowdStrike/C00000291*.sys

The disk can then be attached and re-attached to the original VM.

We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.

Additionally, we’re continuing to investigate additional mitigation options for customers and will share more information as it becomes known.

This message was last updated at 14:20 UTC on 19 July 2024

Source: https://azure.status.microsoft/en-us/status

CrowdStrike’s Explanation and Actions

CrowdStrike issued a detailed statement taking responsibility for the inadvertent bug in their update that caused the widespread failures. They outlined the corrective steps taken, including rolling back the faulty update and working with Microsoft to deploy a fix. CrowdStrike announced an internal review of their update deployment process and committed to more rigorous testing protocols to ensure the integrity of future updates​

CrowdStrike has also assured its customers that CrowdStrike is operating normally and this issue has not affected the Falcon platform system in any way. Just a few minutes ago while I was writing this report, George Kurtz, CEO of CrowdStrike assured his customers, “Today was not a security or cyber incident. Our customers remain fully protected…”

Falcon is a cybersecurity tool used by many businesses for the protection of their digital assets from online threats. In the official statement issued by CrowdStrike on their website, they have given detailed actions to fix this issue or to work around this issue.

Summary

  • CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details

  • Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
  • Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted.
  • Windows hosts which are brought online after 0527 UTC will also not be impacted
  • Hosts running Windows 7/2008 R2 are not impacted
  • This issue is not impacting Mac- or Linux-based hosts
  • Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
  • Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version.

Current Action

  • CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
  • If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

  • Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
    • Boot Windows into Safe Mode or the Windows Recovery Environment
      • NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
    • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
    • Locate the file matching “C-00000291*.sys”, and delete it.
    • Boot the host normally.

Workaround Steps for public cloud or similar environment including virtual:

Option 1:

  • ​​​​​​​Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it.
  • Detach the volume from the new virtual server
  • Reattach the fixed volume to the impacted virtual server

Option 2:

  • ​​​​​​​Roll back to a snapshot before 0409 UTC.

Source: https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/

Reactions from Affected Companies and Industries

Responses from affected companies and industries highlighted significant operational and financial impacts. Many expressed frustration over the disruption but also acknowledged the swift resolution efforts by Microsoft and CrowdStrike. Industry leaders called for greater transparency and collaboration within the tech ecosystem to improve resilience against such outages. Some businesses began reevaluating their IT infrastructure and contingency plans to better prepare for future incidents

Initially, CrowdStrike used the X platform to inform users about this issue, but the post was perceived as insensitive and got backlash from many X users. Earlier CrowdStrike failed to apologised for this incident while posting on X

This was the first post of Kurtz which was not received positively by many:

Here are some of the responses to his post:







Individual Concerns

Do You Need to Be Worried About It?

While the outage was disruptive, individuals should not panic. However, it serves as a reminder of the vulnerabilities in our digital infrastructure.

Potential Impacts on Personal Life and Daily Activities

The outage affected access to emails, documents, and online services, causing inconveniences in communication and productivity. Those heavily reliant on Microsoft services faced significant disruptions in their daily routines.

Advice on Protecting Oneself from Similar Incidents

  • Backup Data Regularly: Ensure your important files are backed up in multiple locations.
  • Use Alternative Services: Diversify the tools and services you rely on to avoid total dependence on one provider.
  • Stay Informed: Follow updates from service providers and cybersecurity experts to stay aware of potential issues.

Solutions and Future Prevention

Steps Taken by Microsoft and CrowdStrike

Both companies have enhanced their testing protocols, improved monitoring systems, and increased collaboration to prevent similar incidents. Microsoft also rolled out a patch to fix the immediate issue.

Recommendations for Businesses and Individuals

  • Regular Updates: Keep your systems and software up to date with the latest security patches.
  • Diversified Systems: Use a mix of different service providers to reduce the risk of widespread impact from a single failure.
  • Cybersecurity Training: Educate employees and individuals on best practices for cybersecurity. Especially technical teams i.e. Development and operations teams. This meme by one of the X users rightly highlighted this issue with sarcasm.

Best Practices for Cybersecurity

  • Regular Backups: Maintain frequent and multiple backups of critical data.
  • Security Audits: Conduct regular security audits to identify and mitigate vulnerabilities.
  • Incident Response Plan: Develop and test a comprehensive incident response plan.

Historical Context

Similar Incidents in the Past

Past incidents, such as the AWS outage in 2017 and the Google Cloud outage in 2019, have shown the significant impact of cloud service disruptions. Each incident provided valuable lessons on the importance of redundancy and quick response strategies.

Lessons Learned from Previous Outages

Key lessons include the need for robust backup systems, the importance of regular updates and testing, and the benefits of having a well-prepared incident response plan.

Recent Updates

Latest Developments and Updates on the Incident

Microsoft and CrowdStrike continue to monitor the situation and provide updates on their progress. Both companies have committed to implementing stronger preventive measures and improving their communication with users during incidents.

Progress on Recovery and Future Preventive Measures

Recovery efforts have been successful, with most systems restored to normal operations. Future measures include enhanced collaboration between service providers and ongoing improvements to testing and monitoring processes.

At the time of writing this report the issue is not fully resolved, both Microsoft and CrowdStrike are working to resolve this. As and when further updates come into light I will keep udating is section. Revisit to stay updated.

Conclusion

Summary of Key Points

The July 19, 2024, Microsoft outage highlighted the vulnerabilities in our digital infrastructure. The incident underscored the importance of robust cybersecurity measures and the need for businesses and individuals to be prepared for potential disruptions.

Final Thoughts on the Importance of Robust Cybersecurity Measures

Ensuring the reliability and security of IT systems is crucial in today’s interconnected world. Regular updates, diversified services, and comprehensive incident response plans are essential components of a strong cybersecurity strategy.

Wisdom nuggets for Readers

Stay informed, keep your systems updated, back up your data regularly, and follow best practices for cybersecurity to protect yourself from similar incidents in the future.

Never forget to Learn. Secure. Succeed. with BasicWeb

Last updated on:

Don’t forget to sign up for the newsletter

Tags:

Leave a Reply

Your email address will not be published. Required fields are marked *