It’s no secret that we believe in the importance of disaster preparedness and business continuity at every organization. But what does that planning actually look like when it’s put to the test in a real-world scenario?
Today, we look at 7 business continuity examples to show how organizations have worked to minimize downtime (or not) after critical events.
For years, healthcare organizations have been a top target for ransomware attacks. The critical nature of their operations, combined with notoriously lax IT security throughout the industry, are a magnet for ransomware groups looking for big payouts.
But despite the warnings, healthcare orgs still remain vulnerable. A prime example was the 2021 ransomware attack on Ireland’s healthcare system (HSE) – the fallout from which was still being felt nearly a year later.
According to reports, the attack had a widespread impact on operations:
All told, the attack was projected to cost more than $100 million in recovery efforts alone. That figure does not include the projected costs to implement a wide range of new security protocols that were recommended in the wake of the attack.
Like several of the business continuity examples highlighted below, the Ireland attack did have some good disaster recovery methods in place. Despite the impact of the event, there were several mitigating factors that prevented the attack from being even worse, such as:
However, there was some luck involved.
As HSE raced to contain the damage from the attack and secured a High Court Injunction to restrain the sharing of its hacked data, the attackers suddenly released the decryption key online. Without that decryption, HSE would not have had adequate data backup systems to recover from the attack. As the group concluded in its post-incident review:
“It is unclear how much data would have been unrecoverable if a decryption key had not become available as the HSE’s backup infrastructure was only periodically backed up to offline tape. Therefore it is highly likely that segments of data for backup would have remained encrypted, resulting in significant data loss. It is also likely to have taken considerably longer to recover systems without the decryption key.”
There has been no shortage of other headline-making ransomware attacks over the last few years. But one that stands out (and whose impact reverberated for at least a year after the incident) was the March 2018 SamSam ransomware attack on the City of Atlanta.
The attack devastated the city government’s computer systems:
Attackers demanded a $52,000 ransom payment. But when all was said and done, the full impact of the attack was projected to cost more than $17 million. Nearly $3 million alone was spent on contracts for emergency IT consultants and crisis management firms.
In many ways, the Atlanta ransomware attack is a lesson in inadequate business continuity planning. The event revealed that the city’s IT was woefully unprepared for the attack. Just two months prior, an audit found 1,500 to 2,000 vulnerabilities in the city’s IT systems, which were compounded by “obsolete software and an IT culture driven by ‘ad hoc or undocumented’ processes,” according to StateScoop.
Which vulnerabilities allowed the attack to happen? Weak passwords, most likely. That is a common entry point for SamSam attackers, who use brute-force software to guess thousands of password combinations in a matter of seconds. Frankly, it’s an unsophisticated method that could have been prevented with stronger password management protocols.
Despite the business continuity missteps, credit should still be given to the many IT professionals (internal and external) who worked to restore critical city services as quickly as possible. What’s clear is that the city did have some disaster recovery procedures in place that allowed it to restore critical services. If it hadn’t, the event likely would have been much worse.
Here’s an example of business continuity planning done right:
In 2013, lightning struck an office building in Mount Pleasant, South Carolina, causing a fire to break out. The offices were home to Cantey Technology, an IT company that hosts servers for more than 200 clients.
The fire torched Cantey’s network infrastructure, melting cables and burning its computer hardware. The equipment was destroyed beyond repair and the office was unusable. For a company whose core service is hosting servers for other companies, the situation looked bleak. Cantey’s entire infrastructure was destroyed.
But ultimately, Cantey’s clients never knew the difference:
It was an outcome that could have turned out very differently. Only five years prior, the company had kept all of its client servers on site. But founder Willis Cantey made the right determination that this setup created too many risks. All it would take is one major on-site disruption to wipe out his entire business, as well as his clients’ businesses, potentially leaving him exposed to legal liabilities as well.
Cantey thus implemented a more comprehensive business continuity plan and moved his clients’ servers off-site. And in doing so, he averted disaster. This makes for an excellent business continuity plan case study that demonstrates how proper planning can significantly reduce the risk of a major operational disruption.
In another post, we highlighted one of the worst business continuity examples we saw in 2016 – before ransomware had become a well-known threat in the business community.
On October 30, 2016, a nasty “computer virus” infected a network of hospitals in the UK, known as the Northern Lincolnshire and Goole NHS Foundation Trust. At the time, little was known about the virus, but its impact on operations was devastating:
Remarkably, a report in Computing.co.uk speculated that there had been no business continuity plan in place. Even if there had been, clearly there were failings. Disaster scenarios can be truly life-or-death at healthcare facilities. Every healthcare organization must have a clear business continuity plan outlined with comprehensive measures for responding to a critical IT systems failure. If there had been in this case, the hospitals likely could have remained open with little to no disruption.
The hospital system was initially tight-lipped about the attack. But in the year following the incident, it became clear that ransomware was to blame – specifically, the Globe2 variant.
Interestingly, however, hospital officials did not say the ransomware infection was due to an infected email being opened (which is what allows most infections to occur). Instead, they said a misconfigured firewall was to blame. (It’s unclear then exactly how the ransomware passed through the firewall—it may have come through inboxes after all.) Unfortunately, officials knew about the firewall misconfiguration before the attack occurred, which is what makes this incident a prime example of a business continuity failure. The organization had plans to fix the problem, but they were too late. The attack occurred “before the necessary work on weakest parts of the system had been completed.”
Here is another example of well-executed business continuity.
After a major electric company in Georgia experienced failure with one of its data lines, it took several proactive steps to ensure its critical systems would not experience interruption in the future. The company implemented a FatPipe WARP at its main site, bonding two connections to achieve redundancy, and it also readied plans for a third data line. Additionally, the company replicated its mission-critical servers off-site, incorporating its own site-failover WARP.
According to Disasterrecovery.org:
“Each office has a WARP, which bonds lines from separate ISPs connected by a fiber loop. They effectively established data-line failover at both offices by setting up a single WARP at each location. They also accomplished a total site failover solution by implementing the site failover between the disaster recovery and main office locations.”
While the initial WAN problem was minimal, this is a good example of a company that is planning ahead to prevent a worst-case scenario. Given the critical nature of the utility company’s services (which deliver energy to 170,000 homes across five counties surrounding Atlanta), it’s imperative that there are numerous failsafes in place.
Among the better business continuity examples we’ve seen, incident management solutions are increasingly playing an important role.
Take the case of a German telecom company that discovered a dangerous fire was encroaching on one of its crucial facilities. The building was a central switching center, which housed important telecom wiring and equipment that were vital to providing service to millions of customers.
The company uses an incident management system from Simba, which alerted staff to the fire, evaluated the impact of the incident, automatically activated incident management response teams and sent emergency alerts to Simba’s 1,600 Germany-based employees. The fire did indeed reach the building, ultimately knocking out the entire switching center. But with an effective incident management system in place, combined with a redundant network design, the company was able to fully restore service within six hours.
Research shows that 40-60% of small businesses never reopen their doors after a major disaster. Here’s an example of one small firm that didn’t want to become another statistic.
In August 2017, Hurricane Harvey slammed into Southeast Texas, ravaging homes and businesses across the region. Over 4 days, some areas received more than 40 inches of rain. And by the time the storm cleared, it had caused more than $125 billion in damage.
Countless small businesses were devastated by the hurricane. Gaille Media, a small Internet marketing agency, was almost one of them. Despite being located on the second floor of an office building, Gaille’s offices were flooded when Lake Houston overflowed. The flooding was so severe, nobody could enter the building for three months. And when Gaille’s staff were finally able to enter the space after water levels receded, any hopes for recovering the space were quickly crushed. The office was destroyed, and mold was rampant.
The company never returned to the building. However, its operations were hardly affected.
That’s because Gaille kept most of its data stored in the cloud, allowing staff to work remotely through the storm and after. Even with the office shuttered, they never lost access to their critical documents and records. In fact, when it came time to decide where to relocate, the owner ultimately decided to keep the company decentralized, allowing workers to continue working remotely (and providing a glimpse of how other businesses around the world would similarly adapt to disaster during the Covid-19 pandemic three years later).
Had the company kept all its data stored at the office, the business may never have recovered.
Some of the real-life business continuity examples above paint a picture of what can go wrong when there are lapses in continuity planning. But what exactly do those lapses look like? What are the specific mistakes that can increase a company’s risk of disaster?
Here are some examples of business continuity failures due to poor planning:
It’s important to remember that business-threatening disasters can take many forms. It’s not always a destructive natural disaster. In fact, it’s far more common to experience disaster from “the inside” – events that hurt your productivity or affect your IT infrastructure and are just as disruptive to your operations.
Example threats include:
The list goes on and on. Any single one of these threats can disrupt your business, which is why it’s so important to take continuity planning seriously.
Within IT, data loss is often the primary focus of business continuity and disaster recovery (BC/DR). And for good reason …
Data is the lifeblood of most business operations today, encompassing all the emails, files, software and operating systems that companies depend on every day. A major loss of data, whether caused by ransomware, human error or some other event, can be disastrous for businesses of any size.
Backing up that data is thus a vital component of business continuity planning.
Today’s best data backup systems are smarter and more resilient than they were even just a decade ago. Solutions from Datto, for example, are built with numerous features to ensure continuity, including hybrid cloud technology (backups stored both on-site and in the cloud), instant virtualization, ransomware detection and automatic backup verification, just to name a few.
Like other BC initiatives, a data backup solution itself won’t prevent data-loss events from occurring. But it does ensure that businesses can rapidly recover data if/when disaster strikes, so that operations are minimally impacted – and that’s the whole point of business continuity.
By now, you’re starting to get the picture: business continuity planning is crucial. But how do you actually create the plan? What does the document look like?
While each business’s BCP is unique to its needs, the foundation of the plan is generally the same for most organizations. The core goal is to document a company’s risks and outline what is needed to avoid an operational disruption.
Here are some examples of business continuity plan components to include in your documentation:
Examples of business continuity plans can differ by industry, but most companies will want to incorporate all of the components above, regardless of business size or sector.
In February 2023, a ransomware attack struck Karmak – a prominent technology solutions provider for the trucking industry. However, the company acted quickly to contain the attack before it disrupted its operations or customers, providing a solid case study for how to maintain continuity during a cyberattack.
Karmak’s business continuity planning played a key role in averting disaster. According to an industry trade publication, Karmak had a “detailed cyberattack response plan, which went into effect immediately after the attack.” The company used security monitoring solutions to detect and thwart the attack. Plus, employees had been rigorously trained on cybersecurity and knew how to respond.
End result: Karmak contained the attack within hours, preventing customer data from breached and minimizing the impact on internal systems.
Any scenario in which a business can continue to operate through a disruptive event is an example of business continuity. For example, a company facing a ransomware attack might maintain business continuity by restoring infected files from a data backup.
An example of a business continuity plan is a comprehensive document that assesses a business’s risk for operational disruptions and outlines the steps for avoiding such disruptions. Example components of the plan include a risk assessment, business impact analysis, communications plan and disaster recovery plan.
The Covid-19 pandemic illustrated many real-life examples of business continuity. Companies took several measures to continue operating during the health crisis, such as allowing employees to work from home, instituting physical distancing and providing protective equipment to critical workers.
Avoid a major operational disruption with today’s best technology for business continuity, disaster recovery and cybersecurity. Schedule a meeting with one of our data-protection specialists at Invenio IT or contact us by calling (646) 395-1170 or by emailing success@invenioIT.com.
