Network Downtime – Costs and Mitigation

Network downtime costs companies, big and small, many millions of dollars each year in lost revenue, and lost revenue is only the beginning. Network downtime, including full blackouts as well as brownouts, can seriously undermine customer’s confidence in a company, can negatively impact stock prices, and can result in many other negative impacts to a company’s bottom line. Understanding the costs of network downtime, and what you can do to prevent or mitigate problems, is fundamental in today’s highly connected society. As a minimum, you should have a solid disaster recovery plan, and you should be using SD-WAN with broadband bonding appliances to intelligently aggregate your WAN resources.

Network Outages Continue to Wreak Havoc

According to Gartner, the average cost of network downtime is $5,600 per minute, or over $300K per hour. Obviously, this figure is very different for every company and every incident. If you’re a major online retailer, an outage could be devastating if it occurs during a new product launch, while many small and medium-sized businesses suffer from little more than minor inconveniences.

The typical cause of a mass outage varies. According to the Ponemon Institute, the three most common causes are power supply failure (25%), cybercrime (22%) and accidental human error (22%).

One of the most comprehensive summaries of global IT downtime and its impact on business is the 2019 IT Outage Impact Study done by LogicMonitor. They surveyed 300 IT decision makers, of companies with at least 2500 employees, from the US and Canada, the UK, and Australia and New Zealand. The complete survey is interesting and definitely worth downloading. Here are some of the key findings, as reported by global IT decision makers:

  • 96% had experienced at least one outage in the past 3 years
  • Approximately 50% had experienced at least 5 outages in the past 3 years
  • Approximately 50% felt that blackouts and/or brownouts were avoidable

Here are five of the most significant network outages in the past decade, as detailed by TechRadar:

  • 2016 Dyn Cyberattack – Multiple distributed denial-of-service attacks (DDoS attacks) targeted systems operated by Managed Domain Name Systems (DNS) provider Dyn. Millions of users of popular sites including Airbnb, Amazon, BBC, CNN, eBay, Netflix, and Twitter were affected as these and hundreds of other sites were offline for hours. Exact costs of this attack are not known, but certainly resulted in millions of lost revenue.
  • British Airways IT Failure (2017) – A global IT crash resulted in the airline’s entire fleet being grounded, leaving thousands of passengers stranded over one of the busiest weekends for travel in the UK.
  • Amazon Web Services (2017) – Amazon’s S3 web-based storage service experienced high error rates, leading to either partial or full outages on a number of popular websites, apps, and devices.
  • Vodafone Data Center (2011) – Mobile phone operator Vodafone suffered major service interruptions after thieves broke into its Hampshire-based data center using sledgehammers.
  • Armenia Country-Wide Blackout (2011) – Armenia lost most of its connectivity when a 75-year-old woman digging for scrap metal damaged land cables and thereby severed most connectivity across the nation.

While you may think that network outages are declining in terms of severity and costs, in fact just the opposite is true. And this makes sense if you think about it. A nice discussion by Andy Lawrence of Uptime Institute states that “Complexity and interdependency of IT systems and greater dependency on software and data are very likely to be big reasons. For example, Uptime’s Institute’s research shows that fewer big outages are caused by power failure in the data center and more by IT systems configuration now than in the past.”

Another interesting (and surprising) culprit in the increases and ramifications of network outages is a fundamental internet routing protocol, BGP (Border Gateway Protocol), which is responsible for routing packets on the internet from one router to the next. It appears that BGP itself is vulnerable, according to this Wired article:

Originally conceived in 1989 (on two napkins), the version of BGP used today remains largely unchanged since 1994. And though BGP has scaled surprisingly well, there’s no denying that the internet is very different than it was 25 years ago. In fact, the way BGP was designed introduces risk of outages, manipulations, and data interception—all of which have come to pass.

When adding reliability and inter office connectivity, one needs to reconsider if BGP is the appropriate approach when compared to more modern, simpler and more effective SD-WAN solutions.

Calculating Network Downtime Costs

Calculating the cost of network outages is simple in theory, and much more difficult in practice – as you well know if you’re involved in this kind of exercise. A clear explanation of the basics are nicely provided by ITonDemand and are summarized below:

Cost of Downtime (per hour) = Lost Revenue + Lost Productivity + Recovery Costs + Intangible Costs

Lost Revenue – How much direct revenue you lost as result of the outage

Lost Revenue = (Revenue/hour) x (Downtime hours) x (Uptime Dependence %)

If you are exclusively an online retailer, your dependence on uptime for your revenue is 100%. If you are more diversified and also sell through brick-and-mortar locations, your uptime dependence would drop depending on how much of your revenue was directly realized through online sales.

Lost Productivity – How much is lost due to salaried employees being offline

Lost Productivity (per employee) = (Employee Salary/hour) x (Utilization %), where utilization percentage is the percentage of the employee’s work that can only be done online.

Recovery Costs – How much the outage costs to “repair”

Typically includes repair/3rd party services, replacement/upgraded hardware, lost data recovery, and other similar factors.

Intangible Costs –

These may include damage to the company brand or reputation, slide in stock price, compliance failures, and others.

The Two Most Important Thing You Should Do, Right Now

We know that there will continue to be network outages for the foreseeable future. So what are some current best practices that can reduce the “avoidable” outages and mitigate damage from the “unavoidable” outages, such as natural disasters or 75-year-old women digging up cables?

Disaster Recovery Plan

Disaster recovery plans entail much more than simply backing up data and network configurations. It involves detailed planning and execution before, during, and after any “disaster” – the definition of which is company dependent and should also be included in the overall plan.

Disaster recovery plans have been around since the first dial-up modem failed. There are now many companies that specialize in either helping companies develop their own plans, or take control and responsibility on themselves.

Below are several recent links to articles that I found to be particularly comprehensive and insightful when it comes to getting your arms around the problem of disaster recovery:

SD-WAN with Broadband Bonding

We know that blackouts and brownouts can be devastating for many businesses and yet many SMB’s still rely on a single ISP for their WAN connectivity and trust the SLA with their ISP will protect them against downtimes (think again). This jeopardizes the entire business by introducing a single point of failure into the office’s lifeblood – Internet connectivity. So, if you’re still relying on a single ISP to transport all of your office’s WAN traffic, you are asking for trouble. When (not if) that single line has performance problems or fails, you lose money.

Some network blackouts, or complete failures, could be completely averted if your office networks utilize SD-WAN broadband bonding appliances, which bonds multiple broadband transports, both wired (DSL, cable, T1, MPLS) and wireless (4G/LTE/5G), from different ISPs, into a single, super-reliable, super-high-bandwidth Internet pipe. In this case, if one of your ISPs goes down, your business might not even notice, as the traffic auto-failovers to the active lines.

Internet brownouts (significant, temporary drop in performance) don’t get the press coverage that major blackouts do, but perhaps they should. We all know that the performance of any single internet line fluctuates on many timescales (time of day, time of year, second-to-second). We also know that despite SLAs, every ISP throttles their performance when usage dictates, and that variations in bandwidth, latency, and jitter come with the territory. But by using best-in-class SD-WAN broadband bonding appliances these performance dropouts can be sidestepped. The SD-WAN appliance will guarantee that every single Internet packet travels through the optimal path within the bonded internet pipe—no matter what is happening within your network at any given time.

Make sure the SD-WAN solution you use includes features such as broadband bonding, advanced layer-7 routing, monitoring portals, dynamic quality of service (QoS), deep-packet inspection (DPI), and application performance optimization for various flow types including live video, VoIP, cloud apps and others.

Conclusion

It looks like network blackouts and brownouts are going to remain a significant issue for most companies for quite a while. It may be impossible to avoid natural disasters and 75-year-old women digging up Internet trunk lines. But having a solid and sensible disaster recovery plan and utilizing the best technology to intelligently aggregate broadband WAN resources, will reduce the frequency and severity of network outages, and save your company time and money.

Rob Stone, Mushroom Networks, Inc. 

Mushroom Networks is the provider of Broadband Bonding appliances that put your networks on auto-pilot. Application flows are intelligently routed around network problems such as latency, jitter and packet loss. Network problems are solved even before you can notice.

https://www.mushroomnetworks.com

 

Share on facebook
Facebook
Share on twitter
Twitter
Share on linkedin
LinkedIn
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments

© 2004 – 2020 Mushroom Networks Inc. All rights reserved.

Let’s chat. Call us at +1 (858) 452-1031 or fill the form:

Get your WAN Cheat Sheet

Download your copy of rare tips and tricks for a better WAN. Get your free copy today!

We respect your privacy.