When they do happen, it\u2019s your best practices that will see you through the other side. Because in the heat of the moment, things can quickly get out of hand.<\/p>\n
This guide is a simple step-by-step primer on what to do for both planned and unplanned outages, and how to get your business back on its feet.<\/p>\n
Making an Emergency Plan<\/h3>\n
![\"\"](\"https:\/\/www.sandbox.backupassist.com\/app\/uploads\/sites\/3\/2019\/10\/icon-emergency-plan-279x300.png\"){kind=icon}
<\/center><\/p>\nYou\u2019re probably not surprised to hear this, but your best strategy is to prepare well ahead of an outage. An emergency plan brings order to the chaos, clearly identifies who should do what, and gives you all the details you need to deal with a crisis.<\/p>\n
If you\u2019re reading this guide and you\u2019re not in the middle of an outage right now, this should be your first step.<\/strong> Doing the work now will save you time during an Outage.<\/p>\n If you are in the middle of an outage, skip this section and go right to \u2018Dealing with an Unplanned Outage\u2019. <\/strong>As a side note, add \u2018Making an Emergency Plan\u2019 to your post-mortem for when the outage is resolved.<\/p>\n Contact details:<\/strong> Your emergency plan should have contact details for management, IT, and other involved parties. This should also include private contact details.<\/p>\n Even though some people may hate being on call, it\u2019s essential to have a strategy to deal with incidents no matter what time they occur. It\u2019s highly recommended to let people go off-call after dealing with an incident (Usually 24 hours).<\/p>\n This will lessen resentment for being on call, act as a positive reward for having successfully dealt with the outage, and let them deal with their fatigue (Minimizing social and\/or health issues).<\/p>\n Types of Outage:<\/strong> You want to write a plan for every sort of unplanned or planned outage that can occur, because they will require different steps to resolve them.<\/p>\n Spend a LOT of time thinking about the sort of outages that can occur with any ITbased service. E.g. Are you hosted on Google Cloud? What happens if the Gmail goes out during a Google Cloud Outage? What happens if there\u2019s a weather event and your phone service is knocked out? What happens if Slack is unavailable?<\/p>\n This is a plan for the worst, so think of the worst!<\/p>\n Steps to Resolution:<\/strong> The steps or checklist for how to resolve the outage. You can use this guide as a framework for writing this.<\/p>\n You want to also include any special procedures your IT-based service or infrastructure requires to work with. E.g. Server \/ program shutdown order, how to bring a single one offline, who and where to obtain passwords from, etc.)<\/p>\n Assign Clear Responsibility:<\/strong> Detail exactly who is responsible for carrying out the steps above, so there\u2019s no confusion or doubt. All anyone needs to do is check the plan to see who needs to be contacted, or who should be doing what.<\/p>\n You can use some simple monitoring platforms to inform the people responsible when an issue occurs.<\/p>\n Make sure there is one clear person in charge called the \u2018Incident Commander\u2019. This means you won\u2019t end up with a scenario in which there\u2019s \u2018too many cooks\u2019 or people in charge.<\/p>\n Also, make sure the people involved have the authorization to fix the outage quickly. E.g. If power goes down in the building, it\u2019s not an IT problem\u2014who has the authority to get it fixed quickly?<\/p>\n SLAs:<\/strong> Make sure to include which outages are covered by a Service Level Agreement (SLA)! This way, you\u2019re not rushing around during an emergency demanding your Service Provider fix the problem, only to find out your SLA doesn\u2019t cover it and you wasted valuable time.<\/p>\n Remember to keep your SLAs in a known, safe spot for when this time comes.<\/p>\n Escalations:<\/strong> Detail in your plan the additional steps you need to take if things are getting held up. E.g. If the problem cannot be easily fixed, if the SLA provider exceeds the time limit, if a large data loss has occurred or will occur, etc.<\/p>\n This way, your plan will cover both minor and major incidents, especially if they become progressively worse. You can also use a monitoring program to provide automatic escalation if the issue is not resolved, sending a message to the next person in the chain.<\/p>\n Internal Communications:<\/strong> Before an outage even occurs, there should be a means of conference set up. This should ideally be something that leaves a record, like a Slack room named #outage. Think of it as a war room for when things go wrong. You also might want to use a issue and project tracking software like JIRA.<\/p>\n Phone calls and verbal conversations have no accountability, so these should be your last resort method. If you must go with this, have a situation room set aside and take notes during the whole process.<\/p>\n External Communications:<\/strong> Identify if your customers are going to be impacted by an outage, and how you\u2019re going to keep them in the loop. You\u2019ll want more than one medium for this. E.g. E-mail, social media, website, etc.<\/p>\n Hardware and Network Description:<\/strong> A list of the hardware and network information that is relevant to a particular outage. All this information will be relevant to the person(s) trying to resolve the situation.<\/p>\n Emergency Laptop:<\/strong> Make sure IT has at least one laptop available for an outage that is kept updated.<\/p>\n Automate What You Can:<\/strong> Human error is the top cause of most disasters, so you want to minimize the human factor as much as possible (E.g. Automated backup schedules). This should also reduce the workload during an outage. Identify anything that would chew up time or effort, and see if it can be automated.<\/p>\n Multiple Plan Copies:<\/strong> Once you\u2019re done with your plan, make sure you have hard copies in certain, predefined places around your company, and a digital one on your company\u2019s intranet.<\/p>\n Confirm that it is, in fact, an outage <\/strong><\/p>\n Before you break the emergency glass and assemble the A-team, make sure you\u2019re not dealing with something else (Lag, internet being down, etc.). You don\u2019t want to tell your manager there\u2019s an outage when it\u2019s something mundane.<\/p>\n Determine the Cause <\/strong><\/p>\n In a time-effective manner, determine the cause. Is it a programming error? DNS problem? Expired domain? Hardware failure?<\/p>\n Just find out generally what you\u2019re dealing with. You don\u2019t want to exhaustively investigate the issue at the expense of not resolving it.<\/p>\n Stay clear of blame through this entire process. Save it for the postmortem after the outage is fixed!<\/p>\n Get Out Your Emergency Plan (If Applicable) <\/strong><\/p>\n If you followed our guide on page 3, \u2018Making an Emergency Plan\u2019, you should already have a plan in place that you can pull out for this scenario.<\/p>\n If not, skip this step, and you\u2019re going to have to figure out everything as you go along.<\/p>\n Designate an Incident Commander <\/strong><\/p>\n There should be one person in charge of dealing with the outage. If you\u2019ve written up an Emergency Plan, this person should already be designated. If not, you\u2019re going to have to choose someone now.<\/p>\n The Incident Commander is the person who has the final decision-making authority, and is usually the first person to respond to an incident. They\u2019re running the show, and any task that isn\u2019t already assigned to someone falls to them.<\/p>\n They will need to bring people in the loop that need to be involved, such as subject experts. Your Incident Commander is vital to resolving the outage.<\/p>\n The Incident Commander will either need the authority to fix the outage quickly or be in contact with people who can.<\/p>\n Find out if the Incident is covered by an SLA <\/strong><\/p>\n There\u2019s a chance that your particular outage is covered by a Service Level Agreement (SLA), and you may be able to outsource part or all of dealing with it.<\/p>\n Find this out immediately and if you can get them to fix it. Escalate the problem if they don\u2019t complete it within an acceptable time limit, the problem can\u2019t be fixed by them, or you\u2019re at risk of large data loss.<\/p>\n Even if you can\u2019t outsource fixing your outage, you may also have access to technical support relevant to this sort of outage, which could help you resolve it sooner.<\/p>\n Get the Right People Involved <\/strong><\/p>\n Assuming you can\u2019t push dealing with the outage off to a Service Provider, your Incident Commander is going to need to get to work.<\/p>\n The first thing they need to do is make sure the right people to be part of the process, particularly the Support Team. They\u2019re going to be fielding questions from your customers, so they need to know what\u2019s going on.<\/p>\n You\u2019ll also need a point person or persons to field incoming calls dealing with the outage. Make sure to enable the support team to do their job.<\/p>\n Communication is key. Ideally, you want it to be recorded, such as a Slack room. As mentioned in an earlier chapter, phone or verbal communication have no accountability, and you want a record for two reasons.<\/p>\n 1. So people can go back over what the Incident Commander told them to reaffirm what they\u2019re meant to be doing.<\/p>\n 2. For post-mortem analysis after the incident is over.<\/p>\n The Checklist <\/strong><\/p>\n The Incident Commander will need to make a checklist of all the things that will need to be done to resolve the outage. This is best done with the input of the support team and any relevant parties.<\/p>\n Include details on how to escalate the situation if one of these tasks cannot be completed or isn\u2019t being completed in the right amount of time.<\/p>\n In Case of Hacking <\/strong><\/p>\n If your outage is the result of hacking, there are a number of other time-sensitive tasks that will need to be done.<\/p>\n Regular Sitreps for Internal Stakeholders <\/strong><\/p>\n Internal Stakeholders (E.g. Employees) probably don\u2019t want to know the nitty-gritty about the outage or what you\u2019re doing. Usually, they want to know some very specific things:<\/p>\n Knowing this information isn\u2019t just a good way to keep them quiet, it also helps them do and plan their own jobs.<\/p>\n The Incident Commander should be doing hourly situation reports (SitReps) to the whole company. It should answer the above questions in a concise manner and avoid unnecessary detail.<\/p>\n You might even want to point them to a place where they can view these SitReps, such as an intranet page, or the discussions going on.<\/p>\n Automate Anything You Can <\/strong><\/p>\n You don\u2019t have tons of time to spare, so automate anything you can. Not only does this remove the human element (one of the biggest reasons something can go wrong), it frees you to deal with other parts of the outage.<\/p>\n It should be easy for your internal stakeholders to get info without contacting the Incident Commander, who has enough on their plate.<\/p>\n Don\u2019t Go Radio Silent <\/strong><\/p>\n When dealing with internal stakeholders, the immediate instinct will be to shut them out to deal with the outage.<\/p>\n Don\u2019t do this! While you should have avenues for people to find out about the incident, you want to be accessible, and nothing is more demoralizing than going radio silent.<\/p>\n People should be able to contact the Incident Commander to find out more about the outage. It\u2019s a balancing act between keeping yourself focused on solving the outage and cutting people out completely, but it\u2019s a tightrope that needs to be walked.<\/p>\n Keep Customers Informed <\/strong><\/p>\n Decide in advance how much you want customers to know. You don\u2019t need to tell them everything, but it\u2019s wise to tell them something and keep them informed. Craft your message carefully.<\/p>\n An unplanned outage is a common way to lose customers, so you want to make sure you\u2019re as transparent as possible so they know you\u2019re dealing with the issue.<\/p>\n Make sure you\u2019re using the right tone. You\u2019ll need to be concise, authoritative, and serious. While you may be tempted, don\u2019t try to joke or be cute\u2014remember, their business is being affected! Instead, you\u2019ll want to show humility, consideration, and don\u2019t be afraid to accept fault<\/p>\n Ideally, you\u2019ll want to keep them informed every 30 minutes, even if there\u2019s nothing to report. Don\u2019t expect users to go to the status page. If they go to e-mail user support, send them all a reply when the incident is solved.<\/p>\n The Post Mortem <\/strong><\/p>\n After the outage is finished, it\u2019s time to win back trust. If you followed the steps above, you\u2019ll have a detailed record and information on what happened and what was done to resolve it.<\/p>\n Be accurate about what happened, what went wrong, and how you\u2019re going to prevent it from ever happening again. You can use this information to follow up with customers to regain their support. They want to know that there\u2019s no flaws anymore in your service, or you\u2019re in the process of fixing it.<\/p>\n In the future, make preventing outages a priority by having your teams spend a portion of their time making proactive steps to prevent it from ever happening. It\u2019s less glamorous than rushing in to save the day, but your shareholders and customers will love you more for a service that never fails as opposed to one that breaks often and is quickly fixed.<\/p>\n Talk to Management <\/strong><\/p>\n Planned outages usually happen as part of a project, one that you discuss with management beforehand. So first, you need to make sure they\u2019re in the loop.<\/p>\n Inform Your Users <\/strong><\/p>\n If the outage is going to affect a number of users, you\u2019ll need to send out an e-mail ahead of time containing the following information:<\/p>\n A step-by-step on dealing with planned and unplanned outages.<\/p>\n","protected":false},"author":2,"featured_media":12850,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[757],"tags":[],"class_list":["post-12737","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cyber-resilience"],"yoast_head":"\nDealing with an Unplanned Outage<\/h3>\n
![\"\"](\"https:\/\/www.sandbox.backupassist.com\/app\/uploads\/sites\/3\/2019\/10\/icon-warning-300x281.png\"){kind=icon}
<\/center>\n
\n
Dealing with a Planned Outage<\/h3>\n
![\"\"](\"https:\/\/www.sandbox.backupassist.com\/app\/uploads\/sites\/3\/2019\/10\/icon-planned-278x300.png\"){kind=icon}
<\/center>\n