• Article

Disaster Recovery

According to Wikipedia, the term disaster recovery refers to:

“. . . a set of policies, tools and procedures to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster.” (Source: Wikipedia) A disaster impacting systems can be any “unexpected problem that results in a slowdown, interruption or failure in a key system or network.” (Source: Fundamentals of Disaster Recovery Planning in IT Management.)

All agencies rely on various systems and equipment to operate and support the delivery of services to supported individuals and clients. Many aspects of agency operations rely on the smooth functioning of these tech tools. The discipline of disaster recovery has become increasingly important as our reliance on technology grows, and cyber events and crime escalate.

For example, did you know that:

  • During 2020, more than two-thirds of IT security professionals believed that a successful cyberattack was imminent
  • Although the volume of ransomware infection rates declined in 2018, almost half of victims resolved attacks by paying the ransom requested by cybercriminals
  • During 2019-2020, the average global cost to remediate a ransomware attack was $761,106 (source: Sophos The State of Ransomware 2020)
  • The average cost of downtime is 24 times higher than the average ransom amount.
  • Phishing attacks reached their highest level in 3 years, and small organizations received malicious emails at a higher rate (source: Symantec’s Internet Security Threat Report 2019)

In addition to attacks, other causes of disruptions and damage to systems include:

  • Natural disasters (such as fires, earthquakes, and hurricanes)
  • Power failures or power surges
  • Human error

Many Benefits

Disaster recovery experts and advisers agree that the benefits of disaster recovery planning are substantial. They include:

  • Saving money and time on the recovery process
  • Ensuring an orderly recovery, repair, and restoration process
  • Avoiding panic when a disaster happens
  • Protecting critical systems and technology assets from damage
  • Increasing awareness about the role of all team members in protecting the safety and security of systems, data, and equipment

Focus Areas

Disaster recovery is fundamentally about considering the various consequences of systems becoming damaged,  temporarily unavailable, or permanently inaccessible. The discipline urges consideration of three areas of focus:

  1. Prevention – Steps and strategies an agency adopts and implements to reduce the likelihood and potential negative consequences of technology ‘events.’
  2. Anticipation – Work teams undertake to identify the sources and circumstances that give rise to potential incidents and the related consequences of those incidents
  3. Mitigation – Steps, strategies, systems, approaches, and tasks undertaken to manage tech interruptions and disasters to minimize negative impacts. Examples of tips and techniques in the mitigation realm include:
    • Conducting regular tests of recovery plans
    • Keeping backup or loaner equipment available for use
    • Ensuring that staff can work from alternative locations, such as home
    • Developing manual procedures that can be used if systems are offline for a prolonged period
    • Coordinating disaster recovery with related organizations, vendors, and across all programs of the agency

Pre-Disaster and Post Disaster

Another way to think about Disaster Recovery (DR) planning is to break it down into two stages: Pre-Disaster and Post-Disaster. Experts at TechSoup, a thought leader organization in the nonprofit sector that offers help desk services, managed IT services, and much more, recommend this bifurcated approach.

Some of the tips in TechSoup’s materials and resources on Pre-Disaster Planning include:

  1. Gather important information
    • Warranties and receipts for computers and peripherals
    • Information about where, how, and how frequently your data is stored and backed up
    • Instructions for how to restore your data
    • Passwords for encrypted data
    • Contact information for any employees, volunteers, or consultants who maintain your agency’s tech infrastructure
    • A phone tree that includes landline and cellphone numbers for all staff members
    • Login information for administrative accounts on all computers
    • Login information for web hosting and backup service providers
    • Contact information for web hosting and backup services
    • Software registration information, including keys
  1. Keep and protect records: store crucial information
    • Hard copies – sheltered from natural disasters and theft, as in a waterproof safe or safe deposit box
    • Encrypted – on personal storage devices
    • Encrypted – off-site and online
    • Use a Master Key – TechSoup recommends using a portable USB flash drive that contains all information needed for IT recovery and ensuring that the key is held by the Executive Director and a backup.
  1. Manage employee onboarding and offboarding with care
    • Include disaster plan information in new staff orientations and onboarding
    • When staff members depart, update the disaster plan, archive email, change any passwords for systems and equipment the employee had access to, back up the former employee’s computer and reformat it
  1. Manage Backups

There are many methods for creating data redundancy. Four common types of backups are full backups, incremental backups, differential backups, and mirror backups. Each method has advantages and disadvantages. The strategy you choose should consider the amount of data you are storing, how long you are willing to wait to restore data, and whether or not you can afford data loss.

Full Backup

A full backup captures every file and folder in the system. This method takes longer and requires the most space. It is best suited for agencies that can acquire the necessary storage space and need expedient data recovery with minimal data loss.

Incremental Backup

Incremental backups execute an initial full system backup, and subsequent backups only capture changes made since the previous backup. This method produces quicker backups after the initial data capture and requires low storage space. Agencies restoring from an incremental backup often experience delays as data is collated, and there is a risk of some data loss.

Differential Backup

Differential backups are similar to incremental backups, except this method captures changes since the last fullbackup, not merely the previous backup. Therefore, this method requires more space than an incremental backup strategy but provides faster recovery times and less potential data loss.

Mirror Backup

Mirror backups create an exact copy of the source data. If a file or folder is deleted from the source data, it is also deleted from the backup. This method is fast in backup execution and recovery but has a high risk for data loss. Many agencies rely on backups to restore accidental deletions. However, this method would not provide sufficient redundancy to safeguard against this type of error.

Aside from the method you use to back up your data, there are also varying approaches to backups. TechSoup offers a helpful chart to compare and contrast backup approaches for Files and Databases:

The experts at TechSoup also offer helpful tips and resources on Post-Disaster Recovery. These tips include:

  1. Develop a technology triage list

  • Identify damaged or lost equipment
  • Focus on restoring systems and equipment that will have the most impact
  • Consider reaching out to other agencies whose systems are intact to explore working with those agencies while your systems are undergoing repair or restoration
  • If your email service has been disrupted, you may need to identify a new provider. To make the switch, you’ll need to update your mail exchange—or MX—record.
  • Consider establishing a ‘help desk’ if you expect to receive a large volume of inquiries from clients or other stakeholders

  1. Recover data

When you think about data recovery, most people only consider the options available for recovering digital files sand records. However, remember that data recovery can also relate to lost information stored on damaged physical records. If your agency relies heavily on paper records and hard-copy storage, consider adding protocols for data recovery relating to this type of information. Consider these tips for mitigating damages to paper records:

  • First and foremost, consider health and safety of staff handling damaged paper records. When necessary, use appropriate personal protective equipment (PPE) which may include gloves, masks, or respirators.
  • If a large volume of documents have sustained water damage, consider freezing them to prevent further damage. Freezing halts the degradation of water damage to paper and allows for documents to be slowly dried in manageable “batches.”
  • Follow these step-by-step tips for drying wet paper on wikiHow: https://www.wikihow.com/Dry-Wet-Paper

The commonplace type of data recovery has become digital data recovery. This process takes place when digital records are corrupted, deleted, or otherwise unusable. Backups are the first line of defense for recovering lost digital data. Keep in mind these tips:

  • The file type (i.e., documents and spreadsheets, accounting databases, emails, website files and databases, program files or operating systems) will likely be the driving factor in the method of data recovery.
  • It’s possible that full recovery can take time. Therefore, you may have to have temporary measures in place as a stopgap while data is being restored. For example, if your website was hacked it may take time to load a copy of your site to the web server so you may consider redirecting your page to a temporary landing page that provides crucial agency information until the full site can be restored.
  • Unfortunately, backups are not infallible. If your back up fails to recover lost data:
    • Look for other places where you might have inadvertently stored your data
    • Try to find the recovery discs for the operating system of the computer
    • Upon finding a copy of the data, back it up and make a copy first

The cloud can be powerful backup and recovery tool. Cloud storage options often have built in redundancies, are cost-effective, and offer added peace of mind. If you are using or considering using the cloud for as your main backup and recovery protocol consider the following:

  • Determine if you have the bandwidth and network capacity to restore from cloud backups. Factors will include the internet speeds available through your ISP contract and whether or not staff will be using the internet during the restore process.
  • Determine how long restoring from the cloud will take. The restoration time must take into account both the backup download time (a simple math equation—file size divided by the bandwidth speed) and the time it takes to upload the backed-up copy of the missing or corrupt files to the original location.
  • Find out if your backup provider can mail physical media if you’re unable to fully restore via the Internet.

  1. Recover and restore equipment

  • Turn the power off before touching any surfaces or equipment that are wet
  • If possible, move equipment to a safe, dry area
  • Use sturdy tables to support any equipment placed on them
  • Ensure that you have reliable electrical power before turning equipment back on. Test electrical outlets by plugging in a light and ensuring that it works properly before plugging in a piece of tech equipment into the same outlet.
  • Make sure that you turn off and unplug computers when a severe storm or power outage is forecast.
  • Ensure that the vents on your computer equipment are unblocked.
  • Review the Hardware Recovery Tips in the TechSoup Guide to Nonprofit Disaster Recovery for additional tips!

  1. Keep stakeholders informed

    • Post updates on your operating status and changes in service availability on your website
    • Also, consider posting status messages on Facebook, Twitter, or other social media sites

  1. Know how to retrieve important information about your website

    • Make sure you know the identity and contact information for your web hosting provider, your domain registrar, and web content managers.
    • If you can’t access website records, you’re missing login or password information, or you’re uncertain who your hosting provider or domain registrar is, go to http://DNSstuff.com or use the “WHOIS Lookup” tool (https://whois.icann.org/)

Resources

  • TechSoup Disaster Planning and Recovery Guide – this helpful resource, developed by TechSoup in collaboration with the Center for Disaster Philanthropy, is a ‘holistic guide to IT disaster planning and recovery’ organized in 3 sections: Preparing for a disaster, Recovering from a disaster, and Helping your staff and volunteers prepare. https://www.techsoup.org/disaster-planning-and-recovery
    • IT Assessment and Continuity Plan – worksheets to help you develop a responsive IT disaster plan and identify potential gaps in your current plan
    • The Resilient Organization Workshop – a guide outlining best practices for preparing an agency’s IT staff for most kinds of disasters, including natural disasters and cyber-attacks, and how to recover.
  • What To Do When Collections Get Wet – this guide, produced by the Library of Congress, focuses on mitigation steps agencies can take when physical records are damaged.
    https://www.loc.gov/preservation/emergprep/dry.html
  • Ask an IT Guy: How Long Does It Take to Restore From a Cloud Backup? – this helpful post breaks from a third-party IT support company answers some of the chief questions about cloud backups and considerations for using this tool as your primary restoration plan.
    https://blog.accentonit.com/how-long-does-it-take-to-restore-from-a-cloud-backup

 

All Risk Resources