By the time you read this, Hurricane Florence will be a distant memory for most of you. However, for those of us who live at Ground Zero, part of which is Wilmington, NC, the memory of Hurricane Florence will last for a long time.
As I write this, the sound of chainsaws cutting up fallen trees, and trucks carrying away debris in my neighborhood, are loud and clear and will remain so for days to come. There are many homes here that were flooded, had trees fall on them, or both. These homes are damaged beyond repair, which means their owners will be on a first name basis with an insurance adjuster and the mortgage rep at the bank for the foreseeable future. Even now, I can’t leave Wilmington, because all of the roads are washed out. Yes, it is really that bad.
At the time of writing, it is exactly seven days after the hurricane, and businesses are trying to get back up on their feet. Some are having far more success than others. For some businesses that were not flooded and didn’t have a 3-ton Oak or Pine fall through the ceiling, getting up and going again is just a matter of getting back power and Internet (more on the Internet part in a bit). For others, it means calling IT Providers and asking for offsite backups. And this is where things get dicey.
Here is what I learned about Disaster Recovery at Ground Zero of a Hurricane:
Not everyone had their servers, workstations, firewalls, cables, printers, etc. insured with their property insurance. Many had the building insured, along with the furniture, but their $8,000 Dell Server, and their twelve $900 Dell workstations, and their $800 SonicWall Firewall, and $120 NetGear switch were not insured. So, the first step these businesses must take is to order new equipment and pay for it all out of pocket. And don’t forget the labor for IT to put it all back together.
Not everyone had tested their backup by doing a test restore previously, nor had they kept the backup software/device updated. This disaster is the first time they are restoring from a backup many had in place for years. You can smell the fear as they pray it works because they are attempting something they have never done before. Still many more had backup software that was three or four versions behind which means it is no longer supported by the vendor. Again, you can smell the fear.
Not everyone has a line of credit to keep the business going while rebuilding. Getting new equipment shipped in from Dell, HP, etc. is going to take a couple of weeks or longer. The roads are washed out so there is no way in or out of town. Offsite backups must be copied to a USB drive, and then shipped out because no Internet means they can’t be downloaded, but just getting backups in may take days. Again, the roads are washed out. Even after all that arrives, the building must be repaired and made ready and all the construction crews are already tied up on other jobs. Once all of that is done, it will take IT two or more days to put everything back together depending on the size of the business. Some businesses will be down a month or more while bills are stacking up in the mailbox, with no option other than to call creditors and beg for mercy.
Not everyone can find installation media and license keys for line-of-business software. For some, once the equipment arrives and is set up, they are still stuck. They can’t re-install line-of-business applications, because they have lost installation media, don’t remember the password to log in to the vendor portal, or can’t find the license key. Vendors are often happy to help, but this still means more time waiting on hold on the phone or waiting for replacement media to be shipped out. Furthermore, some vendors are happy to help, but for an additional fee.
For some that must reinstall their line-of-business application and can’t find the installation media or license keys, they must also upgrade at the same time. Some small businesses have a line-of-business application, yet it’s several versions behind. The vendor no longer supports the old version they are on. Therefore, not only does the business have to contact the vendor to get a new copy of the software, they also must perform an on-the-fly upgrade at the same time. This adds unwelcome layers of complexity and takes more time to get things going again. Not to mention costing way more money as the upgrade must be paid for. Oh, and the best part, the new software has a new user interface that users aren’t familiar with.
The Cloud may not save you. You have all your stuff in the cloud, which means you don’t have to back up anything. You have nothing to worry about-until you’re told it will be another week before you get Internet back at your location. There is nothing you can do but wait. And all that is assuming your building and other assets are undamaged. Even with no flooding and minimal damage, you still have no choice but to wait through the countless days until Internet service is restored. Your neighbors who have all their stuff on-premises and suffered only minimal damage as you did are already back up and running and seeing customers.
For some, they discover not everything was being backed up. The business has new servers and new workstations in and backups restored. Yet, things are still missing. For example, the Excel spreadsheet of clients who are behind on invoices, the PDF files from the attorney regarding an existing case, and other such documents that were being stored on a workstation-rather than the server-are gone. The workstations weren’t being backed up like the servers were.
These are the things I discovered/observed during Hurricane Florence here in Wilmington, North Carolina. While no disaster recovery plan will be perfect against something as powerful as a hurricane, many, if not all, of the situations described above can be avoided. And most of it is just common-sense and thinking ahead. Yes, you should have property insurance on your IT equipment. Yes, you should be able to put your hands on license keys and installation media. Keep that material in a lock box or brief case instead of scattered about the office. Keep line-of-business software updated. Depending on the vendor, you may not need to update every single time a new version is released, but at least stay on a version that is supported. Talk to your bank about having a line of credit for emergencies and make sure it’s enough to handle replacing all IT equipment and keeping the bills paid for at least 30 days. Also, keep documents you need backed up on a file share on the server instead of on the local computer. Or, have your IT set up folder redirection so your My Documents, Desktop, etc. are redirected to the server behind the scenes for backup even though you are saving files to your My Documents.
And last, but far from least, test your backups. If you haven’t taken a backup and restored it to a test server, then verified that backup by logging on to that test server to make sure it is an exact replica of what you have in production, then you have zero proof your backup works. Zero proof. Don’t let the day you need your backup be the day you test it. And keep your backup software/devices up-to-date.
Many businesses here in Wilmington will survive to see the sun rise.
Some will not.
James Summerlin
