Because cyberattacks have become so commonplace it’s understandable that IT teams facing acute cyber malfunctions will jump to the conclusion that they’ve been hacked. But a high-profile incident last month points up the fact that entire computer systems can appear to be hacked and closed down, when they’re actually brought down by innocent mistakes. Regardless of the cause, your incident response team still has to respond quickly and appropriately, and that means having a state-of-the-art incident response mobile app.
Case in point: On Monday evening, September 23, video editors of the popular TV show, “Modern Family,” based in Hollywood, were working under their typically extreme deadline pressures. Suddenly the editors found that they couldn’t start their Macs and get to their editing projects using software called Avid Media Composer. Avid makes the most commonly used professional editing and post-production software for movie and TV production.
Immediately, social media sprang to life revealing that the issue was not just in the “Modern Family” editing suite. The same inability to reboot and open the Avid software was happening to professional editors across Hollywood and elsewhere throwing a big slice of the TV and movie industries into turmoil.
This was one post by Avid Editors of Facebook:
Tuesday, September 24th, came and Avid released a statement that it was aware of the issue and was working hard to fix it, erroneously believing that the glitch might have been caused by a problem with its software. By the afternoon, Avid’s CEO announced that the company was working “around the clock, whatever it takes” to fix the problem. At the same time, social media users were speculating that it might be an Apple problem if video editors hadn’t updated their Apple operating system. One blog site called Mr. Macintosh hypothesized on Tuesday afternoon that the Avid shutdowns may be connected to Google’s Chrome browser.
Cyberattack or cyber mistakes, nobody knew what the problem was.
Finally, Google stepped forward to announce that the problem had nothing to do with Avid or Apple directly, but actually was the result of a fault in its recently released version of its Chrome browser. The update included something called Keystone software, designed to automatically download Chrome updates. The bug in the update corrupted any Mac computer whose user had previously disabled Apple’s System Integrity Protection, which was necessary to do in order for them to connect to external audio and video devices. Once on board the bug would corrupt the computer’s file system, shutting down the computer and much of the TV and movie editing in Hollywood and beyond.
No cyberattack this, but rather four organizations entangled in a subtle, but high impact, series of human errors:
- The Google Chrome update bug
- Apple’s System Integrity Protection software that had to be disabled in order to accomplish certain editing tasks
- TV and movie editing studios whose Mac users had disabled their Apple System Integrity Protection
- Avid whose editing software was affected by the bug
Did editors, Google, Apple and Avid coordinate immediately? They did not. They didn’t know what they didn’t know. Avid’s IT team and most everyone else erroneously thought the Avid software was to blame.
Thankfully, the problem was solved within 24 hours after Google, Avid, Apple and editors were finally able to coordinate with each other to compare notes, and Google could announce corrective measures to get the Avid software up and running again.
Now, consider what we can learn from the experience and how it can be solved.
When production houses’ editing computers crashed, they were in a crisis, completely unable to work.
An event that might at first appear to be a cyberattack could in fact be completely innocent human error, or, as in the Google/Apple/Avid/Editors case, a combination of innocent errors.
Verizon’s 2019 Data Breach Investigations Report states that while rogue insiders stealing data and otherwise creating cyber havoc “makes for a good story” insider-generated computer problems are mostly the result of very human error.
The possibility that different companies’ software may interact in strange or damaging ways with other companies’ software, should have led professional editors, so dependent on a critical piece of software for their livelihoods, to be able to coordinate their responses to an IT incident with all of their software vendors. And those vendors, likewise, should have been able to coordinate with each other seamlessly.
Their incident response planning should have included the very real possibility of destructive software interactions that have nothing to do with a cyberattack. Every organization’s incident response team could have had contact information for their counterparts at the various software vendors readily accessible on their incident response mobile app. The app would have enabled much faster coordination and collaborative management of efforts, resolving the problem much quicker and, importantly, with a lot less stress for all.
If response teams at affected editing studios been equipped with the incident response mobile app, they could have…
- Assessed the scope of damage across facilities
- After seeing how social media was speculating that the Avid problem may be related to Apple and/or Chrome software and how they all interconnect, they could have alerted their counterparts in the relevant software companies, and they could have worked together to manage the incident.
- Secured all production work verifying the existence of backups
- Implemented any preliminary measures to minimize additional loss (i.e., don’t reboot the machine, which many postings on social media suggested)
- Determined the impact on production schedules and modified timelines to accommodate delays
- Notified clients
- Mobilized resources to repair machines once a fix became available
- Kept everybody posted during the entire process
- Once fix was available, tested and brought facilities back online
- Re-established full production
None of these vital incident response tasks could ever be performed well or even at all in the heat of the moment without a pre-vetted and interactive incident management plan that had been incorporated into the response team’s mobile app.
Whether your organization edits sitcoms or develops software, it pays to have the best tool to manage the unexpected, cyberattack or human error, and get back to business as soon as possible.