Knocked offline: The Finalsite outage

On Tuesday, Jan. 4, thousands of school websites suddenly began taking unusually long to load and, ultimately, crashing. The common link? They all were maintained by Finalsite, the Glastonbury, Conn., company that, along with Blackbaud, dominates the business of providing sites to K-12 institutions.

Finalsite reports that its clients’ sites receive 50 million page views a day. At the height of admissions season and with updates to share about Omicron and COVID-19, however, these sites would be useless to schools for most of a week.

This month’s Refill will look at this widespread outage, including how Finalsite communicated it, how some schools responded with workarounds, and how communicators can prepare for the next time the website goes down.

The first Tuesday of 2022 goes awry

On the morning of Jan. 4, Michael Petitti was primed to announce a website redesign for Radnor Township School District, in the Philadelphia suburbs. This was Radnor Township’s first site redesign in more than a decade, and Petitti’s team had proudly built a video introduction, which would go live at lunchtime.

In northern Colorado, meanwhile, Randy Barber was preparing back-to-school updates for Boulder Valley School District. This information to families was a critical need for a community recovering from a devastating wildfire on Dec. 30, one that destroyed more than a thousand homes.

Around 1 p.m. EST, both districts’ websites joined thousands in returning error messages that pointed to an issue at Cloudflare, the content-delivery network Finalsite uses.

The outage drew little immediate public notice, perhaps with people assuming it was both temporary and isolated. Radnor Township was one of the few to mention it Tuesday on Twitter; Petitti said that was only because people were looking for the redesign after seeing the promo video.

Finalsite acknowledged the outage on a channel used primarily by technology offices, with a status update that said, “We are currently investigating an issue leading to increased error rates and performance issues across our legacy modules, and our team is investigating a fix as a critical priority.”

This explanation, however, was not entirely accurate.

Reading, ’riting and ransomware

The error rates and performance problems were caused, Finalsite would acknowledge 24 hours later, by the company itself abruptly taking client sites offline.

Why had it done so? That information would take yet another day to come out: Finalsite had detected ransomware in its systems.

Ransomware is a serious and growing problem for U.S. businesses, especially those that hold personally identifiable information, such as names, Social Security numbers and dates of birth, and protected health information. The concern is particularly potent for schools, which commonly hold troves of such information.

Education Week reported this month that K-12 schools are increasingly being targeted for such attacks. So, too, are their providers: According to Inside Higher Ed, companies such as Finalsite are a more attractive target than individual schools because attacking the firm has a broader impact, which makes a ransom payment more likely.

Blackbaud, Finalsite’s keenest competitor, suffered a ransomware attack in early 2020 and paid ransom to restore service. Blackbaud initially denied that any personally identifiable information was compromised in its attack, but it later walked that back in an SEC filing. Blackbaud faces a class-action lawsuit over the matter in a South Carolina federal court.

The website TechCrunch reported that most losses in a ransomware attack come “through lost productivity and the time-consuming task of containing and cleaning up.” It also cited recent research from Cybereason showing that more than half of U.S. companies saw their brand and reputation damaged by a ransomware attack.

Finalsite disclosed the ransomware to clients on Thursday, Jan. 6, in an email from CEO Jon Moser. He also described its discovery and how the company had responded.

This did not signal that the problems were solved. Websites would remain buggy, with styling, images, and directory information missing in many cases, into the following week.

Assessing the company’s messaging

But Moser’s email did mark a significant change in Finalsite communications after two days of public silence.

The company’s typically active Twitter account returned to regular postings, responding at last to increasingly anxious tweets from clients looking for information to pass along to parents or administrators.

A week after the outage, the company published a “communications hub” that collected status updates, emails that were sent on the matter, and even sample messages schools could send to parents about the situation. It is a model of transparency about a painful corporate episode.

Rick Kaufman, an expert on crisis communications who works for the Bloomington, Minn., school district, says, “Finalsite took the right steps.” He credits the company’s comms team, which “took control of the narrative to ensure transparency over the whole process for stakeholders, clients and its employees.”

I can’t agree with this view. Eventually, Finalsite got exactly to the point Kaufman describes. But its initial messages, I believe, left much to be desired.

I assess crisis communications by four criteria:

Speed matters. You want to be the one talking publicly about a critical event before anyone else. Finalsite acknowledged the problem only to its clients for the first 48 hours. (To its credit, the company has recognized that this was a mistake.)
Accuracy matters. In a crisis, you have only one chance to make a good first impression, and establishing yourself as trustworthy is imperative. Holding back on important details will cost you credibility if and when they come out later. Here, Finalsite’s initial implication that forces out of its control caused the outage was, I believe, misleading.
Completeness matters. Your audience will pay far more attention to your first message in a crisis than to any of your follow-ups. Finalsite published 21 updates about the outage before acknowledging that the central issue it faced was ransomware. Moser’s letter suggested that the reticence was advised by Finalsite’s third-party experts; if so, that strikes me as poor communications counsel.
Tone matters. In crisis communications, you must put yourself in your audience’s shoes: How will this information be heard and internalized? What needs or concerns of theirs must you address? In far too many of its early updates, Finalsite focused on Finalsite — how hard its employees were working, the successes achieved, the assistance it had enlisted — rather than on the needs of their clients. Meanwhile, schools were left on their own to explain the outage — and, later, the ransomware attack — to their communities.

I offered Finalsite Director of Communications Morgan Delack a chance to respond to my Monday-morning quarterbacking, and she replied:

“We are always looking to grow and improve and know we didn’t do everything perfectly regarding the ransomware incident. However, we do feel that we were as transparent as possible with our communications and always seeking to put our clients’ needs first. This included holding many client webinars, writing parent communication templates and even providing my own personal cell phone number that clients could use for communications support 24/7.”

I’m grateful to Delack for her willingness to engage, and this is a good time to mention that I admire what Finalsite does regularly to inspire and help schools. Initiatives such as School Marketing Day and Finalsite University have deservedly built a lot of good will in our industry.

Clients get creative

By Wednesday, Jan. 5, with no fix to the outage in hand, Finalsite clients started implementing their own solutions.

The 56-school Boulder Valley district in Colorado published a placeholder website on Google Sites in time for the resumption of classes. At first, it was a rudimentary page with stock photos, said Randy Barber, the district’s communications director. But his administrative assistant took charge of remaking the site throughout the day to fit the district’s brand and of adding content that parents and administrators were requesting.

“We thought that we could stay really slim, but we also were listening to what people were needing, like access to COVID dashboards,” Barber said.

By Thursday, the district faced a new wrinkle: President Biden was coming to town to see the wildfire damage firsthand, forcing a bunch of afternoon bus-route changes to announce online.

The website outage has prompted Barber to reconsider his office’s resilience to a future loss of computing resources.

“When your website is not there, you suddenly realize what you don’t have: pages that are in our quick links but not in our bookmarks, systems that hold your photos,” Barber said. “We’ve got to have redundancy on all the things we would need to stand up a backup site.”

In Pennsylvania, Radnor Township School District would also establish a placeholder site on Google Sites. Communications director Michael Petitti said he was grateful that his district already had some online redundancy.

“We were fortunate, in a way, that we don’t use Finalsite for our alert system,” Petitti said. “We were still able to let parents know not only about the website outage and that another site was available, but also about our virtual school Friday,” when snowy weather pushed classes online.

Petitti said he also reached out for information from his Finalsite representative, “who was really responsive; he didn’t duck behind his email.” Conversations with the company led, for instance, to a better page-timeout message than the one that mentioned Cloudflare.

“It was tough for us, navigating [the outage]. We weren’t expecting it,” Petitti said. At the same time, he added, Finalsite “did all they could.”

Finalsite’s Delack said the company heard similar sentiments from more than 100 clients, complimenting how this incident was handled technically and how information was communicated.

Action steps for schools to consider

What can schools learn from this experience, even if they aren’t with Finalsite? The big takeaway is that web outages will happen. Before one hits you, here are suggestions to minimize the pain:

Bring in your IT department for a chat. If your web provider goes down, the entire site will go with it, but you can quickly stand up a single page of information — a static home page. This is something you and IT can build ahead of time offline and then activate when needed. When you activate it, you’ll want also to redirect your web traffic, so that people who type in your primary URL are directed to, for instance, a Google Sites URL, rather than your normal web provider. This step is not technically difficult. Nevertheless, work with IT to create a process for the redirection (including the assignment of who will execute it).
Make a plain-text file of your essential web information: What are the must-haves for your static page? Some of this will be evergreen, including basic contact information and other online channels a visitor can use. Other content will be situational, such as an update of the web situation. And still more will adjust by the seasons, such as admissions information or graduation details. It’ll be easier to finalize your static page if you already have a rough draft on hand before trouble arrives.
Grab and store one image from your home page. A static page won’t run an entire carousel of photos, but an iconic brand image for your temporary site will visually assure visitors that they are in the right place.
Save these break-glass-in-case-of-emergency files in two different ways. Use a cloud-storage service (think Dropbox or OneDrive) so you can access them anywhere, along with a removable-media option (a thumb drive) for redundancy. That way, if the cloud goes down, you’ll still be in luck.
Look into an outside mass-communications provider. When Finalsite took sites offline, it also deactivated bulk-mail service. As Radnor Valley found, an outside provider of mass communications — such as Blackboard, Brightarrow, SchoolMessenger, and SwiftK12 — gives you a second way of contacting families in this situation. You’ll incur a per-user cost, and you’ll need to practice using the secondary system. Both are annoyances, but when you need a system like this, you really need it.
Draft a holding statement about the web outage. Remember, you want to inform people of an outage before they find out on their own. Keep the statement simple and brief, and use this opportunity to highlight your other online channels. And when the website goes down, use all your platforms to alert people to the outage. Don’t force people to guess that something’s gone wrong.

Finally, schools should use this episode as a prompt to assess their specific risk management for a ransomware attack.

With such attacks becoming common, schools should look into their vulnerabilities, consider hardening their systems, and prepare for the worst-case scenario. Communications offices, with their many contact points with the public, should be a major player in this sort of institutional review.

Don’t-miss clicks

I was surprised to see the National Association of Independent Schools claiming copyright of presentations made at one of its conferences. NAIS acted to block a disgruntled educator from publishing on Twitter video clips from the association’s People of Color Conference. Still, the implications of this move are complex and novel. Do presenters understand they are turning over “intellectual property rights” (NAIS’s language) when they submit a conference proposal? Since it’s common for presenters to repurpose slides for different audiences, how would NAIS establish what is “theirs”? Or is the act of videostreaming the material to attendees what establishes copyright?
Undergraduate college enrollment fell 3.1% last fall, the 10th consecutive year of decline, reports the National Student Clearinghouse Research Center. The decade-long recruitment-and-retention crisis “has translated into deep cuts at many schools as well as an uptick in school mergers and closings,” according to the Wall Street Journal.