A quick response, statewide partnerships and a long planning process helped the City of New Orleans mitigate what could have been a disastrous ransomware attack just over a year ago.
City officials spotted suspicious activity on New Orleans' servers around 5 a.m. on Dec. 13, 2019, and quickly went into recovery mode within two days, while also declaring a state of emergency and calling in state partners to help out. The Ryuk gang, which holds data hostage in return for payment in bitcoin, was identified as the culprit.
The attack came after the city put in weeks of preparation, inspired in part by similar cyberattacks across Louisiana, which included carrying out emergency exercises and having a plan for how to back up the city's data.
"We just started bracing and making sure that we took every action," Kimberly LaGrue, CIO of the City of New Orleans, said. "We've been developing our cyber response program and our disaster recovery program over a period of years. For us, these were well-rehearsed plans that we just had to enact immediately."
Jumping to action
In the immediate hours after the attack, the city brought in officials with the Louisiana National Guard, Louisiana State Police (LSP) and the Louisiana State Analytical and Fusion Exchange to help with response and recovery. Spokespeople with the National Guard and Fusion Exchange did not respond to questions about their involvement in the mitigation and recovery of the attack.
"For us, these were well-rehearsed plans that we just had to enact immediately."
Kimberly LaGrue
CIO, City of New Orleans
A spokesperson for LSP said its Cyber Crime Unit (CCU) is mainly focused on criminal investigation of a cyberattack and determining whether charges need to be brought but did not comment on the status of any ongoing investigations.
"LSP Cyber Crime Unit's role is to make a determination of whether a crime has occurred and if so, collect evidence and conduct an investigation," the spokesperson said in an email. "CCU then makes a determination of whether the victim requires assistance from our partners at the Louisiana Office of Technology Services and/or Louisiana Army National Guard for rebuild/restoration."
Local staff mitigated the attack by quickly shutting down all the government's 470 servers and thousands of computers, meaning local services were temporarily stopped. But that action meant that the malware could not infect all the city's data, so the attackers were not in a position to demand a ransom like they in other cities.
The incident also served as a stark contrast to similar incidents that occurred previously in Louisiana. Earlier in the year, a number of state agencies were hacked, forcing them to temporarily shut down their operations and only reopen slowly as computers and servers had to be rebuilt. Several school districts in the state also suffered from similar attacks.
But for New Orleans, having a plan in place to quickly shut down systems in the event of an attack, as well as the decision to declare a state of emergency, gave the moment a sense of urgency and prevented further spread. And having quickly backed up its data, the city was able to resume services a couple of days later.
"Planning is the most important part of the game," LaGrue said. "The maturity of an incident response plan is one thing, but just having an incident response plan is key to surviving any of this, understanding what type of data you have and where it lives."
Moving into recovery
The recovery process has been ongoing, as the city needed to replace its infected IT infrastructure. Those recovery costs are expected to total $5.2 million, LaGrue said, with the city claiming the money back through its insurance and the state also looking to provide assistance. Mayor LaToya Cantrell said last year those costs exceeded the city's $3 million insurance policy, but that she planned to raise it to $10 million this year.
The recovery also involved replacing some of the city's IT storage infrastructure to help it get back online quickly and with minimal disruption to service. That new infrastructure came from Pure Storage, which brought in its FlashArray and FlashBlade systems for storage and disaster recovery and got them functional in a matter of hours.
"The biggest thing that we find, especially in these particular situations, is that time is really the enemy, and complexity adds on to that," Mike Wiseman, VP of State, Local and Education (SLED) at Pure, said. "From our perspective, it's really making sure that the organizations that are impacted, especially in state and local government, have a disaster recovery plan in place, they have the ability to go ahead and isolate the traffic that's being impacted, and between different nodes on the network."
Adapting to COVID and looking ahead
The onset of the coronavirus (COVID-19) pandemic could have disrupted the city's recovery efforts, especially as its employees were forced to work remotely, as in many cities nationwide. But LaGrue said New Orleans' planning process mitigated any negative effects, as city employees have contingencies in place if they are forced out of the office due to a natural disaster or other unforeseen incident.
"We have to work smarter; we have to stay ahead of the game."
Kimberly LaGrue
CIO, City of New Orleans
And while there were new tools and new infrastructure added after the cyberattack that still needed to be learned, the pivot to fully remote work was less painful than it could have been.
"In the three months that led up to the pandemic, from the time of the cyberattack, we got really adept at working differently," LaGrue said. "That was because of the cyber response and the challenge from the city and the mayor to keep city operations going, so we were already in that mindset."
Ransomware attacks on public entities, including cities, have intensified in recent years, with attackers using increasingly sophisticated techniques. For cities, with staff all forced to work remotely due to COVID-19, the vulnerabilities have only increased. Cyber threats need to be taken seriously, LaGrue said, especially as the cost of recovery far outweighs the cost to prepare.
"We have to work smarter; we have to stay ahead of the game," LaGrue said. "We have to realize these are legitimate threats, and the attraction that cyber actors have to governments. You would think that they would be more attracted to banks and private companies, but the risks to governments are real."
City leaders can never start planning for those cyberattacks too soon, LaGrue said, especially as the attacks are growing more sophisticated and make increasingly onerous demands on those held to ransom.
"The challenge for government and city agencies, unlike the private sector, is our resources are limited and often constrained," LaGrue said. "We have to recognize these as legitimate threats ... My advice to my colleagues is to understand and recognize that these legitimate threats are taken very seriously."
