SolarWinds became the poster child for novel nation-state cyberattacks last December. The company was used as a tool in a supply chain compromise in a way no one had seen before.
When FireEye, now Mandiant, disclosed its breach, the incident raised eyebrows across the security community — a security company is not supposed to get hacked. And then the downstream victim list began to grow beyond FireEye, launching a collective effort to understand how and why hackers leveraged SolarWinds Orion.
Cybersecurity Dive spoke with SolarWinds CISO Tim Brown about his experience as a security and business leader during an unprecedented cyberattack amid an already unprecedented year.
This interview has been edited for clarity and brevity.
CYBERSECURITY DIVE: What do you find people get wrong about the SolarWinds hack most often?
TIM BROWN: Just a number of things, right? So I guess most often, they're not understanding who the attacker was and what the attackers' capabilities were. And that's a big one.
When you're attacked by a nation state, that's [being] attacked by an army. And who do you expect to be able to defend against an army? You expect the private company to be able to defend it against an army? Probably not.
The amount of thought that they put into it was absolutely a military mission. … The sophistication of the actor is one thing that everybody needs to be prepared for. And they need to be prepared for that level of sophistication to start doing ransomware attacks, organized crime attacks. That's one of the other things that people don't quite, you know, understand.
I just kind of want to hear about your "oh, crap" moment. When you got the call and the minutes after.
BROWN: It was crazy. So our CEO got a call in the morning from [Mandiant CEO] Kevin Mandia. And then he called me, and then the CTO for FireEye called me. That's our nightmare moment.
So we were in the office for a couple of weeks straight, we didn't get out until two or three in the morning. So much goes on — you can test and test and test — but the level of intensity, the level of things that need to happen, the level of things that need to happen immediately, is just extremely hard to anticipate until you live with it.
We were spending time with customers all day long, and with the researchers. We were in here at night reviewing docs and making sure everything was right before we published. ... You've got to go through every word. It's got to be correct. It's got to be well-phrased.
The first few weeks were just crazy from everything that you had to do. Support lines were going mad, as you would expect, so we had pulled people across the company to answer support calls, and try to get our queues down.
One of the things that we did well was we split ourselves up into different war rooms. We had people focused on the internal investigation, we had people focused on the development investigation, we had people focused on marketing communication, people focused on business escalation, we had people focused on legal. But each one of them had a leader and we came together every afternoon to talk about where we were on each one of those streams. If you tried to do this with one stream, you would fail.
And then you can't forget your people, the employees themselves. [Using] the analogy of somebody breaking into your house — they're all going through emotional roller coasters here too. Communicating to the employees about what's going on also is really, really important.
How did you get started in cybersecurity? I ask because I want to know, at any point from your start up to now did you anticipate getting attacked by a nation-state actor?
BROWN: I ran development teams, I ran engineering teams, I built cybersecurity programs. I was one of six Dell fellows. So I really built up a lot of focus, both internally and externally for people.
A lot of advice I've given to governments, I've given to the world on how to head forward. So did we think about nation states? Yeah, we thought, hey, this could be possible. But possibly, theoretically, is very different than possibly, practically.
I talked to a number of large companies or entities that would be [susceptible] to a nation state. They'll tell me, "Yes, it happened to us. Yes, we know these threat actors, but it just didn't go public. We didn't talk about it, but they're there." So by us coming out and talking about it, I think we make it real for people making it so it's not theoretical anymore.
We're humble, but we're trying to be accurate and truthful around what we do. Most CISOs start from a technical perspective, and don't necessarily have that same level of model or skill set. So it's something that we have to develop as CISOs — being both internal and externally focused.
Is there anything you feel like non-technical or non-security business leaders didn't understand until the last 11-12 months?
BROWN: One thing is that they didn't understand that this can happen.
I think that the incident sparked a lot of conversations for many companies on what their security programs were, what their security programs needed. [When I talk to] CISOs around the world, they thank me for helping with their budget.
So in that way, the event itself has spurred more security across the world. Not just for us, not just for our clients. … "Could we get hit in this way?" Is the question the boards are asking, which is a very, very healthy conversation to occur.
One of the interesting parts is when this happened, we got calls from every country of the world — believe it or not, the countries call you. And the countries say, "Can you give me a list of all the customers in my country?" … We found them very great partners, and that their agenda was simply to amplify messages. Their agenda was to help the customers become safe.
One of the things SolarWinds did almost immediately was create a cyber-specific committee on your board. Did you have any realizations throughout the year of "why didn't we already have that?" Or "why didn't we already use that?"
BROWN: So the technology committee was there, but it just brought up the fact that we need to specifically have that cyber committee as part of the board.
Some of the things we should really think about from the industry perspective is, how much thought are you putting into all of these things. A lot of people put technology in place and the technology is there. But in this case, the actor was, again, very, very thoughtful. So activities that we've been doing since the incident had been both technology implementation … and thinking ... That's what they did well.
The message is that we have to out-think the adversaries, out-mission the adversaries, out-prepare from the adversaries. It's not just about putting technology to it. It's not just about processes. It's taking that time and effort to really out-think them.
Is there anything else that you wish that SolarWinds had communicated more clearly? I know that the investigation was unraveling in real time in December.
BROWN: We found that we didn't have the right list of people to communicate to. We put information to every email contact that we had, but those email contacts happened to be the sales contact. One of the things we've revamped was to give ourselves a security contact for each one of the companies. Now we have that as part of Salesforce. …
We tried to amplify messages. We tried to push information out as much as we could. We agreed with a lot of what the [Cybersecurity and Infrastructure Security Agency] folks were amplifying. So that helped.
Our first message said, "18,000 companies were affected," simply because that was everybody that downloaded [the update]. That was a high number, right? … Everybody was affected, everybody had to do an investigation, so I'm not lessening that. But the ones that went to a stage two attack, the ones that were actually targeted, were under 100.
So big difference from 18,000, down to 100. So do I wish I knew that was 100 on day one? Yeah. But you can't necessarily know everything when something starts.
Is there anything this taught you or changed you as a cyber leader, a business leader or as a person?
BROWN: Yeah, one of the things that was really, really helpful — which is something completely aside that I didn't expect — was the number of people that reached out with support. And that gave me a big lesson. I reached out to the Kaseya guy to say, "It's okay, you'll get over it."
But the level of support from the community and people coming back and just saying, "Yeah, we've got your back." The level of appreciation that both government entities and corporations gave us for being transparent and helpful goes a long way. It goes an incredible amount of way to just kind of humanize the problem and say, "You're doing the best you can, and we know it's difficult, but you'll get through it."
That's one of the things I encourage others to do: When you see people that are in a difficult situation, reach out and give them a hand if you can. A kind word goes a long way.