The year-end scramble to tie up business before the holidays was particularly fraught last year. FireEye, now Mandiant, disclosed a nation-state cyberattack. Then the Treasury Department confirmed a breach. What became apparent was a widespread supply chain attack against one common source: SolarWinds. 

SolarWinds Orion, a network management system (NMS), is a piece of software — and the leading NMS player — throughout the public and private sectors internationally. While about 18,000 Orion customers were impacted by the compromise, only a fraction suffered secondary attacks. The Russia-based threat actors knew who and what they wanted, targeting a tool with a privileged position in customer networks. 

Unlike highly regulated industries, such as the automobile or pharmaceutical sectors, software providers lack high levels of legal accountability for vulnerabilities and risks. But the SolarWinds hack caused government and industry leaders to rethink how software is made and secured, giving rise to close scrutiny of the software supply chain.

For SolarWinds, the newly minted poster child of software vulnerabilities, the attack, recovery and ensuing fallout came at a cost. 

In the first nine months of 2021, the Orion breach cost SolarWinds $40 million, the company's quarterly report from October said. Though partially offset by insurance, the company is carrying the financial and reputational burden from its unprecedented cyberattack disclosed last year.

"Although the ultimate magnitude and timing of expenses or other impacts to our business or reputation related to the cyber incident are uncertain, they could be significant," the SEC filing said. 

Russian threat actors manipulated SolarWinds' software build environment — injecting malicious code within a millisecond window of the build process. The changes were subtle and undetectable, leading SolarWinds to entirely rethink its build process. 

As part of its recovery, SolarWinds is upgrading its software build environments using its "Secure by Design" initiative. The company estimates the ongoing costs of "Secure by Design" will reach $20 million annually, SolarWinds said in its earnings statement. 

"Changing a build environment and changing out your build systems for legacy stuff doesn't come cheap," said SolarWinds CISO Tim Brown. "It's six months of effort, working very diligently on something that is not necessarily going to return you revenue because it's going to build the same thing it was building last time." 

"So you'll get some efficiency gains, but you won't necessarily get a new product feature to sell," he said. 

Software makers have to calculate how much risk exposure build environments have and how much they can tolerate, mapping the potential damage if one account or machine is weaponized, according to Brown.  If the entire environment could suffer from one incident, safeguards either need to be added or upgraded. 

"Hopefully the incident provided the catalyst for folks to be thinking about it," Brown said. 

Renewed software scrutiny

The changes to SolarWinds' build process preceded President Joe Biden's May cybersecurity executive order, which established improvements for the software supply chains. Among its recommended enhancements was a requirement for a software bill of materials (SBOMs) for federal contractors, which includes SolarWinds. 

What the executive order aims to achieve is a trickle-down effect for SBOMs across the tech industry. And the government has wide-reaching influence solely based on its procurement powers. 

"Companies should not look like deer in the headlights" when building a secure development lifecycle, according to Eric Byres, CEO of aDolus, a software supply chain firm for OT. Companies have access to training and resources for running a systems development life cycle (SDLC) process. 

Companies have two concerns: How is the development system protected and how is the software protected? 

"People kind of knew that if you didn't protect your software, you're going to get your head handed to you," he said. 

Room for error

The SolarWinds hack showed the U.S. government the rising sophistication of adversaries' cyber capabilities. The Orion compromise is "probably one of the two most significant attacks ever. And the other one was Stuxnet," said Byres. "Both of them were intended for military, political, or intelligence purposes." 

Stuxnet was a worm originating from the U.S. National Security Agency, and in 2010 the hardware of organizations in utilities and energy in Iran were targeted. 

"Stuxnet sort of set the bar that everything goes — it's fine to attack another nation state using cyber, for good or for bad. That's where we're now," Byres said. One of the predominant domino effects of Stuxnet was a collective international interest in obtaining similar cyber capabilities. 

Even a year later, Sunburst, the malware distributed when customers updated Orion, is a monumental piece of malware because it personified how low and slow a foreign adversary can live in a private sector company to reach government targets. 

Sunburst "demonstrated creativity, technological advancement, patience, and scope that has never been seen before," said Lotem Finkelstein, director of threat intelligence at Check Point Software. For Finkelstein, labeling the SolarWinds breach exclusively as a supply chain attack "ignores the fact that this attack was designed to keep a grip on technology companies." 

The Russian threat actor, Nobelium, used its knowledge of SolarWinds' products and build process to launch espionage campaigns on the tech and security industry. By January, SolarWinds was working to identify its assets, and overhaul what develops and protects them. 

In the first three months following the disclosure of the breach, as part of SolarWinds' "Secure by Design" initiatives, the company began to: 

• Adopt multiple software building pipelines using different administrative domains and controls
• Use just-in-time access for multiple environments
• Evolve security recommendations to accommodate customer environments, not just SolarWinds products
• Develop a Patch Tuesday-like cadence for security updates

The changes SolarWinds made to its build process were "pretty much the first of its kind," in addressing security in the build process, said Finkelstein. "I think all major victims acted in pretty much the same manner" because it allowed for better post-incident investigations across the tech industry.  

"This was extremely significant. We should not take it for granted," said Finkelstein. 

Brown's background is steeped in engineering and development, and the company is using his expertise to oversee SolarWinds' security operations and products. "My team is not doing the work, but my team is reviewing what's going on," he said. "My team is involved in watching and recommending — not so much doing the work," as it's primarily owned by the CTO and architecture. 

For engineers at SolarWinds, the hack felt like someone "came into your house, stole your stuff, and left or changed it," said Brown. While the security and engineering teams had a working relationship prior to the hack, "this just set into another level of importance" for collaboration. 

"So some of the barriers that people face, you know, we didn't face as hard barriers for change," between security and development, said Brown. 

An IT and security disconnect

When SolarWinds disclosed the breach, CIOs may have known what role Orion had in their technology stack. But CISOs "and the network defender teams may not have had a deep understanding of the Orion product and platform," or an understanding of how critical Orion is to maintain network operations, said Chris Krebs, former director of CISA and founding partner of the Krebs Stamos Group, during the virtual Gartner IT Symposium/Xpo in October. 

The disconnect between IT and security, pokes holes in overall defenses. Part of that issue is the sole reliance on protection capabilities, even though the National Institute of Standards and Technology (NIST) recommends five functions of cybersecurity: identify, protect, detect, respond and recover. 

"We got super obsessed with keeping the bad guys out, but we forgot about the other pillars," said Byres.  

NIST's framework is universal, it's not just for software makers. In addition to companies knowing what assets are in their networks, they also need to know what's running in those devices. 

"And that's what the world and America really lacked. And SBOMs really give us that ability to identify and detect and respond," said Byres. On their own, they're useful — but SBOMs are really only meant to be building blocks, according to Byres, who has worked on the National Telecommunications and Information Administration (NTIA) and CISA committees for the SBOM movement. 

Part of the lingering issue of the Orion compromise was the fact that companies did not have the techniques to identify what was in their system — it took weeks to find the corrupted version of Orion. SBOMs, theoretically, could help reduce this time, though Finkelstein wants to see more concrete technological improvements, as opposed to concepts that aid investigations after an incident. 

However, a sense of liability for compromised software providers is not something security experts expect SBOMs to do. At their best, SBOMs establish negligence standards. 

"If it's plainly obvious that a company didn't do the basics to secure their products, then you may find some relief, but there's also a distinct difference here," said Krebs. "There will always be vulnerabilities from now till the end of eternity on the softwares that we use," which demands greater expectations of the build process.