The SolarWinds compromise is causing businesses to reevaluate vendor trust. The answer, some experts say, is to altogether rethink how software is built.
"In my experience, working with hundreds of companies on security issues, I don't think I've seen a build process that is perfectly designed against this kind of level of adversary," said Alex Stamos, professor at Stanford University, and independent consultant for SolarWinds, during a webcast Feb. 4.
The hackers used custom malware designed for the SolarWinds build cycle to observe each step. The actors replaced a file in "something like 100 milliseconds," allowing them to download and delete their tracks without a trace, said Stamos. The sophistication of the intrusion — in the middle of a software build — will influence the security footprints made in software coding.
Software supply chain attacks are a top business security concern, but due to the scope of the SolarWinds compromise, which impacted up to 18,000 customers in the public and private sectors, security organizations and engineers are revisiting gaps the industry knew existed, but found difficult to avoid.
"The modern software and infrastructure stack is a heterogeneous mash-up of code, components, infrastructure, services, APIs and even humans that an organization cannot have 100% security assurance over," said Jimmy Mesta, head of security research at Fastly.
Using third-party software presents customers with "choosing how much is enough security to enforce into the build cycle and how to quantify risk," said Mesta. The dance between third-party reliance and acceptable security highlights the continued tensions between DevOps and up-and-coming DevSecOps.
More than three-quarters of enterprises stuck in the planning phase of DevSecOps adoption view the practice as a way to cut costs and move technology to market faster, as opposed to improving security, according to Security Compass' 2021 State of DevSecOps report. The report surveyed 250 executives and practitioners in risk and technology in the U.S. and U.K. More than half of companies with a majority of their applications infused with DevSecOps cite improved security and resilience as their top driver of adoption.
Where to draw the line of separation
It's fairly rare for companies to have air gaps between IT and DevOps, though there are security benefits in doing so.
"On paper, it sounds great to air-gap these teams, but in practice access control and identity is often more of a messy art than science," said Mesta. IT typically needs access to code repositories and developers across an organization for different access needs.
At SolarWinds, hackers jumped from the company's IT infrastructure to its build system. Stamos recommends companies isolate systems to narrow accessibility, ideally by using containers or instances in a cloud environment and just-in-time authentication upon configurations.
The configurations should be accessible to a small group of people, and this is preferable for systems "born just for building the production release," before being torn down, said Stamos. It's a difficult undertaking, especially because build systems typically "grow organically as the product grows."
It will change how engineers are able to make changes, he said.
At least half of professionals tasked with product development say reactive or proactive software security processes slow their time to market — the view concerns professionals in risk more than those in IT, according to Security Compass.
Throughout the build cycle, companies should have stamps of forensic artifacts logged in something like a Security Information and Event Management (SIEM), which is atypical to do, according to Stamos. If an attacker tries to erase their attacks, the logged artifacts could guide the build back to before the deviation.
Security teams can run into issues with applying forensic stamps to third parties. "Strong chains of trust throughout the build cycle are still an uncommon occurrence in most environments me and my teams have interacted with," said Mesta.
Security teams need to cultivate a sense of accountability — backed by technology — for each portion of shipped code. "A really smart thing to do is to have them pick out a specific function in a DLL, a specific function in a ship executable, and then to work backwards," said Stamos.
Engineers tied to accountability will identify who wrote the code and whether it matches the intended object code. Companies can also use accountability to enforce encryption or hashing to provide context to each stage of the build. "This traverses all the way back to the developer laptop and up the chain to a SIEM or other logging mechanism," said Mesta.
Eventually security teams and engineers will widen their focus beyond vulnerabilities, and onto spaces in the attack surface attackers find attractive. Breaking down the build cycle beyond coding manipulations will make it so less vulnerabilities can be made.
For SolarWinds, having just-in-time access for multiple build pipelines and environments is reducing consistent compromise, said SolarWinds CEO Sudhakar Ramakrishna, during the webcast. "Those things are in process."
