Dive Brief:

• Codecov, an open source code testing provider, is rolling out a new Uploader to replace its Bash Uploader offering, which was the target of a supply chain attack in April, according to a blogpost from company CTO Eli Hooten last week. The beta release of the uploader uses NodeJS and is shipped as a static binary that is executable in Windows, Linux, Alpine Linux and macOS. 

• Codecov has been working for the past eight months to develop the new uploader, which does not rely on the bash script that it currently offers, Hooten said. As the company grew, the Bash Uploader became more difficult to properly maintain, he said. 

• The company's Bash Uploader depreciation plan begins this month and will fully sunset by February 2022.

Dive Insight:

Codecov's replacement uploader solves some of the biggest security concerns found in the Bash Uploader, according to the company. More than 29,000 customers, including some of the leading tech firms, currently use Codecov services.

The company conducted a post-mortem analysis and found flaws built into the system it needed to address. The attacker leveraged an HMAC key for a service account from Google Cloud Storage using an intermediate layer of the Codecov Self-Hosted Docker image, according to the analysis. Bad actors modified the Bash Uploader in Google Cloud Storage using the key. 

Codecov officials said the new uploader will have a compiled binary, which will make it tougher for an intermediary to modify the code. The new uploader includes a more secure, verifiable distribution compared to the Bash Uploader. 

Codecov's investigation indicated some larger problems with software distribution and signing, key management and with Docker layer attacks, according to the company. 

"While many tools exist to help with the secure distribution of keys and secrets, few solutions exist to properly track all the metadata associated with a secret," Codecov officials said in a post-mortem.

Security experts and developers are raising concerns that some of the core issues that led to the attack have not been adequately addressed with this update. 

"In our view, the main lesson to be learned from the Codecov breach is that you need to perform cryptographic verification of the tools that you download and execute as part of the CI/CD pipeline," Ronen Slavin, co-founder and CTO at Cycode. "And this is true, not just for Codecov, but for all the tools the organization uses."

If such a verification had been done prior to the attack, that would have mitigated the damage, as the replaced script would not have passed the verification test and would therefore not have been executed, Slavin said. A third-party client of the company was how the breach was originally detected. 

"So if we look at the changes Codecov is releasing, there is no major change in this specific aspect," he said. "With this new uploader, still it is the responsibility of the users to perform the verification and make sure no tampering was made." 

The Codecov update makes the distribution mechanism a bit more difficult, but questions remain about the underlying issues that allowed the code to be changed, Sandy Carielli, principal analyst at Forrester, said.

"So the question I think we all need the answer to is, regardless of how they change the uploader or change the mechanism, how are they protecting and alerting themselves on unauthorized access to the code," Carielli said.

Federal authorities have been investigating the Codecov breach since April. The original attack was launched January 31, but was not discovered until April 1. 

The impact of this incident was felt across the DevOps community and Codecov is also used by a number of Fortune 500 companies, including IBM. At the time of the incident, IBM said it was investigating the incident, but had not found any indications of compromise. 

Twilio, a firm that enables voice, text and other messaging through the development of APIs, announced in early May that a limited number of email addresses were exfiltrated by the Codecov attacker. 

To prevent such attacks in the future, Twilio said it has a third-party security team to evaluate new and existing vendors. The company also has an internal service called Deadshot that scans GitHub pull requests. 

Codecov is making additional changes to enhance its security posture, including: 

• Ensuring the validation process is more visible in documentation, especially after a customer alerted the company about the breach.
• Ensuring integrations, including the Circle CI Orb, Bitrise Step and Github Action were based on proper SHASUM validation. SHASUM is used to check the integrity of files. 
• Created in depth documentation for how to properly validate the Bash Uploader. 

Codecov is also enhancing its in-house staffing, with two positions around product and infrastructure security, the company said.