How to Protect Your Pipeline against Supply Chain Attacks: the Codecov Breach

Eran Orzel
Apr 16 · 4 min read
protect supply chain attacks

A new “double” software supply chain attack that compromised Codecov Bash Uploader was revealed to have gone undetected since January and exposed sensitive secrets like tokens, keys and credentials from various organizations around the world.

As published on SECURITYWEEK on April 15, the attack occurred four months ago but was only discovered by a Codecov customer on April 1, 2021, the company said in a note acknowledging the severity of the breach. According to the company, the breach allowed the attackers to export information stored in its users’ continuous integration (CI) pipeline environments, which was then sent to a third-party server outside of Codecov’s infrastructure”.

The following breach steps, we have managed to learn so far:

  • Attacker took advantage of a misconfiguration in the docker image creation process.
  • The attacker was than able to extract credentials to GCP, and the Google cloud storage access was gained.
  • Then the attacker altered a bash-uploader script that run as part of the CI of CodeCov customers in different variations (github actions, circleCI orb and others).
  • The new process then sent out credential, tokens, keys, and sensitive data from the context of the CI run.
  • Every customer using it as part of its CI, was compromised.
  • The attack was detected by a customer comparing the hash of the formal bash-uploader script and the one downloaded.

The breach allowed a malicious third-party to alter a version of the bash uploader script to potentially export information subject to continuous integration (CI) to a third-party server, which according to Codecov, could potentially affect:

  • Any credentials, tokens, or keys passing through their CI runner that would be accessible when the Bash Uploader script was executed.
  • Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
  • The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.

 

Why CI/CD Pipeline security posture is important?

In response, organizations need to take proactive action to secure their supply chain from such attacks. This requires a conscious approach to CI/CD security posture that takes into account the sophisticated nature of today’s attacks.

Organizations need to watch their configuration files closely. Whenever any change is made to a config file, this change needs to be checked for any possible privilege escalation, or suspicious access. Config files are central to the operations of a software supply chain and should not be overlooked. Any suspicion should trigger notifications and alerts to the appropriate stakeholders, and stop certain tasks until it’s verified to be safe. This automation, being immediate, can act as a strong defense against attacks, as it’s significantly faster than manual human intervention.

 

3 Actions to Secure from Software Supply Chain Attack;

Here are 3 actionable things you can do to secure your software supply chain starting tomorrow:

  1. Get Visibility: You need to know what your CI/CD pipeline consists of, otherwise you can’t protect it. You should gain wide and deep visibility into every phase of your CI/CD pipeline. This requires a solution that is purpose-built to understand how a CI/CD pipeline works, and the different components and people involved at each stage. Visibility needs to be complemented with alerts and notifications for it to be useful and actionable.
  2. Manage your CI/CD posture: Your CI/CD security posture is the sum of the design decisions you make related to security. It includes how you handle changes in configuration, or open source code, or requests originating from outside your network. The first step is to assess your posture today, and then you can consider what your ideal security posture should be. You may need to make changes to your CI/CD process, tooling, and team culture accordingly. Your security posture is key to guarding you against any attacks before they happen.
  3. Ensure CI/CD integrity: As discussed above, any change that is not initiated or allowed by your organization compromises the integrity of the system and should not be allowed. Software code is the most likely target for compromising integrity. The key is to have integrity checks running regularly and at various parts of the supply chain.

 

Running CI/CD pipelines without strong security posture open your organization to severe cyber risks

The last thing any organization wants is to be in the news for the wrong reasons – a security attack or data breach. This threatens to break customer trust, cost businesses real revenue, and continues to have lingering damage for months or even years. The better alternative is to determine and implement actions to better secure your CI/CD pipeline, which will help you keep supply chain attacks (and the long-lasting damage they incur) far away from your organization.

Argon can help you gain the visibility and control you need over your CI/CD pipeline to mitigate such supply chain risks. Argon CI/CD pipeline security will help you:

  • Detect potential sensitive data leaks through the CI/CD pipeline, prevent publishing of any environment variables such as credentials, tokens, or keys and alert on any such exposure
  • Map out external scripts used in your CI/CD pipeline and analyze them according to DevOps and security best practices, alerting on any vulnerability or risk
  • Enforce security policies to control what runs in your pipeline, making sure your best practices policies are intact and preventing any violations by external entities

 

Eran Orzel, Argon’s Chief Revenue Officer

Eran Orzel
Apr 16 · 4 min read

Related Articles

21 Top DevSecOps Tools

What is DevSecOps? DevOps is now the default approach to agile software development and deployment in most tech companies. With…

Nurit Bielorai
Oct 11 · 9 min read

8 Fundamental Steps to Secure Cloud Data

The COVID-19 pandemic forced the world to rethink not only their lives but also their business operations. There was a…

Eylam Milner
Oct 06 · 10 min read

5 Common Risks for Supply Chain Cyber Attacks and What to Do About The...

The year 2020, despite the coronavirus pandemic, was an opportunity for hackers to create major upheaval. As the world dealt…

Eran Orzel
Sep 19 · 7 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner