The importance of having visibility over your pipeline’s plugins: The Codecov case

Eilon Elhadad
Jun 21 · 4 min read
Plugins Visibility

Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet another massive data breach, this time affecting the company Codecov. In this post we look at the benefits and risks of plugins in a CI/CD pipeline, and how organizations can use plugins without succumbing to their vulnerabilities.

The Codecov data breach

On Jan 31, 2021 Codecov, a leading provider of QA testing and code coverage solutions was hacked. This  breach was discovered on April 1, 2021 after a customer spotted it. It was related to Codecov’s bash uploader, which is a service that creates coverage reports from customer systems and sends it back to Codecov. From the customer side, it is executed using a curl command.

Codecov have since released Indicators of Compromise for their customers to check if their organization was affected by the breach. They’ve included the exact script that was modified, and a list of IP addresses that were confirmed or suspected to be involved in the breach. Three notable companies that have disclosed a breach stemming from the Codecov data breach are Twilio, HashiCorp, and Monday.com. 

The FBI is involved in the investigation and this could be an indicator that this attack is linked to the larger SolarWinds attack that was discovered in December 2020. 

Plugins are a double-edged sword – The benefits and risks

CI/CD tools like Jenkins are heavily dependent on the use of plugins. These plugins are usually developed either by third-party services like Codecov, open-source communities, or even individual developers. There are plugins for everything, from storage to networking to security. Some advantages of CI/CD pipeline plugins are that they allow developers to quickly test changes to their code, and automate tasks like deployment. The capabilities of plugins include enhancing the build server experience, automatically executing integration tests, and automatically performing feature and design reviews. All of this results in more interactive code testing, realistic unit test coverage, and better and faster deployments.

However, these plugins can come with hidden vulnerabilities. The security guarantees for these plugins vary greatly – some strictly follow security best practices while some completely abandon them. The risks with CI/CD plugins are related to storing unencrypted passwords in environments, cross-site request forgery (CSRF), and avoiding permission checks. If an attacker can break into your Jenkins server, they could easily steal your credentials and impersonate you. Once they’re in, they could use this access to change the default environment variables and modify the default credentials for Jenkins users on the Jenkins server, allowing them to download and execute arbitrary code. 

Attack vectors related to CI/CD plugins

Let’s look at the potential attack vectors in a CI/CD pipeline according to the MITRE ATT&CK framework.

  1. Initial Access: Adversary scans repositories looking for secret information stored in a repository or sent to/from a plugin. This gives them access to the CI/CD pipeline.
  2. Persistence: Uses stolen credentials to create a new account on the CI/CD service provider (Jenkins, GitLab, etc).
  3. Privilege Escalation: Uses a valid account to change access permissions.
  4. Defense Evasion: Creates a new VM or container instance to bypass firewall rules.
  5. Credential Access: Steals access token to a database or Git repository.
  6. Discovery: Locates target database or repository.
  7. Lateral Movement: Uses application access token to access database or repository.
  8. Collection: Mines information from the database or repository.
  9. Exfiltration: Exfiltrates to their own Git account or server, sometimes locking access to the database/repository.

The attack starts out small with access to some seemingly inconsequential secret information that a developer forgot to remove or encrypt. This is made complicated by weak IAM policies that allow users to create new accounts and give those accounts risky privileges. Finally, access tokens are managed in a traditional way – unchanged for months or years, and exposed unencrypted. All of these combined make the perfect storm, usually resulting in a victim organization being taken by surprise when they find out how complex the attack was.

The Solution – Pipeline Visibility to Plugins

Plugins are unavoidable, but they should be used with caution. Organizations should restrict the access CI/CD plugins have to their CI/CD pipeline.

  • Take an audit of all plugins you use today, and check the kind of access levels and privileges they have.
  • Analyze which repositories and databases they read data from, and if any of this data is transported elsewhere within the system or outside the system.
  • Assess the kind of activities they perform, especially activities that send or receive data to/from a third-party who developed the plugin. 

Apart from plugins, general CI/CD security best practices should be adhered to. This includes alerting on any attempt by any user to create a new user account or to escalate privileges. You also need behavioral monitoring that establishes a baseline behavior for every user and can alert whenever there is abnormal activity. This may be as routine as creating a new container instance, but it needs to be tracked.

Conclusion

Don’t let vulnerable plugins hold you back from leveraging a CI/CD pipeline. Remember that CI/CD plugins are not necessarily bad, but you need to be aware of the risks, especially when plugins that are not documented or secured well are being used.

The Argon solution can help you detect the second a plugin starts making your pipeline vulnerable, or when a user us using the plugin in a suspicious manner. You should not rely on your manual efforts, but leverage a state-of-the-art security tool like Argon that is purpose-built to secure your software supply chain. 

Eilon Elhadad
Jun 21 · 4 min read

Related Articles

Securing your GitLab: Best Practices To Implement

What is GitLab GitLab is a free open-source service designed to manage and share code in a distributed version control…

Eylam Milner
Jul 14 · 4 min read

President Biden’s Executive Order Demands Cybersecurity for Software...

The SolarWinds Attack Was the Industry’s Wakeup Call The new wave of software supply chain attacks that targeted SolarWinds, Codecov,…

Eran Orzel
Jun 23 · 5 min read

The importance of having visibility over your pipeline’s plugins...

Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…

Eilon Elhadad
Jun 21 · 4 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner