Jun 21 · 4 min read
Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet another massive data breach, this time affecting the company Codecov. In this post we look at the benefits and risks of plugins in a CI/CD pipeline, and how organizations can use plugins without succumbing to their vulnerabilities.
On Jan 31, 2021 Codecov, a leading provider of QA testing and code coverage solutions was hacked. This breach was discovered on April 1, 2021 after a customer spotted it. It was related to Codecov’s bash uploader, which is a service that creates coverage reports from customer systems and sends it back to Codecov. From the customer side, it is executed using a curl command.
Codecov have since released Indicators of Compromise for their customers to check if their organization was affected by the breach. They’ve included the exact script that was modified, and a list of IP addresses that were confirmed or suspected to be involved in the breach. Three notable companies that have disclosed a breach stemming from the Codecov data breach are Twilio, HashiCorp, and Monday.com.
The FBI is involved in the investigation and this could be an indicator that this attack is linked to the larger SolarWinds attack that was discovered in December 2020.
CI/CD tools like Jenkins are heavily dependent on the use of plugins. These plugins are usually developed either by third-party services like Codecov, open-source communities, or even individual developers. There are plugins for everything, from storage to networking to security. Some advantages of CI/CD pipeline plugins are that they allow developers to quickly test changes to their code, and automate tasks like deployment. The capabilities of plugins include enhancing the build server experience, automatically executing integration tests, and automatically performing feature and design reviews. All of this results in more interactive code testing, realistic unit test coverage, and better and faster deployments.
However, these plugins can come with hidden vulnerabilities. The security guarantees for these plugins vary greatly – some strictly follow security best practices while some completely abandon them. The risks with CI/CD plugins are related to storing unencrypted passwords in environments, cross-site request forgery (CSRF), and avoiding permission checks. If an attacker can break into your Jenkins server, they could easily steal your credentials and impersonate you. Once they’re in, they could use this access to change the default environment variables and modify the default credentials for Jenkins users on the Jenkins server, allowing them to download and execute arbitrary code.
Let’s look at the potential attack vectors in a CI/CD pipeline according to the MITRE ATT&CK framework.
The attack starts out small with access to some seemingly inconsequential secret information that a developer forgot to remove or encrypt. This is made complicated by weak IAM policies that allow users to create new accounts and give those accounts risky privileges. Finally, access tokens are managed in a traditional way – unchanged for months or years, and exposed unencrypted. All of these combined make the perfect storm, usually resulting in a victim organization being taken by surprise when they find out how complex the attack was.
Plugins are unavoidable, but they should be used with caution. Organizations should restrict the access CI/CD plugins have to their CI/CD pipeline.
Apart from plugins, general CI/CD security best practices should be adhered to. This includes alerting on any attempt by any user to create a new user account or to escalate privileges. You also need behavioral monitoring that establishes a baseline behavior for every user and can alert whenever there is abnormal activity. This may be as routine as creating a new container instance, but it needs to be tracked.
Don’t let vulnerable plugins hold you back from leveraging a CI/CD pipeline. Remember that CI/CD plugins are not necessarily bad, but you need to be aware of the risks, especially when plugins that are not documented or secured well are being used.
The Argon solution can help you detect the second a plugin starts making your pipeline vulnerable, or when a user us using the plugin in a suspicious manner. You should not rely on your manual efforts, but leverage a state-of-the-art security tool like Argon that is purpose-built to secure your software supply chain.
What is GitLab GitLab is a free open-source service designed to manage and share code in a distributed version control…
The SolarWinds Attack Was the Industry’s Wakeup Call The new wave of software supply chain attacks that targeted SolarWinds, Codecov,…
Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…