Lessons Learned from Recent DevOps Pipeline Breaches

Eran Orzel
May 05 · 5 min read
DevOps Pipeline Breaches

The new world of software development is inherently collaborative — development teams are geographically dispersed and require easily accessible and automated tools to ship products and features quicker with confidence. The new generation of development and cloud-based tools now enables previously siloed teams to share and work together in an easy way, but it also brings into the picture a new type of security risk. The pivot to CI/CD pipeline creates a new attack vector to organizations that can expose their networks, IT infrastructure and even source code to bad actors. An integrated and continuous security approach that scales with your new development effort is now more crucial than ever.

Securing CI/CD pipelines and the software release process in general relies on 3 core components:

  1. People
  2. Processes
  3. And Technology

Only by combining these three elements together we build a defense that ensures you stay vigilant.

 

  • People

The process of building, testing, deploying, and securing your products is still very much a human process. To fully secure your development environments, security awareness and training for the development teams is required. 

The Security and DevOps teams must now work together and establish collaborative practices. 

Getting the developers to take more responsibility for security and be part of the process of solving security issues is crucial for the security process and solutions to be effective. 

The impact of a misconfiguration mistake – It is all about the people.

In this example a source code leak occurred due to a common misconfiguration where default admin credentials were left in use. This incident emphasizes the importance and impact of the developers on the security posture of the CI/CD pipeline.  

Nissan Source code leaked online after Git Repo misconfiguration. Swiss-based software engineer Tillie Kottmann said Nissan North America’s misconfiguration of a Bitbucket Git server led to the online leakage of the automaker’s source code for its mobile applications and internal tools. Nissan was allegedly running a Bitbucket Git server with the default credentials of admin/admin, which the developer should have modified as part of the system setup. 

Security teams need to engage with the DevOps team and developers, build awareness to the tool’s security risks and make them part of the security process, controlling their own path. Such level of cooperation might take time to build, but we are starting to see some initial success in this area.

 

  • Process

DevOps processes and CI/CD pipelines work at lightning speed and change continuously; therefore, security must be built-in-by-design and move just as fast, if not faster. The security processes need to fit the test-fast, fail-fast mantra of the CI/CD process. Embedding security as part of the DevOps process at the right steps, will maximize its effect and will create the right cooperation level with the development teams that is needed to make it successful. 

If you are not there you can’t stop it – It’s an ongoing process.

Embedding the security controls as part of the DevOps process adds context to it and can help in identifying unusual activities not in line with the pipeline process. In this example – the external pull requests, new pipeline script and the crypto mining activity on the GitHub servers.

Attackers mine cryptocurrency on GitHub servers, abusing the GitHub Actions automation workflow tool to mine cryptocurrency on GitHub’s servers in an automated attack. This particular attack abuses GitHub’s own infrastructure and generates a pull request that executes the attacker’s code which instructs GitHub servers to retrieve and run a crypto miner and mine cryptocurrency on their servers.

Building the security enforcement as part of the DevOps’ process is essential in order for the security to be effective and not delay the development process. Security needs to be part of the core CI/CD process and provide actionable information that is derived from understanding the process and its outcomes. That way you are enabling, rather than blocking or delaying, the development activities, increasing the adoption and participation of the development team.

 

  • Technology

The tools and technologies used in the CI/CD pipeline are typically point solutions that come with limited security capabilities and do not interact with one another. This siloed approach does not provide the consolidated and unified way to view and analyze issues that is required in complex environments such as CI/CD pipelines. This means that potential threats and breaches might be overlooked, as your current security only relates to one area of your environment (usually scanning for images, vulnerabilities, secrets) and isn’t being viewed in the context of the entire pipeline process. 

You need to see and understand it to protect it – Holistic security approach is a must.

In the latest Dependency confusion supply chain attack, a researcher managed to breach over 35 major companies’ internal systems, including Microsoft, Apple, PayPal, Shopify, Netflix, Yelp, Tesla, and Uber.

The attackers uploaded malware to open-source repositories including PyPI, npm, and RubyGems, which then got distributed downstream automatically into the company’s internal applications. The researcher discovered a design flaw that when a dependency package used by an application exists in both a public open-source repository and your private build, the public package would get priority and be pulled instead — without needing any action from the developer.

To protect your CI/CD pipeline against such dependency confusion attacks, you need a security solution that is connected to all pipeline tools, understands the process and its dependencies and alerts on any change, such as internal package being pulled from an outside repo, or on any new repository added to your pipeline.

 

Summary

Protecting a complex and ever changing process such as the CI/CD pipeline requires a holistic security solution; one that takes into consideration the pipeline process, tools and operation scripts. Such a solution that is embedded within the pipeline process will be able to alert on vulnerabilities in real time, prevent attacks and auto-remediate human mistakes and misconfigurations before real damage is done.

As seen in the examples above, only by combining strong security measures over the DevOps pipeline with the right technology embedded into the DevOps process and involving the development teams in enforcing them, we can create a strong security posture for development environments.

While it may seem hard to achieve, there is a security solution that is built based on deep understanding of the DevOps process and tools, engages the developers, adds no additional work and can be set up quickly and seamlessly. 

Argon CI/CD security solution provides end-to-end security for DevOps pipelines, eliminating the risk from misconfigurations and vulnerabilities in your DevOps environments and from supply chain attacks. It seamlessly connects to your development environment and provides a unified view of the entire environment and enforces security best practices on all stages of the software delivery process, including real-time alerts and auto-remediation that minimize your exposure.

By Eran Orzel, Argon’s Chief Revenue Officer

Topics: DevOps Pipeline Breaches, DevOps, DEvSecOps, CICD, Supply Chain Attacks, CI/CD Pipeline Security, security best practices

Eran Orzel
May 05 · 5 min read

Related Articles

Securing your GitLab: Best Practices To Implement

What is GitLab GitLab is a free open-source service designed to manage and share code in a distributed version control…

Eylam Milner
Jul 14 · 4 min read

President Biden’s Executive Order Demands Cybersecurity for Software...

The SolarWinds Attack Was the Industry’s Wakeup Call The new wave of software supply chain attacks that targeted SolarWinds, Codecov,…

Eran Orzel
Jun 23 · 5 min read

The importance of having visibility over your pipeline’s plugins...

Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…

Eilon Elhadad
Jun 21 · 4 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner