5 Urgent things to Secure your Software Development CI/CD Pipeline now

Eylam Milner
Feb 25 · 9 min read
secure development environment
  • Today’s software development life cycle process (SDLC) connects the development and operation activities and teams through automation in building, testing, and deployment of applications and software updates. This process, often referred to as the CI/CD pipeline, uses a series of solutions including cloud-hosted services and open source tools, all intertwined and connected to each other. As such, the software delivery pipeline often contains all of the data, proprietary IP, trade secrets, code, and credentials invested in building and maintaining your software products. 
  • The use of sophisticated attack methods and malware can be extremely detrimental to your application, taking advantage of holes in your defense line. CI/CD pipeline attacks can lead to leaked repositories and malicious code injections, as well as access to API keys and access tokens. All of these ultimately enable attackers to steal personal and proprietary data, access production environments, and leave you and your customers exposed. 
  • The SolarWinds attack demonstrates how a variety of related factors were leveraged. It is an exemplary instance of how dangerous a CI/CD breach can be. As they reported themselves, the vulnerability “was introduced as a result of a compromise of the Orion software build system,” and in November 2020, a security researcher posted to Twitter that he had discovered a public repository leaking file transfer protocol (FTP) credentials.  Simultaneously, while the attack was only recently exposed, it had been in process for months prior without being suspected by anyone. And SolarWinds is only one of many recent examples.
  • So, what can be done about it and how can you protect your assets to avoid such a damaging outcome? We collected the top 5 steps you can start working on today to protect your CI/CD pipeline, your code and continue to develop in your agile workflow.

Here’s the 5 Steps list to secure development environment:

 

Audit and monitor your ecosystem and keep it healthy

    • More often than not, the setup of your ecosystem is complex, including loads of repositories, your build system set up, communication and tracking tools and more. All of these are interconnected, and when one is exposed, so are the rest. This means that one hole in the system anywhere can lead to a vulnerability of the entire process. 
  • Maintain visibility

    • Unfortunately, this complexity in and of itself can be the very vulnerability that enables an attack since it’s the not knowing that often leads to an attack. Thanks to this problem, not only can pipelines be attacked easily and provide huge benefits to bad actors, but to top it off, their exploits often go completely unnoticed! According to Gartner, IT security is often unaware of the security posture and risks of these CI/CD systems and aren’t hard-pressed to secure them. In fact, statistics show that most organizations aren’t protected against CI/CD pipeline attacks at all.
    • To make sure you’re on top of it all, study and map out your pipeline environments and tools and understand what kind of information each part processes and stores. Take advantage of available features to help you with this. GitHub Enterprise offers a management console for example. 
    • Once you’ve got this covered, make sure all of your tools and applications are up-to-date and know how to prioritize issues as they arise.
  • Stay updated

    • Oftentimes, instabilities and vulnerabilities are swiftly identified by app owners and handled through hotfixes, patches and version updates. If you miss an important update, you’re putting your entire system at risk. 
    • While it’s true you wouldn’t want to update Jenkins every day, if you look at their scores on the CVE site, you can quickly see that in addition to the regular cadences that you plan, there were a couple of critical updates in 2016 and 2018 that you would have wanted to implement immediately. Along the same lines, you can also consult with CVE Products list and other pages on that site to help you verify security issues and speed of action when choosing new tools.
  • Implement continuous access control

    • Since you can’t actually watch every component every second of the day, finding bad actors, or identifying suspicious activity, can only be accomplished by monitoring access to the repository and your pipeline with properly configured access policies and automated monitoring. 
    • For end-to-end access control, follow these essential best practices: 
  • Know who is using your systems and why

With a global workplace and endless numbers of useful tools, it might be too easy to forget that it’s not only your developers and your teams who need access, but also contractors and other external parties contributing to your work, and APIs from other systems assisting with automation. Be pedantic in the access you give, and limit it based on the nature of work for each individual and group. 

Know what is using your systems and why

The authentication granted between your different systems, to enable them to operate smoothly, is referred to as trust relationships. Inevitably though, there are many threats and vulnerabilities in essential components or interactions of the continuous deployment pipeline. While some of the innate threats can be attributed to vulnerabilities in the tools themselves, most issues are related to either improper configuration of access or unencrypted transfer of information. 

The deceiving notion that you have created “trust relationships”, combined with the large amounts of sensitive inadvertently-stored data can lead to disaster. Make sure you know which application is “talking” with which applications and when, what kind of data they’re sharing, and so forth. 

  • Enable access based on the principle of least privilege

  • Create user roles, permissions and groups based on the principle of least privilege, enabling each user and machine to access only what they absolutely need, and enabling them only the breadth of access they need for their responsibilities. Leverage fine-grained user access control features in the tools you use. AWS tools offer robust security and deployment features to help you implement policies and track all of your moving objects.  Configuring exactly who can perform which action on which code project minimizes the risk of a leak and the chance of human error. See more here [link to another argon piece about SCM security].
  • Additionally, make sure you implement multi-factor authentication (MFA). GitHub, for example, lets you mandate MFA for users in your organization. 
  • Ensure that all of your repositories and code stores are private

Misconfigurations can leave doors wide open. Make sure your Git is secure, keep yours. Git files private and monitors your Kubernetes configurations as well. You should also implement data policies, and make sure that your secure data is properly encrypted. 

Audit and monitor access

  • Similar to the ongoing administration of the ecosystem itself, you should also monitor your users continuously. Look for and regularly remove stale users and stale applications from the system. Audit and monitor access continuously and be extremely sensitive to any anomalous behavior. Pay attention, and leverage features in the tools you use, such as GitLab auditing (for events and for users both). 

Manage your secrets

Data and code leaks are the two biggest dangers in your pipeline, and so these should be your focus when prioritizing. One study recently found that “hundreds of thousands of API and cryptographic keys leaked at a rate of thousands per day,” in GitHub alone. Start protecting against these leaks by securing all of your credentials and secrets. After all, secrets should be kept secret. And as recently reported by SANS in their 2019 Cloud Security Survey, “In 2019, almost 50% of all breaches came from the misuse of credentials, frequently found in code.”

Implement processes that spell out where and how credentials for access to cloud resources and toolchains should be stored. Otherwise, developers are likely to either inadvertently hard code credentials for simplicity’s sake or use environment variables. While it’s true that the latter is certainly preferable over the formal, neither is secure enough. Attackers can dump variables to get all the information they need to exploit other resources. 

Hashicorp, for example, helps you manage all of your credentials and secrets across the entire organization seamlessly, simplifying the use of those credentials by your teams. Similarly, GitLab offers native automated secrets detection for your repositories and Azure offers its own vault for proper secrets management.

Establish development processes and ensure everyone is onboard

A lot of the advice we’ve offered up until now relies on visibility, proper tools to support you, and awareness. Part of the awareness component is syncing between individuals and groups, and making sure everyone is on the same page. 

As part of this, it’s important to outline your development processes in a way that is adaptable to your specific ecosystem and organization, making sure you identify who owns which roles and responsibilities, and governing and updating those processes on an ongoing basis as things change. This can all feel hard to handle, but when everyone is included, it can be much easier to delegate, administer, monitor and keep it all together.

It’s particularly important that you involve the different DevOps and DevSecOps teams at your organization to ensure that everyone is aligned. You should also take into consideration all of the other related roles that are involved such as your developers (of course), AppSec teams, Quality Assurance teams SOC teams, and the like. Make sure everyone knows who’s responsible for what. 

Automate your best practices

We’ve already talked about the top four best practices: auditing and monitoring your ecosystem, continuously controlling access, managing your secrets, and implementing flexible development processes. 

These steps, however, won’t keep your assets airtight. It’s obvious that you can’t manually map out every tool, framework, and utility being used, and no team can manually check every release or enforce best-practice configurations for every single new developer or code project. Immediate visibility into your development environments is crucial however. 

Automating security can help you cover more of these gaps, and detect the issues that might slip by otherwise due to the human factor. As we’ve discussed, most vulnerabilities are directly related to improper access control and unencrypted data, and so you should place particular emphasis on these issues when planning automation to tighten up and lock your open doors. Argon is an excellent tool to support you in this quest, helping you automate across the ecosystem as much as you can to help you keep tabs on it all. 

Bottom line

While it might seem like an impossible feat, there really are some fast and reliable ways you can increase your security posture overnight. With 5 simple steps, you’ll be amazed at just how secure your development environment will become: map out your entire ecosystem and know how everything is connected, implement, audit and monitor policies, secure source-code and credentials, establish organizational processes, and last but not least – automate processes in order to swiftly identify and mitigate risks.

How Argon Security can help you avoid catastrophe?

We know that attacks at any stage of your CI CD pipeline can be stressful and impact your employees, customers, and culture on a deep level. Talk to us about our 3 pipeline security engines, designed to level-up your abilities to keep your Software integrity and protect your entire DevOps Pipeline.

Eylam Milner
Feb 25 · 9 min read

Related Articles

Dependency Confusion: An open door to your on-prem

Modern software development and delivery is not done in a silo, on a single-developer machine. It is written in collaboration…

Eilon Elhadad
Sep 09 · 4 min read

The Essential Guide to Dependency Graphs

When building legacy or cloud-native applications, codebases can quickly become entangled. This complexity becomes an issue when your teams add…

Eylam Milner
Aug 29 · 7 min read

The importance of least privilege access in your CI/CD pipeline

There are many aspects to securing a software supply chain, and these keep changing and growing as technology advances. One…

Eylam Milner
Aug 23 · 5 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner
Join our CTO in a thought-provoking discussion on software supply chain attacks with Cyberint
Join our CTO in a thought-provoking discussion on supply chain attacks