Understanding consequences of Build Stage Code manipulation and how to avoid

Eilon Elhadad
Mar 16 · 6 min read
source code manipulation

Your devs have put in the hard work, you’ve committed the first lines of securely written code and you’re ready to build your first artifact. Good work! However, it’s at this point that your precious code lines are often at their most vulnerable. The hand-over of raw code to the build stage comes with new security fallibilities. 

The build stage of your CI/CD pipeline is the point where you generally lose visibility of your code, which makes identifying changes to it – difficult. Moreover, compromises at this level are concerning because they are frequently and quickly signed off as legitimate code, making their way into testing and deployment stages – and most horrifyingly – to your Users. 

Why the Build Stage is so unique:

  • Loss of code visibility creates a “blind-spot” in your pipeline.
  • There’s no feedback loop where you might ordinarily find code discrepancies – like you would in GitHub.
  • It’s naturally a highly involved and technical point of your pipeline where many actions take place on your code – making it easy for alterations to go unnoticed.

 

What might an attack on my Build Environment look like?

Without getting too technical, an attack on your build environment is an attempt to inject malicious code into your environment with the end-goal of it reaching your users. 

Attackers who access your build environment may try to exploit vulnerabilities in this “blacked-out” stage, for example, by: 

  • Stealing sensitive information (e.g., access keys, passwords, encryption)
  • Injecting malicious code or malware into your project without your knowledge.
  • Using a compromised development device (e.g., a server) as a proxy to further attack your build and deployment pipeline.

Learning from recent code manipulation scenarios 

Unless you’ve been under a rock, you’ve most likely heard of the I.T. behemoth SolarWinds hack, which is a terrifying example of code manipulation in the build environment of your CI/CD pipeline. 

What we know so far: 

  • A government-linked Russian group dubbed ‘Cozy Bear’ infiltrated SolarWinds Orion build environment.
  • Popular build product TeamCity (a JetBrains product), used by SolarWinds, may have provided the backdoor for the attack. 
  • Attackers injected undetectable malware into Orion’s code, affecting 18,000 SolarWinds clients.
  • Attackers patiently learned naming conventions and other highly nuanced details to go undetected within the environment. 
  • Victims included Microsoft, U.S. Homeland Security, State, Commerce, and Treasury Departments, and many more. 
  • Investigators estimated the entry point to be sometime in 2019.
  • SolarWinds Orion was used as a Trojan-horse to enter the systems of high-profile, sensitive agencies and companies, with the view to potentially do far more damage to those companies. 

 

 The New York Times calls the SolarWinds breach a potential “entry point for a huge U.S. hacking.” 

 

“By compromising TeamCity, or exploiting gaps in how customers use the tool, cybersecurity experts say the Russian hackers could have inconspicuously planted back doors in an untold number of JetBrains’ clients.” 

At the time of writing, the full investigation is ongoing, so it may be a while till we are entirely in the know of what went down. What’s truly disturbing about the SolarWinds hack is how the attackers played the long-game: learning naming conventions and writing code that looked identical to SolarWinds’ own. 

What are the consequences of a code injection? 

Who are the losers, and what damages can be expected in a build time source code manipulation? 

The impact of a single code manipulation depends on the size and nature of the infiltrated product. With SolarWinds, the full damage is being unraveled day by day because of their users’ breadth and authority. Affected parties can break out the consequences into the following categories: 

Consequences to your Product: 

  • Distrust of product security 
  • Suspicion of future product roll-outs 
  • De-valuing of your product 
  • Pushing clients to consider competitor products 

Consequences to your Company:

  • Distrust of company security practices overall 
  • Identification of weaknesses throughout CI/CD
  • Dent to reputation – implementation of damage control  

 Consequences to your Customers & customers’ Customers:

  • Attackers may now have accessed their systems/I.P./code
  • Inconvenience to systems 
  • Potential downtime and financial implications 
  • Loss of trust in product security 
  • Exposure of private company data to unknown recipients
  • Additional security needed to mitigate further damages 
  • Costly response strategy – see Microsoft response below. 

Risks to your pipeline are risks to your customers & their customers. The bottom line is that attackers are continually looking for opportunities to exploit weaknesses – don’t let that happen. Flaws found in your pipeline are potential weaknesses for your customers, leading to your company’s long-term reputational damage. 

What next? How do you recover from an attack? 

The first 24 hours is critical to regaining control of ill-acquired systems, files, or servers. Much like when a person is reported missing, the chances of a swift recovery operation hinge on speed, deft decision-making, and a collaborative approach. 

Software giant Microsoft, which was also a user of SolarWinds Orion, played an integral part in setting the bar for response and helped to put out the fire. Microsoft’s response to the consequent file manipulation was one of incredible scope, scale, and speed. For an excellent reason – they are Microsoft. They focused on four main defense mechanisms, all executed within three days of the crisis (Dec 13 – 16 2020), to undo the work of the attackers:

  1. Removal of the digital certificates that the malware used. Through this single action, Microsoft could tell all M.S. systems not to accept these certificates, effectively blocking those files from being used. 
  2. Microsoft updated Windows Defender; the anti-malware software would signal an alert if it encountered the malware. 
  3. Microsoft then chose to ‘sinkhole’ one of their breached domains, a severe and technical tactic serving to take access and control away from the attackers completely. 
  4. Finally, MS set Windows Defender to “quarantine” when the files were detected – an action that would crash some systems but would kill the malware.

These four reactions enabled Microsoft to reassure customers, negate further penetration, and disarm the attackers. 

Application security does not mitigate software pipeline vulnerability 

You may securely write your code. You may have excellent security processes in your coding stage. Your company may have ‘shifted-left’ your security approach – this is all excellent – for protecting your code. 

What about your pipeline? Can you hand-on-heart say your SDLC is covered from risk? Can you halt incoming threats in time to mitigate damages? 

In any breach of the build environment, the damage can travel far and wide throughout your supply chain. The end-users having to upgrade their product version doesn’t sound so bad – but the broader harm done is often unknown.  

Taking security one step further after a source code manipulation or tinkering in your build environments is advisable. Taking precautions to secure your entire pipeline from code commitment to deployment is now critical. 

DevOps is complex – with many moving pieces and tools – each offering their own security protocols. For this reason people who are newer to DevOps might assume that vulnerabilities are minimized or controlled at each stage, however the truth is that protection of your entire software delivery process is the only way to ensure your entire pipeline is secure and healthy. 

How Argon Security Solution can help you avoid catastrophe?

We know that attacks at any stage of your pipeline can be stressful and impact your employees, customers, and culture on a deep level. Talk to us about our 3 pipeline security engines, designed to level-up your abilities to keep your integrity and protect your entire DevOps Pipeline.

Eilon Elhadad
Mar 16 · 6 min read

Related Articles

Securing your GitLab: Best Practices To Implement

What is GitLab GitLab is a free open-source service designed to manage and share code in a distributed version control…

Eylam Milner
Jul 14 · 4 min read

President Biden’s Executive Order Demands Cybersecurity for Software...

The SolarWinds Attack Was the Industry’s Wakeup Call The new wave of software supply chain attacks that targeted SolarWinds, Codecov,…

Eran Orzel
Jun 23 · 5 min read

The importance of having visibility over your pipeline’s plugins...

Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…

Eilon Elhadad
Jun 21 · 4 min read

End-to-End CI/CD Security Platform

open source vulnerability scanner