Mar 16 · 3 min read
The notorious SolarWinds hack in December has brought to prominence the importance of securing the software supply chain. SolarWinds uses TeamCity a CI/CD solution from the company JetBrains to manage its software supply chain. There is suspicion that TeamCity could have been the entry point for the attack.
NYTimes ran an article suggesting that Russian hackers likely used TeamCity to gain access to SolarWinds’ system. They cite the fact that JetBrains is located in the Czech Republic, and was founded by three Russian engineers as a clue to this. The article alleges that the purpose was to eventually attack various departments of the US government, many of which are SolarWinds customers.
JetBrains denies being the cause for the attack, noting that along with TeamCity, SolarWinds uses many other tools as well. JetBrains clarifies that SolarWinds’ vulnerability of their CI/CD pipeline could be a cause, and JetBrains would not be responsible for any misconfiguration. Further, SolarWinds itself has not confirmed that TeamCity is the origin of the attack.
The investigation into the hack is ongoing and will take many months to be completed. We’ll have to wait for the investigation report to know what really happened. Even with the investigation complete the exact source and extent of the attack will not likely be known. However, knowing the essential facts behind it will help organizations be better prepared against such attacks in future.
SolarWinds is now taking steps to respond to this situation by securing their systems. They will be digitally signing every build to let customers be absolutely sure which code comes from SolarWinds.
The fact is, it was SolarWinds this time, but it could well have been any other organization with these similar data breach vulnerabilities. This incident was found out and exposed only a year after it actually began. There are many such hacks that are still unknown and unreported.
CrowdStrike, a security company hired to help SolarWinds with the investigation into this hack reports that “StellarParticle developers (the hackers) invested a lot of effort to ensure the code was properly inserted and remained undetected, and prioritized operational security to avoid revealing their presence in the build environment to SolarWinds developers.”
Any organization that uses a CI/CD pipeline to manage their software delivery needs to take note of this incident. It is telling that SolarWinds is still unable to spot the source of the attack, or the attacker, or the extent of the attack. Attacks are hard to spot in modern software supply chains. This is because the supply chain has become very complex with numerous tools being used for unique purposes. The solution is not to use fewer tools, that would be restrictive and will stifle innovation and slow down operations. Instead, the solution is to have stronger security measures at every stage of the pipeline and to have deeper monitoring at each step.
Every action taken by a user or a machine needs to be defined. Every piece of code or configuration needs to be verified to be sure it is from a trusted source. But this means potentially tens of thousands of checks per day. It is NOT possible using manual human review, or even traditional security monitoring tools. It takes tools that can automatically discover misconfigurations, new identities, and components as they are added to the supply chain. It should enable security that is not static, but dynamic, and policy-based.
This leads to integrity at every step of the pipeline, and prevention of risks before they occur. There should be strong defaults to prevent suspicious code from getting deployed into production.
Security testing and penetration testing are essential practices that often get overlooked in most DevOps teams. Investing in ongoing security testing will pay off in the long run. This security testing should be done from the start of the supply chain, and not only in production. One of the key strengths of the hackers of SolarWinds is their ability to slip under the radar of monitoring tools. They did this for over a year. When done properly security testing should check the effectiveness of alerts.
In conclusion, we will never know the full extent of the SolarWinds vulnerability attack. But what we know is enough to alert us of the desperate need to better secure software supply chains. It takes deep awareness of the many tools being used in the CI/CD pipeline, enforcing strong security measures at every step, and setting up intelligent security monitoring that cannot be fooled by hackers.
What is GitLab GitLab is a free open-source service designed to manage and share code in a distributed version control…
The SolarWinds Attack Was the Industry’s Wakeup Call The new wave of software supply chain attacks that targeted SolarWinds, Codecov,…
Hardly a week goes by these days without hearing about a new supply chain attack. A recent headline featured yet…