According to Comparitech, the number of software supply chain attacks every year has been steadily climbing from 2016 with 12 reported incidents to 2022 with 43 reported incidents resulting in exploited software. The three highest profile attacks happened within the last two years alone. A software supply chain attack negatively impacts the build and delivery of software to a hosting solution. Before the rise of CI/CD pipelines, developers had to run a series of scripts and manual tasks to successfully distribute their software to the world. In modern software development, we rely a lot on repeatable automated processes to do the bulk of that work. There are many benefits to the modernized process, however I believe that in the coming years we will need to put more and more focus on securing these delivery routes.
To demonstrate how vulnerable these CI/CD pipelines can be and the damage that can be inflicted as the result of an attack, I’ll be walking through the first part of a puzzle that I worked on during the holiday season of 2017 called Jeeves, created by mrb3n on HackTheBox. This machine has been retired for several years now, and public walkthroughs and videos have been released.
The Demo:
To speed through the first bit of this. I fired up the machine in the HTB console, connected to the VPN and fired off my first couple commands at it to enumerate its services. I would typically DirBust both the IIS and Jetty web servers but I’m pretty sure the Jetty server is the point of interest here.
Looks like DirBuster found a 200 response to the /askjeeves/ directory. When we visit that location, we are greeted with a Jenkins landing page.
CI tools, like Jenkins, use build agents that run these script-like jobs one by one on the underlying operating system. The idea is that these commands prepare the underlying operating system to build or compile applications so that they may be distributed to their hosting solutions or other machines in the world.
My plan is to serve up netcat on my attacking machine with Apache, create a job in Jenkins to download the executable from my machine, and then run it to get a reverse shell back to my attacking machine. At the end, I should have command line access to the underlying operating system and ultimately, depending on which stage of the pipeline, the backbone of the software delivery system.
I did test the download to make sure that my server was working and my executable was in the correct location. In the Jenkins console, I created a new freestyle job called “Test”, and added a build step to “Execute Windows batch command”.
I click apply and I start up a listener on my attacking machine as such.
After running the build, I go to the console output of my new job and it does seem to run my commands as intended.
It looks like it’s hung up or it’s waiting for something to happen. This is because my Netcat listener is still talking to it, preventing it from finishing the job. As long as I keep my shell running and the job isn’t canceled by someone else, the job will continue. As the attacker, we wouldn’t want to give away our attacking machine and we would want to cleanse this log output so as to not alert our victim to the breach, but that is a different blog post.
I essentially have control over this “kohsuke” user, but more importantly, I am now free to run whatever commands I want to influence how this hypothetical application is built and distributed. We should be able to run just about any command we would want to tamper with this
The Retrospective:
The Jenkins tool we took advantage of in this example is not special. This can be done with many different CI tools. While this is inherently dangerous, there are many reasons why your DevOps and Software Engineering teams might want to interact with a pipeline in this way. Perhaps they need to troubleshoot a stage of the process, or maybe they need to analyze why a build takes an increasing amount of time. In CircleCI, it’s actually a feature that you are able to remotely connect via SSH into a running pipeline to oversee processes. Thankfully, it is only available using SSH keys to authenticate users that also have access to the Git repository.
While the method used to gain access is not particularly difficult to pull off, I think it does enough to let the imagination run wild on the other kinds of attacks we could use depending on the build process and project. Perhaps instead of running netcat, I want to run a script that adds a disreputable python repository that includes out of date or vulnerable dependencies. I could also deploy my own unauthorized images inside the victims hosting solution. Environment variables are often used during the build process, these sensitive variables could be changed to influence how the application is built. It could be that this hypothetical application is a remote access tool used in IT environments and, instead of reporting back to the authorized set of control servers, now only talks to the attacker’s servers. The possibilities are near limitless.
So what can be done about this?
- Never give public access to your CI tools and be restrictive as to who can view them.
- Audit any and all external/open source scripts that run in your pipelines.
- Audit the sources of your dependencies and build in checks to verify integrity during build and deploy processes on a routine basis. Implementing security scanning on dependencies can also help with this.
- Have a plan in place to swap all sensitive environment variables in the event of a breach.
Comparitech has some pretty comprehensive charts and diagrams to show the increasing threat to software supply chains as well as some analysis of the Log4j, Solarwinds, Codecov, and Kaseya breaches.
.jpg)
