In part 2 of our AWS hacking series, we’re taking a look at the open-source automation server Jenkins.
Jenkins helps automate those parts of software development that are related to building, testing and deploying, and plays a major role in ensuring continuous integration and delivery.
It is also used as a management tool to help develop pipelines for building, testing and deploying Cloud infrastructure; it’s a popular choice for many development teams who work within the AWS cloud.
In this article, we’ll review potential security issues and establish what you can do to secure your AWS accounts against any unwanted access through Jenkins.
So, without further a due, let’s get strapped in and start the ignition.
Hack and Slash
Jenkins requires a lot of privileges to deploy and destroy infrastructure and resources in the Cloud, making it a great entry point for hackers!
That’s why most companies usually restrict access by hiding it behind a Virtual Private Network (VPN), making it available only for their employees.
It does happen, however, that a firm’s Jenkins server is publicly accessible and that anonymous visitors use it through different access levels – the easiest being read-only permissions.
Once again donning my white hat, let me first shed some light on this particular access type.
Burned After Reading
You’d be surprised what damage can be done with “just” a read-only permission. Access to AWS credentials, data and secret information… these and more can be revealed to you – if you know where to find them.
Let’s look at three specific entry points that can grant potential access to some juicy details.
Within Jenkins, Workspace gives you an overview and easy access to your project structure and project files.
Here, an unauthorized user can easily find the source code of a given application, which they can then use in conjunction with the read-only permission to preview all project files.
Besides the source code, you can also find configuration files, data dumps, CSV files containing PII or even database dumps.
That’s all fine and dandy, but here’s the real kicker: you can find plain text AWS credentials in config files or even hardcoded in the application.
Once a hacker finds this, they can do as much as IAM Policies attached this user allow him to.
By the way: another entry point is the Step Workspace. If you work with pipelines, this is where the project structure and files in your pipelines are accessible.
These are defined for each job individually and are available only during the job run. With the read-only authorization, you can find AWS access keys, logins, passwords and other sensitive information here. The Environment Variables page is available for most already completed jobs.
The console log contains the results and output of every command run for each job. You can find a lot of interesting information here, like logs of credentials, endpoints and e-mail addresses.
Example jobs which I found containing sensitive data:
- Creating new AWS IAM Users (and printing access keys);
- Generating temporary AWS access keys using STS (and printing them);
- Testing the application on different access levels (and printing login and password used).
Username/Password allows you, for example, to log into the stage environment where the test was conducted. Now, the stage environment can be connected to a production database. And when that’s the case, then we’re in it for the big time!
Jobs for Everyone!
Another access level commonly found on publicly exposed Jenkins machines allows any anonymous user to create a Jenkins Job.
A Job could be a script that is then run by the Jenkins machine. This means you can create any script you like, and Jenkins will run it for you.
Do you want to extract temporary access keys from the Jenkins EC2 metadata? No problem:
Do you want access keys stored in Jenkins Credentials? No problem, print them on the screen or send in an email.
How about Jenkins using IAM User access keys instead of a role? Great:
If that’s not enough, there’s another powerful permission level – Manage Jenkins.
This access type allows you to open “Manage Jenkins” page with all configuration details.
With this permission level, you can preview (and modify!) the current configuration of all extensions. It can give you email addresses, private keys, credentials, git repository addresses and more.
Manage Jenkins also gives you access to a script console. The console may be used to run any scripts without running a job. Attack scenarios are similar to the previous access level with job creation. But the console is also useful to decrypt Jenkins Credentials. Just grab the hash of the interesting credential (inspect the html form field) and run the following script:
It will print the value hidden behind selected credential hash.
If you get access to AWS access keys that belong to Jenkins, you can do everything the Jenkins user/role is allowed to do.
The above examples are all true and possible. To illustrate that this does indeed happen, let me tell you a little story of one case I was involved in recently.
PGS Software was doing a security audit for one of our clients a few months ago. As part of this audit, I was given AWS credentials to the client’s account with read-only access.
Using these credentials, I checked every part of the project infrastructure – and, not after long, I found a Jenkins machine running on one of the AWS regions.
The machine was located in a public subnet and with a public IP. It also looked like it was created using terraform templates. Security group allowed everyone to access it (0.0.0.0/0).
Unfortunately, there was a login page with no anonymous access.
I also found a full log from the provisioning of the Jenkins instance on CloudWatch. Astonishingly, the log ended with the credentials of a newly created Jenkins admin user.
So I opened the Jenkins login page once again – but this time I was able to login to an admin account. By checking IAM Role and Policies attached to the EC2, I already knew that Jenkins had an AWS AdministratorAccess policy attached to this role, and so I now have full access to the AWS account.
The Client assumed that their account was secured because users had to log in. They didn’t count on the login data being easily accessible on the CloudWatch log with a read-only permission.
Why Does This Happen?
The above access points do not have to be vulnerable. As mentioned in my last article, cyber security is a strategy that needs to be implemented on all fronts.
If you reveal your secrets to the wind, you should not blame the wind for revealing them to the trees. H. Kahlil Gibran
Any security breaches become possible when security measures aren’t applied consequentially and effectively. In the case of Jenkins, we can identify four main issues:
Jenkins is Publicly Available
Not having a VPN in place can have serious consequences. The internet is a vast jungle of connectivity that has its shares of predators, and they will not hesitate to jump on easy prey. If not VPN, lock it at least at a Security Group level, allowing only approved IPs to connect. Home IPs of your employees are not considered safe, as a single IP could cover half of a big city.
Jenkins Security is Disabled
Jenkins has some security options. For example, by default it requires you to log in. Yet, setting these up can be cumbersome and time consuming, so some companies choose to disable the security module and... forget about it later. “How does someone know the IP of my newly created Jenkins machine? Only my employees know that, and they don’t need to login to do things”
Bad Virtual Host Configuration
Domain access is properly restricted, but IP access is using default vhost with no restrictions. This means that when I try to open https://jenkins.example.com the connection will NOT be permitted. But then if I try with the IP behind this domain, it will use a different configuration file on the server and will let me in.
Broken Principle of Least Privilege
“Give Admin rights to everyone, they may need it at some point”. This is a common one – admin rights are given to people because they think that, at some point, they might need it. This can happen when you don’t want your technical lead to be bothered every 5 minutes with a request.
How to Avoid These Leaks
The above points can definitely be avoided with the right security measures and strategy. Here are some tips on what you can do so secure your AWS accounts running Jenkins.
- Make sure that Jenkins is only accessible from a local network or use a VPN
- Ensure that the security settings in Jenkins are properly set up - check the domain and direct IP access.
- Only give Jenkins access to your AWS environment as required. The Principle of Least Privilege is your friend.
Security is a never-ending story. As you expand your business and start using more digital tools, you must make sure that you stay on top of the weaknesses of your platforms and accounts. Educating yourself and conducting internal and external security audits will help you in understanding and owning the tools to secure your systems and AWS environments.
The ultimate security is your understanding of reality. H. Stanley Judd
My advice to you is make the security analysis of each component of your digital solution a standard practice. That way, you stay in control and will be able to fend off any unwanted intruders.