During our work in the field we get a lot of questions regarding security and compliance. With Cloud getting a more prominent place in the digital world and with that Cloud Service Providers (CSP), it triggered the question on how secure our data with Google Cloud actually is when looking at their Cloud Load Balancing offering. This is especially the case for the solutions that do SSL offloading. In this blog you will find some information we've gathered to answer this question.
This blog is targeting an audience using Google Cloud Platform (GCP). This means that the information in this document contains references to technical solutions and documentation as offered by GCP. Applicability may be inferred to other CSP's as well, but is not validated.
For companies with a high security profile it's important to identity all places where the data passes and how to trust that this data cannot or will not be used by the CSP. During threat modelling, the SSL Load Balancing offerings often come into the picture. The biggest question asked here is:
"If the load balancer does the SSL offloading, how can we be sure that this data is not accessible by third parties?"
Types of load balancers
The first part of answering this question is which types of load balancer are on the table. When looking at the offerings there are multiple types of load balancers available.
- Global/Regional External HTTP(S) Load Balancer
- External SSL Proxy Load Balancer
- External TCP Proxy Load Balancer
- External TCP/UDP Networking Load Balancer
- Internal HTTP(S) Load Balancer
- Internal TCP Proxy Load Balancer
- Internal TCP/UDP Networking Load Balancer
From the documentation on these load balancers we can identify that only the following three load balancers do actual SSL offloading. The others merely process and forward the requests.
- Global/Regional External HTTP(S) Load Balancer
- External SSL Proxy Load Balancer
- Internal HTTP(S) Load Balancer
The identified risk with SSL offloading is that at that point in time the data is no longer in encrypted state and can theoretically be used for other purposes than intended. Looking at a very basic network topology, it becomes visually apparent where the risk lies.
The load balancers technically act as a man-in-the-middle, moving the trust boundary of this risk fully towards the chosen load balancer with SSL offloading.
With the identified risk we need to look at the possible mitigations that have been put in place, to prevent any malicious use of the data. Unlikely sources would be individuals that gain access to these load balancers, but far more likely sources would be state actors that (by law) require this data to be forwarded/mirrored without notice. We need to dig through the documentation to identify the mitigations that have been put in place for us.
Google puts a lot of effort into reducing the physical vulnerability.
One aspect is access to the actual hardware running in Google's data centers. The physical security boils down to a six-tier security perimeter, where each tier requires even more and more stringent permissions to be granted access. This means that the load balancers are only physically accessible by an extremely limited amount of employees. Because of the technicalities involved, they won't be able to directly access the data on the machines and all interactions are carefully monitored, audited and recorded.
Another aspect is that of the hardware itself. All these measures can become useless, if the security risk is built into any of the hardware components. To mitigate this issue, Google only uses highly trusted vendors for hardware components and integrates these in server boards and the networking equipment that has been in-house designed by Google. This is further enhanced by Google's proprietary Titan chips, to validate en ensure the correct and secure operations of all components within the hardware stack.
After identifying the mitigations put in place to ensure the physical protection of the hardware our attention moves to the operations side of the risk. Within the operations side we identify the following topics:
- Locations of the load balancers
- Physical configuration of the load balancers
- Software on the load balancers
- Locality of the data after SSL offloading
- Other uses for the data after SSL offloading
- Propagation of the data through Google's network
Location of load balancers
The way Google's network has been set up, is to differentiate between the locations where the Cloud resources reside (so-called regions and zones) and where traffic is processed that comes from (mostly) outside Google's network. The latter is done on the "edge Point of Presence" (more commonly referred to as Google Front End or GFE). This can be visualized in below image.
The load balancers reside in the GFE's. As can be seen in the image, this doesn't necessarily mean that these locations are physically in the same geographic location.
If you are concerned about the geographic location where the traffic enters Google's GFE's, then you can opt to use only the Standard Tier network options from Google. This will result in all requests being processed by the ISP's networks and only enters Google's network at the GFE closest to the configured resources. This can have significant impact on the experience of your service, as it will no longer use Google's high speed fiber network.
Physical configuration of the load balancers
The load balancers that are used on Google's GFE's are custom-made and fully designed according to strict specifications by Google. This includes the previous mentioned hardware components, including the security measures in and for these components. Only the components that have passed the strict validations are allowed to be used.
Software on the load balancers
Similarly, the software on the load balancers is purpose written for use on the designated machines, and is called Maglev . Even though the source code for this system is not available publicly, the validity of this software is frequently verified and documented in the SOC reports, amongst other audit reports. This ensures that the software used is not directly vulnerable for actual attacks. It does however move the trust balance towards the auditors. Compliance officers in your company should verify the validity of the mentioned reports and determine the level of trust with the auditors.
The load balancers themselves are also equipped with secure boot through the Titan chips and Binary Authorization, amongst other measures to prevent tampering with starting and/or running instances.
Locality of the data after SSL offloading
When the traffic is offloaded on the load balancer, the data will be in raw format. Depending on the type of data, it can mean that it's in readable state. This raw data on the load balancer will remain within the physical machine at all times, while it's in this state. When temporary storage of this data is needed, it will always be encrypted while at-rest. This is in line with Google's overall security requirements, and is not optional.
Other uses for the data after SSL offloading
There are other services that might be applied that will need access to this data. One of these is Cloud Armor. Depending on the type of protection rule, this service will inspect the data from the request to determine the expected outcome. Similarly to the Cloud Load Balancing offerings, Cloud Armor is also executed on specialized hardware and adhere to the same security restrictions and scrutiny. That does mean that the same trust boundary applies, and by extension requires the same process with your compliance officers.
Propagation of the data through Google's network
Within Google strict guidelines are in place if it comes to encryption. This means that data in-transit will always be encrypted. If it has been offloaded by a service it will always be re-encrypted using Google's Virtual Network Encryption. This also applies to the data coming from the load balancers, once it's ready to leave the machine towards the designated next service.
Auditing and certifications
As mentioned a couple of times in this document, Google puts in a lot of effort to ensure the security of the services it offers. This is done by adhering to a plethora of certifications and audits, which are being done by internal and external parties and in high frequency. The results can be viewed and/or requested at the Compliance Resource Center.
Limitations when avoided
It can be that the information provided doesn't convince you and/or the compliance officers at your company. This can mean one of two things:
- We choose not to use the Google Cloud Load Balancer offerings that do SSL offloading
- We choose to only pass through data that has been encrypted on application level
We opt not to use the load balancer
If you opt not to use the load balancer offerings, then this will mean that you cannot profit from all the capabilities that these load balancers have to offer. These are capabilities like out-of-the-box (D)DoS protection, access management through Identity Aware Proxy before your workloads and/or services are reached, the more advanced security measures provided from services like Cloud Armor and BeyondCorp and others. This also means that if you decide to handle these capabilities in-house because of the security restrictions, you will need to perform the development, maintenance and auditing yourself.
We opt to double encrypt to avoid scanning
If you chose this option, this will inevitably add more complexity to your workloads. It will depend on the application you will build and/or use to make requests to your services, if it is capable of performing encryption itself. This is for example not something you could easily implement when providing your service to a broad public through a web browser, but might be useful for in-house developed software like mobile applications.
Be aware that this may also limit the possibility of the use of other advanced security services, like Cloud Armor or BeyondCorp.
With the use of Cloud Service Providers you will inevitably need to put some trust in the chosen Cloud vendor. Google, like probably all other respectable vendors, will put in a lot of effort to earn and prove their trustworthiness. This is done by frequent audits and compliance validations and certifications.
When specifically looking into the technical details of Google Cloud Load Balancing with SSL offloading, we identified that there is a risk with your data when it's offloaded. We also identified that Google has put in place a lot of measures to ensure that this data will be kept secure.
On the physical side, security is ensured by defense-in-depth with a six-tier security layer protocol, stringent requirements on the used hardware and special hardware to keep track of the behavior of this hardware. Even when employees with permission are able to access the physical devices, they are still unable to access the data itself and all actions are strictly monitored and audited.
On the operations side, this is done by the deployment of specialized software that again is audited. This is constantly monitored and maintained by techniques as Secure Boot and Binary Authorization, amongst other techniques. If data needs to be temporarily stored, this will always be encrypted while at rest and always within the same physical machine. Whenever it leaves the machine, it will also always be encrypted using Google's Virtual Network Encryption.
If these measures don't convince, you can still consider other solutions. These will however come with downsides and should be considered carefully, as it will require much effort to even get on-par with Google's security measures. This is something you will always want to decide together with your compliance officer.