How To Use Squid Proxy To Access Services In Another VPC Using A Multi-NIC VM

This blog uses a Squid Proxy VM to access services in another Google Cloud VPC. While Squid can be used as a transparent proxy, this blog uses an explicit proxy for simplicity’s sake.

Connectivity overview

The Squid Proxy allows clients in the Source VPC to connect to services in the Destination VPC:

The Squid Proxy uses two network interfaces to connect the VPCs. One interface accepts requests from the Source VPC and one interface proccesses requests in the Destination VPC. Combined, the Squid Proxy enables clients in the Source VPC to access resources in the Destination VPC.

Find the full example on GitHub.

Bridging the gap

Clients configure a load-balanced Squid Proxy to connect to Destination VPC resources like such:

curl -x http://proxy.xebia:3128 http://example-server.xebia/

The following configuration enables the Squid Proxy to route requests accordingly:

• Accept connections on port 3128

The Squid configuration listens on port 3128 and happily forwards all HTTP traffic:

http_port 3128

..

# Allow all HTTP traffic, use a more fine-grained setup in production..
http_access allow all

..

• Lookup resources in the Destination VPC

By configuring the Destination VPC as the primary interface, DNS queries and IP packets go to the Destination VPC:

resource "google_compute_instance_template" "proxy" {
  ...

  # NOTE: Order of interfaces matter. DNS et al is bound to primary NIC.
  network_interface {
    subnetwork_project = var.project_id
    subnetwork         = google_compute_subnetwork.destination_vpc_nat.self_link
  }

  network_interface {
    subnetwork_project = var.project_id
    subnetwork         = google_compute_subnetwork.source_vpc_proxy.self_link
  }

  ...
}

• Return load-balanced traffic to the Source VPC

The Squid Proxy uses an internal Network Load Balancer to balance requests. An internal passthrough Network Load Balancer routes connections directly from clients to the healthy backends, without any interruption. To accept load-balanced traffic, Google Cloud configures each backend VM with the IP address of the load balancer.

To return load-balanced traffic, a route is configured to use the Source VPC gateway for traffic from the load balancer IP:

echo "Adding 'source' route table for load balanced traffic in source VPC"
SOURCE_CIDR=$(ip -o -4 addr show dev ens5 | awk '{print $4}')
SOURCE_IP=$${SOURCE_CIDR%"/32"}

SOURCE_GW_IP=$(ip route | grep 'dev ens5 scope link' | awk '{print $1}')

# Return load balanced traffic over source VPC interface
echo "1 source" >> /etc/iproute2/rt_tables
ip rule add from ${load_balancer_ip} table source
ip route add default via $SOURCE_GW_IP dev ens5 src $SOURCE_IP table source

Discussion

While this setup prevents you from configuring peerings and/or exchanging too much networking details. The setup does come with several limitations.

First, it’s a one-way communication channel, because of the reliance on the built-in feature of the primary network interface (DNS et al). An additional proxy would be needed to facilitate two-way communication.

Second, it’s prone to overlapping IP addresses. Requests to IP addresses of the Source VPC proxy subnet will leave the Source VPC gateway, instead of the Destination VPC gateway. Additional IP rules can steer the traffic accordingly, but require a more throrough change of the IP routing rules, especially for the local load balancer IP address. Preferably, a safe IP address space is agreed upon, or the problem is addressed using a managed service such as Private Service Connect.

Finally, clients must know which proxy to use for which service. While the proxy name could hint on a network, clients don’t care about network names. Likewise, network administrators want to prevent having a single point of failure and/or complicating the network setup. Hence, for service mesh connectivity, consider alternative tools such as Service Directory or Anthos Service Mesh when running Kubernetes clusters.

Conclusion

There are endless ways to connect Google Cloud VPCs. Squid Proxy VMs with multiple network interfaces allow clients to access services in the target VPC and control/monitor traffic. Note that while the solution seems non-invasive, clients must be made aware of the proxy and the proxy requires a network segment without overlapping IP addresses.

Image by lumix2004 from Pixabay