The simplest way to analyze your cloud configuration: select * from cloud

In this blog, we aim to inspire you to use Steampipe to analyze your cloud configuration. Steampipe is elegant, fast, and works with any API!

When analyzing cloud configurations, you often start by exploring the console to understand what is there. Once you have a grasp, you may start to use some shell scripts. Although shell scripts work well for simple queries, they quickly become complex when dealing with resource relationships. Enter Steampipe: Steampipe provides access to your cloud resource configurations through a plain old PostgreSQL engine!

Analyzing Your Cloud Configuration

Let’s say you want to check that all Google Pub/Sub configurations in your organization are robust. This means:

• Every topic has at least one subscription, preventing messages from disappearing.
• Every subscription has a dead-letter topic, ensuring failed messages are not lost.
• Every subscription has an exponential backoff configuration for retries, preventing rapid-fire retries.

Every topic has a subscription – shell script

Let’s look at the implementation of the first check: whether every topic has at least one subscription. According to chatGPT, the shell script will look like this:

projects=$(gcloud projects list --filter="parent.type:organization AND parent.id:$ORG_ID" --format="value(projectId)")
for project in $projects; do
    topics=$(gcloud pubsub topics list --project="$project" --format="value(name)")
    if [ -z "$topics" ]; then
        # No topics found in project: $project
        continue
    fi

    for topic in $topics; do
        subscriptions=$(gcloud pubsub subscriptions list --project="$project" --filter="topic:$topic" --format="value(name)")
        if [ -z "$subscriptions" ]; then
            echo "$topic"
        fi
    done
done

This looks correct. Let’s run this:

$ time ./print-topics-without-subscriptions | tee topics-without-subscriptions.out
Topic projects/xyzabcdf/topics/custodian-auto-audit-notify-on-disallowed-services does not have any subscriptions.
Topic projects/ladilada/topics/cloud-builds does not have any subscriptions.
Topic projects/somehere/topics/bucket-events does not have any subscriptions.
....
./print-topics-without-subscriptions  41.29s user 8.21s system 24% cpu 3:23.29 total

$ wc -l topics-without-subscriptions.out 
      63 topics-without-subscriptions.out

It found over 63 topics without a subscription which took more than three minutes to retrieve. Let us see how this works with steampipe.

Install steampipe

First install Steampipe and the Google Cloud Platform plugin:

$ brew install turbot/tap/steampipe
$ steampipe plugin install gcp 

generate the Steampipe configuration for gcp

Run the following script to generate a Steampipe configuration which allows you to query across all projects in your Google organization and replace the GCP plugin configuration at ~/.steampipe/config/gcp.spc:

#!/bin/bash

export org_id=$1
[[ -z $org_id ]] && echo "Usage: generate-gcp-org-steampipe-config <org-id>" && exit 1

projects=$(gcloud projects list 
  --filter="parent.type:organization AND parent.id:org_id" 
   --format="value(projectId)")

cat<<!
connection "gcp_all" {
  plugin = "gcp"
  type        = "aggregator"
  connections = ["gcp_*"]
}
!

for project in $projects; do
  cat <<!
connection gcp_$(sed -e 's/[^a-zA-Z0-9]/_/g' <<< $project) {
  plugin = "gcp"
  project = "$project"
}
!
done

Now you are ready to use the plain old SQL!

Every topic has a subscription – Steampipe

The PostgreSQL query to find all topics without a subscription is shown below:

select * from gcp_all.gcp_pubsub_topic topic
where not exists ( 
  select '' from gcp_all.gcp_pubsub_subscription sub
   where ('https://pubsub.googleapis.com/v1/' || sub.topic) = topic.self_link
) 

To run this query, type:

$ time steampipe query topics-without-subscriptions.sql --output json > topics-without-subscriptions.json
steampipe query topics-without-subscriptions.sql --output json >   0.07s user 0.12s system 46% cpu 0.390 total
$ jq '.|length' topics-without-subscriptions.json
63

So here you get the same answer, but 200x as fast! Of course, you can now use SQL to count too:

select count(*) as total, 
       count(sub.name) as with_subs,
       count(sub.name)*100/count(*)  as percentage
from gcp_all.gcp_pubsub_topic topic 
left outer join gcp_all.gcp_pubsub_subscription sub
on ('https://pubsub.googleapis.com/v1/' || sub.topic) = topic.self_link

In our case, it tells us that only 7% of all our topics have a subscription.

Every subscription has a dead letter topic – steampipe

To find all subscriptions without a dead letter topic, use the following query:

select * from gcp_all.gcp_pubsub_subscription
where dead_letter_policy_topic is null
and name not like 'dead-letter-%'    // exclude dead letter topic subscriptions

In our case, none of the subscriptions have a dead letter topic.

Every subscription has a exponential backoff – steampipe

Finally, to find all subscription without exponential backoff, use the following query:

select * from gcp_all.gcp_pubsub_subscription
where retry_policy_maximum_backoff is null 
or    retry_policy_minimum_backoff is null 

Again in our case, none of the subscriptions have a backoff policy.

Steampipe plugins

The examples above use the Google Cloud Platform plugin, but Steampipe offers over 100 plugins ranging from GCP, AWS, Azure and Kubernetes to Google Sheets. This allows you to get insight into different platforms and products using the same language.

Steampipe IDE access

You can also start Steampipe on your local machine as Postgres database instance and use your favourite IDE to explore the tables and data.

Lastly, you do not have to write all queries yourself: you can find many example queries for compliancy checks on github.

Conclusion

steampipe is an awesome tool to analyze your cloud configurations. It is fast, and provides the well known SQL query language to explore your cloud configuration. The fact that any application with an API can be explored using steampipe, makes it a incredibly elegant tool in your toolbox.

The examples above show how to analyze your Google Cloud Platform configuration. But the same queries can be used for analyzing AWS, Azure, or any other platform that has a Steampipe plugin.

So, go ahead and try it!

Image by
Davinder Sanghafrom
Pixabay