In my previous blogpost I showed how to read gas and electricity measurements from a smart meter. Although it is a nice accomplishment to be able to read these measurements, the next step is obviously to do something useful with them. So, let's create some nice dashboards based on these measurements. In particular I want to have dashboards that show the total amount of electricity and gas usage as well as the current electricity usage. I decided to use the TIG stack for this purpose. The TIG stack consists of Telegraf, InfluxDB and Grafana. Before I describe in debt what they are and how to configure them I will first introduce the Influx line protocol.
Influx line protocol
Telegraf supports multiple data input formats, which you can find here. The influx line protocol is one of the supported formats and is a text based format for writing points to the database. Each line in the influx line protocol represents one data point in InfluxDB. And it has the following format:
measurement,tag_set field_set timestamp
The first part called measurement is the name of the measurement that you want to write the data to and is a required field. The tag_set represents the tags you want to include with the data point and are optional in Influx line protocol. Then the actual measurement(s) follow in the field_set part of the line protocol. And finally an optional timestamp. Note that Influx line protocol has a strict format and uses comma and whitespace as separator characters. You can read more about the Influx line protocol here. Then I changed dsmr-forwarder to publish measurements in Influx line protocol format. In this case I prefer the Influx line protocol format over another format, because this way gas and electricity measurements are each represented by a single line. Thus, information that belongs together is grouped together in a single data point. The method below parses the given telegram and produces two lines in Influx line protocol, one for electricity and one for gas measurements. In the next step I publish these lines on MQTT with each line on it's own topic, as in:
Telegraf is a plugin-driven server agent for collecting and reporting metrics and has several integrations available out of the box. In this case I use Telegraf to read my smart meter measurements from MQTT and write the results to InfluxDB. You can generate a default Telegraf configuration with the following command:
docker run --rm telegraf telegraf config > telegraf.conf
Next, I adjust the default Telegraf configuration file to enable the MQTT inputs section by uncommenting [[inputs.mqtt_consumer]] and let the servers property point to my MQTT broker. The basic configuration for the MQTT consumer input is shown below:
# # Read metrics from MQTT topic(s) [[inputs.mqtt_consumer]] ## MQTT broker URLs to be used. The format should be scheme://host:port, ## schema can be tcp, ssl, or ws. servers = ["tcp://your-mqtt-broker:1883"] # ## MQTT QoS, must be 0, 1, or 2 qos = 0 Connection timeout for initial connection in seconds connection_timeout = "30s" # ## Topics to subscribe to topics = [ "dsmr0/p1/#" ] data_format = "influx"
Note that I configured Telegraf to consume all topics from MQTT starting with 'dsmr0/p1' and that I set the data format to 'influx'. Then I adjust the Telegraf configuration file to enable InfluxDB outputs section by uncommenting [[outputs.influxdb]]. I set the target database and the username property to 'telegraf'. We will need to create those in InfluxDB later. See the basic configuration below:
# Configuration for sending metrics to InfluxDB [[outputs.influxdb]] ## The full HTTP or UDP URL for your InfluxDB instance. ## ## Multiple URLs can be specified for a single cluster, only ONE of the ## urls will be written to each interval. # urls = ["unix:///var/run/influxdb.sock"] # urls = ["udp://127.0.0.1:8089"] urls = ["https://influxdb:8086"] ## The target database for metrics; will be created as needed. database = "telegraf" ## If true, no CREATE DATABASE queries will be sent. Set to true when using ## Telegraf with a user without permissions to create databases or when the ## database already exists. skip_database_creation = true ## Timeout for HTTP messages. # timeout = "5s" ## HTTP Basic Auth username = "telegraf" password = "a-strong-password-here"
Now the Telegraf configuration is prepared to consume our dsmr MQTT topics and write the measurements to the database. Before we start Telegraf we will first configure InfluxDB.
InfluxDB is an open-source time series database and is like Telegraf developed by InfluxData. It has the ability to store large amounts of time-stamped data. First, we create a docker network such that Telegraf and InfluxDB containers can easily communicate with each other:
docker network create influxdb
We can run the InfluxDB docker image with the following command:
docker run -tid -p 8086:8086 -v /home/pi/influxdb:/var/lib/influxdb --net=influxdb --name influxdb influxdb
The next step is to create the database and user for Telegraf. So, let's connect to the InfluxDB container by invoking the following command:
docker exec -it your-influxdb-container-id bash
Inside the running container you can run:
to launch the InfluxDB console. Let's create the telegraf database with:
CREATE DATABASE telegraf
And create a database user with:
CREATE USER telegraf WITH PASSWORD 'a-strong-password-here' GRANT ALL ON telegraf TO telegraf
Now we have InfluxDB up and running with a dedicated database and user for Telegraf. Once InfluxDB is running we can start Telegraf with the following command:
docker run -d --net=influxdb --name=telegraf -v /home/pi/telegraf:/etc/telegraf:ro telegraf
Then we can check the logs of Telegraf to verify whether it successfully connected to the MQTT broker with:
docker logs -f telegraf
And verify whether the measurements are successfully read from MQTT and inserted into InfluxDB. First we connect to the running InfluxDB container and launch the InfluxDB console again. If everything is working correctly then our measurements are now ingested into InfluxDB and we can verify that with the following commands:
USE DATABASE telegraf SHOW MEASUREMENTS SELECT * FROM p1_elec LIMIT 5 SELECT * FROM p1_gas LIMIT 5
Note that p1_elec and p1_gas should be part of the listed measurements.
Grafana is a data visualisation and monitoring tool with support for multiple input datasources like Graphite, InfluxDB, Prometheus and Elasticsearch to name a few. Recently, Grafana added support for ARM architectures. So now you can install the official Grafana releases on your Raspberry Pi. You can download and install Grafana with the following commands:
wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.2.2_armhf.deb sudo dpkg -i grafana_5.2.2_armhf.deb
Then you browse to https://grafana-host-ip:3000 and login in with admin/admin. In Grafana we configure a datasource for InfluxDB using the telegraf database and user created during InfluxDB setup. Then finally we can create a dashboard and plot our desired graphs. Here is the configured query for the current electricity usage graph. And below are some graphs from my dashboard. Happy monitoring ;-)