Cloud Engineer Spotlight – Youri’s Perspective On IoT With AWS

Back in January of 2018, Xebia achieved the AWS IoT Competency. Since then, we have had the opportunity to work on a variety of IoT customer cases while making use of the full potential of the AWS IoT stack. In order to stay ahead with IoT technologies, we have created a dedicated cloud innovations team that leads the development and execution of innovative solutions for our customers.

Part of this team is Youri van den Berg, who has been working with us as a Cloud Engineer for three years now. Educated in IoT on AWS through bootcamps and hands-on experience, he is now branching out our IoT knowledge to our customers. To better explain how we tackle IoT cases, we’ve asked him for some insights on how to get the most out of the AWS IoT services.

The case

One of our biggest IoT projects enabled a customer to accurately and timely measure temperatures of public outdoor platforms. During winter, it is common for a layer of ice to form on the ground and applying salt is necessary to remove snow and black ice to keep areas safe. Previously, when considering whether applying salt would be needed, our customer only had a weather forecast available to base their decisions on. A smart IoT solution was needed that could actively measure ground surface temperatures in order to prevent unnecessary applications of salt. Also, the data had to be made securely available and usable for an external weather platform to be used for predictive modeling.

Setup of the solution

Together with an external party we developed a sidewalk tile embedded with sensors, one directly below and another about 30cm into the ground to guarantee reliable readouts. The temperature sensor is built as a standalone device without external power supply and thus runs on batteries. To ensure the battery’s longevity, a LoRaWAN (Long Range Wide Area Network) solution is used. LoRaWAN is a network layer on top of LoRa, which allows long range communication with low power consumption. Every 15 minutes, the sensor sends a temperature reading over the LoRa Gateway which has nationwide coverage.

Youri: “One challenge that surfaced was that the range of the LoRa signal was heavily limited by being underground and therefore could not effectively transmit the data, leading us to set up an additional LoRa Gateway on site. The data is encrypted in transit and can only be decrypted by the application backend, preventing access and modification to the data itself before it reaches the application backend.”

Once the data has been received by the API it is published to a topic on AWS IoT core. AWS IoT Core is a managed cloud service that lets connected devices easily and securely interact with cloud applications and other devices. Additional Lambda functions are then triggered based on a set of rules to transform the data from raw bits into an interpreted form so it can be easily stored in the database. At the end of its journey, the data is sent to an internal data warehouse (Redshift) where it is stored for further analysis. All raw data received from the ‘things’ is vented into Amazon S3 (and eventually Amazon Glacier) using a trigger on all of our incoming raw topics. This enables us to replay any further preprocessing or extract useful data from these raw messages or the meta-data by reinjecting them.

The Lambda functions can pre-process data and can in turn use different functions, or republish the data into a different topic. At the end, the data is stored in a raw datastore. We can process this data further into different slots and data marts. The data marts are the sources for the M2M interfaces and reporting applications while the raw data can be used for further data exploration. To comply with the requirements for third- party sensors/devices and different LoRa providers, the solution exposes a MQTT or REST-endpoint using API Gateway and pre-process the data before we feed it into AWS IoT.

Once the data reaches the data warehouse it is made available for external customers, like the weather platform, through an API for further processing. A frontend web application allows the customer to visualize the data, compare readings or view historic data. The web application is a Rails application and runs as an auto scaling ECS Fargate service. The M2M API that is exposed for external consumers is documented using OpenAPI to allow for a more streamlined integration.

Youri: “Further data interpretation happens at the client. There we have set up the right BI tooling which allows our customer to visualize the data, completing the solution. The complete IoT platform evolved gradually, but most of the effort was put into efficiently developing the sensors. This included aligning the specs of the sensors, establishing communication and implementing the LoRaWAN connectivity. In cases where the range was not sufficient, we added extra LoRa gateways on the customer’s site.”

Security

Configuration of applications and LoRa devices and accompanied keys are done within the used platform. A device is provisioned using OTAA (Over The Air Activation) which will generate a unique session-key based on AES cryptography. Authentication and authorization is handled by the LPWAN middleware by creating and/or revoking keys for devices. Data which is transmitted over the air is encrypted based on the unique session keys and can only be decrypted in the final handlers of the data. Data is decrypted in the Handler stack just before it is delivered onto the AWS IoT topic.

Lessons learned 

Youri: “We learned a great lesson on having to balance cost optimization and flexibility of the platform. The IoT services had to be finetuned on various topics. We started doing this very thoroughly by setting up rule triggers for every response through the IoT channels: when certain requirements were met, predetermined actions followed. However, you also pay for every time this process is triggered. We solved this by moving to a single Lambda function. Data processing was sufficient through this one function and saved a lot of costs.

We were confronted with the fact that real world scenarios are significantly different from a lab setting. Once you’re outside the lab with your solution, new factors come into play. For example, the cold temperatures had a greater effect on the battery life of the sensors than we anticipated and needed to be adjusted accordingly. Additionally, there was a drop in the predicted signal range once the sensors were placed underground, as I mentioned earlier.”

Results

In the end, our client was now able to collect data from the public areas and to use a BI platform to analyze the data generated by the sensors. From there, they are able to predict what kind of action, preventive or responsive, they can take in order to reduce operational and maintenance costs, but ultimately to mitigate weather related risks to the customer’s infrastructure and ensure a safer environment while using less resources than before.

How will your story start?

Curious as to how your career could go with us? Make sure to check out our careers page to learn more about Xebia and what you can expect from a career choice with us.