What if you were able to point your AR device at an object and gain actual insightful information from it? That's the idea behind my IoT/AR Fridge project. I had a couple of old small refrigerators laying around and I realized that something like this could make a great portable IoT/AR demo. So, I bought an Arduino and a few sensors in order to work on prototyping the hardware aspects of the device.
What I wanted to accomplish can be achieved with just three sensors. The DHT-11, which measures temperature and relative humidity is perfect to let the user know if the drinks inside the refrigerator are cool enough to consume. I also wanted two strain-gauge sensors, each hooked up to its own HX711 D/A converter. I then placed the two strain-gauges on either side of the cooler. This is underneath a platform on which the can of soda will sit with their tops facing the fridge door. The sensor data gets pulled into the Arduino, which can transmit it over a Bluetooth module directly to a phone and be used in a custom app. Alternatively, a WiFi module can be used in order to have the information transmitted to an IoT backend like AWS IoT Core or PTC Thingworx.
First I wrote an Arduino sketch that gathers all of the information from the sensors. Then, I started to figure out the AR aspects of the experience in Unity. I used Object Detection in ARKit to determine where to place the AR elements. In the future, I would love to have this work with an iBeacon in conjunction with object detection to increase accuracy. Once I was able to get that working, I started brainstorming about how these UI elements will look. In its current iteration, the UI is made up of a bunch of circles, one representing each insight, billboarded to the camera. It's simple and easy to understand.
What I wanted to accomplish can be achieved with just three sensors. The DHT-11, which measures temperature and relative humidity is perfect to let the user know if the drinks inside the refrigerator are cool enough to consume. I also wanted two strain-gauge sensors, each hooked up to its own HX711 D/A converter. I then placed the two strain-gauges on either side of the cooler. This is underneath a platform on which the can of soda will sit with their tops facing the fridge door. The sensor data gets pulled into the Arduino, which can transmit it over a Bluetooth module directly to a phone and be used in a custom app. Alternatively, a WiFi module can be used in order to have the information transmitted to an IoT backend like AWS IoT Core or PTC Thingworx.
First I wrote an Arduino sketch that gathers all of the information from the sensors. Then, I started to figure out the AR aspects of the experience in Unity. I used Object Detection in ARKit to determine where to place the AR elements. In the future, I would love to have this work with an iBeacon in conjunction with object detection to increase accuracy. Once I was able to get that working, I started brainstorming about how these UI elements will look. In its current iteration, the UI is made up of a bunch of circles, one representing each insight, billboarded to the camera. It's simple and easy to understand.
The two channels of data from the strain gauges allow me to determine how many cans of soda are in the fridge as well as their orientation. With this information, I can overlay the cans in AR in real time, giving the user the impression that they can see the cans inside the fridge.
In the future, I would like to have this data constantly streaming into an IoT backend so I can do things like send a user an alert if the door is left open, or track the temperature reading over time on a graph.
This project shows how you can combine AR with IoT data to create some very interesting experiences. I hope that when people see this project, it sparks their imagination and allows them to see some of what's possible with AR.
