An Insight into Payoda’s Experience in IoT Quality Assurance
At Payoda we analyze, test, observe, report, regress, ensure and certify the correlative behavior of a variety of devices with their associated hardware and software components as a part of our IoT Quality Assurance services. We have built a proven list of test cases for the IoT system that comprises of the basic, in-depth, practical, and out of the box scenarios. Our test suite takes into account not just the ambient but also the external factors that impact the functioning, not just the characteristic of the device but also the mindset of the end user — both of which should always be considered in the case of any testing exercise, especially IoT.
The Manager Software
The Manager device is the heart and the Energy Center software acts as the brain of the innovative integrated energy management IoT system. The Manager device coordinates all energy management functions utilizing Ethernet/ wireless connectivity to receive information from associated third party devices and shares it with the Energy center application. The energy center application, in turn, processes the information against preset mode/values and sends appropriate control signals which produce an effect on the endpoint devices.
A general brief on the crucial role of Sensors
Sensors form the basis of modern building energy efficiency, connecting HVAC systems, lighting, refrigeration, and other facility operations into a powerful network that proactively manages energy in the best possible way. Some of the IoT integrated sensors that we provide quality assurance for, monitor occupancy, daylight/night, door contact, temperature, humidity, and other environmental variables. They provide real-time data that are essential to appropriately tune, improvise, and increase the operational performance of the related devices.
Occupancy Sensors
We test several occupancy sensors in combination with light sensors in several modes such as the Smart on/off mode also known as adaptive lighting control mode, the vacancy mode, occupancy/non-occupancy mode, dim level change among others. Each mode has different functionality and different practical use. Each sensor works differently and is best suited for a specific practical installation. For example, the Enocean occupancy sensor uses RF waves to communicate with other devices. We also have wall-mounted, ceiling-mounted, and wired occupancy sensors that differ in the type of energy source, coverage, installation points, and in their mode of communication.
Testing of Occupancy & Light Sensors and their different modes of operation:
To test the Adaptive Lighting Control mode, we connect the occupancy sensor in combination with a light sensor. The practical scenario reconstructed in our premise for testing purposes was a long hall with two doors at either end, with windows and lights placed throughout the space. At both the entry points, we placed an occupancy sensor, and above each window where light is situated, we placed an occupancy/light sensor combo. We configure the door sensors to work in the occupancy mode wherein the light is turned on when motion is sensed. This light can be turned on to its maximum brightness or to the desired lux level as configured in the energy center software. The light/occupancy sensor combo works in the adaptive lighting control mode wherein the lux level of the external light source is calibrated and the internal lighting brightness is controlled accordingly. If the external light through a certain window is bright, the lighting device above it will be tested to be glowing at a lower lux level when compared to a lighting device above a window that has a lower level of external light.
During our course of testing, we found that the adaptive lighting control does not work for external artificial lights as it does for natural sunlight. When there is an external artificial light coming through from an adjacent room, the light in our room need not glow at the same lux level as it should when there is no external light at all. But we found out that it does. We reported it to our clients, and they welcomed the issue and took it up for further scrutiny and eventually modified the behavior of the sensor. We were able to think on these lines solely because we always have the ultimate practical goal of every IoT system that we test embedded in our mind, in this case, energy efficiency.
Another interesting case in our quality assurance journey of these sensors was with the occupancy mode of the light/occupancy sensor combo. Consider a series of light/occupancy sensor combos connected to lights that are affected by a single command. When there is occupancy detected by one of these sensor combinations, all the lights in the hall were turned on. That is the expected functionality. But when one of these sensor combinations detected vacancy, all the lights were turned off. A logical error in terms of devising the sensor behavior was discovered. We reported that the vacancy in this series connection of sensors should effect turning off all the lights only when all the sensors in the circuit report a vacancy and not just one. The client provided his acceptance of this issue as well and they are currently performing a revamp of the code that deals with this functionality.
Testing of the Thermostat devices and their different types:
The wireless duct sensor has a steel rod attached to the sensor which can measure the temperature of the water it is in contact with. The practical implementation of which is to turn the heater off via the IoT network when the desired temperature configured in the software is attained. The wireless duct sensor is installed within the heater and relays temperature readings continuously to the energy center software. When the desired temperature is reached, the energy center software provides a signal which turns off the heater. But, we found an issue with the logic when the temperature of the heated water drops below the threshold level. When this happens, the signal from the software should again turn the heater on, but it did not. There was no logic implemented for this to happen. Our client appreciated our insight and approved the issue as it improves and makes the IoT system even more intuitive.
Types of Switches and how we test their function in the IoT system
Our client manufactures both wired and wireless switches. These control the light levels and can be used in day light-harvesting in combination with devices like dimming fixture controllers, light sensors, occupancy sensors, and HVAC systems. Let’s say that the end-user has left his mobile at home. He will not be able to access the Energy center software application to control his hall lighting. But assume that he is somewhere around the vicinity of his home and he has a small rocker pad switch with him. He uses the rocker pad to turn on/off the lights by clicking on the button once. He can also adjust the lighting level in his hall by clicking up/down the rocker pad button multiple times continuously. During testing, we found that the continuous click of the up button caused the light to be turned off completely instead of the level of light is adjusted. Also, the rocker pad has a coverage range of 80ft. We found that it fails to communicate with the sensors and does not send signals even from a distance of 70ft.
Testing of Schedules in the Energy Center Software
The energy center software provides the facility for the end-user to configure schedules for specific home appliances. Suppose the user is away and wants to turn on his porch lights by the night and turn them off by the day, he can configure this in the software. But we found glitches in the software where the signals sent to the light sensors turned them off when the Lights On the schedule was on. This did not happen right away but after several hours of observation. Also, we found that when we canceled the schedule altogether during the night time, the light instead of being ON was turned OFF. During the day if the same schedule cancellation was made, the Light remained OFF, which was expected. Both findings were accepted by the client.
Testing of CO2 sensors with Temperature Control Appliances
The CO2 sensors measure the carbon-di-oxide levels in a room in PPM. When the number of people in the room goes up so does its CO2 levels and its temperature. This is measured by the sensor and is communicated to the energy center device, which relays the information to the software. The software then sends a control signal to the air conditioning system to lower the temperature to maintain the comfort level of everyone in the room. But in our experience, we found that the Co2 measurements aren’t always accurate when there are several people in the room. Due to this temperature, controlling using Co2 readings wasn’t as precise as we wanted it to be. The Co2 sensor is currently in development review to find the root cause of the issue found.
Testing Approach to IoT system & the Journey Forward
Testing an IoT system requires an accurate understanding of the device’s function, its correlation with other devices in the circuit, the communication modes and channels, the device’s behavioral modes, and its specifications in terms of temperatures, lux levels, etc. Apart from all these, Payoda QA invests time and effort to understand the practical implementation of the devices, the expectations of the end-user in order to think differently, and to bring out instances that are crucial to perfect the functionality of the device but aren’t implemented or functioning incorrectly. We have a robust number of carefully curated scenarios for each device, which we ensure are properly working before we certify any live installations. In order to reduce the regression testing efforts, we divide and conquer. We have completely automated the energy center software, so that when our automation script is executed, we just need to verify the behavior of the devices based on the configuration that each script provides to the software, thus making the regression rounds more effective and less time-consuming. The above-mentioned devices are just a small subset of more than hundreds of devices delivered to us, that were successfully quality assured and signed off for live installations by our client to make the lives of our users that much easier. Scenarios might differ based on the characteristics of the device in hand, but our strategies, innovativeness, and the sympathetic user based thinking we implement for IoT testing have remained and will continue to remain the same for any future prospects. With our hard-earned knowledge and experience on the IoT testing front, that has matured over the years, we feel we possess the terrific capability to cater to the challenging demands of any future client who decides to foray with us.
Author: Mohan Bharathi & Sathiyapriya
Stream: Quality Assurance
