The Fragile Edge: Chaos Engineering For Reliable IoT

Chaos engineering is a great way of detecting possible failures in IoT devices. This technology has evolved well for testing cloud failure, but open source communities are still working towards building an efficient chaos engineering toolkit for testing IoT devices.

Billions of devices around the world today are connected to the internet, performing a variety of tasks in real time. According to IoT Analytics, the total number of IoT devices worldwide is expected to exceed 21 billion by 2025.

While the number of IoT devices continues to grow, they remain vulnerable to malfunctions. Most IoT devices use basic microcontrollers with only a few megabytes of memory. A smart thermostat worth a few hundred rupees has very little processing power. This limits it to handling only a few tasks at a time. As a result, these devices often fail when they encounter unexpected operating conditions.

Network instability makes IoT systems even more difficult to manage. Edge devices often rely on Wi-Fi, Bluetooth, cellular networks or LoRaWAN — a low power wireless technology that connects devices over long distances while using very little power. These networks are not always reliable, particularly in remote areas or congested cities. For instance, a smart doorbell may momentarily lose connectivity. In large industrial setups, millions of devices can experience disconnections, lost data packets and delays in communication. These are network disruptions. Studies show that in real world IoT systems, up to 5–15% of data packets can be lost. Latency can vary from milliseconds to several seconds depending on conditions. These factors make the smooth operation of IoT systems challenging.

The use of many different hardware components makes it harder to keep IoT systems stable. The Internet of Things includes tens of thousands of different chips, operating systems and communication protocols. Some devices run Linux, while others use tiny real-time operating systems to manage their functions. The storage in these devices mainly uses flash memory and volatile memory (RAM) for processing. As there are so many different types of devices, it is impossible to create a single solution to ensure stability and performance for all of them. Real world events demonstrate how easily these systems can fail. For example, a factory may stop production because its sensor network stops sending data due to firmware problems.

The combination of these factors causes edge failures, which have become increasingly common. Batteries may run out sooner than anticipated, devices may freeze, connections may malfunction, and sensors may provide inaccurate readings. Serious issues arise when transportation networks, healthcare systems and industrial operations malfunction. They can cause delayed medical alerts and stop factory production. Smart home system failures also create real problems, such as smart locks that do not work and security cameras that stop operating. This is where chaos engineering helps solve the problem. Through controlled failure tests, engineers can predict when systems may fail. This enables them to create systems that function normally even in the event of failure.

Table 1: Causes and impacts of IoT system failures

What is chaos engineering?

System testing using chaos engineering involves intentionally inducing system failures to see how they react. Chaos engineering began in the early 2010s when Netflix engineers experienced continuous outages on Amazon Web Services. The platform was new and still evolving. To address this, the Netflix team created Chaos Monkey, a tool that could simulate unexpected system failures in production environments. Chaos Monkey employed a straightforward but effective technique. It randomly shut down servers to see how the system behaved. This taught engineers to never take system reliability for granted. Through controlled failure tests, the team was able to identify weaknesses in the system, and this helped them prevent future real-world breakdowns.

The development of Chaos Monkey gradually expanded into a complete and structured approach known as chaos engineering. Large tech firms like Microsoft, Google and Amazon have adopted this strategy. Smaller businesses like Shopify and Atlassian, too, have begun conducting controlled failure tests. Shopify uses its online store to create artificial database failures, which help the platform sustain its operations during times of heavy user traffic. These controlled failure tests enable businesses to improve their recovery processes.

Chaos engineering is mostly used in cloud environments because it works very well there. However, it is more difficult to apply to IoT and edge computing systems. IoT devices are physical, often located in remote places and sometimes perform critical tasks. This makes managing them even more challenging. Restarting cloud servers using scripts is usually simple. But rebooting medical devices like pacemakers, industrial robots or warehouse sensors is much more complex and can be dangerous. Resetting edge devices also takes longer because system failures often have immediate physical outcomes.

Chaos engineering in IoT systems has both benefits and challenges. Engineers need to design methods to test failures safely without harming devices. The testing process aims to detect equipment breakdowns while developing systems that function during actual operational conditions. The proven cloud software methods of chaos engineering enable organisations to meet the requirements of edge devices.

A commercial home appliance company can use chaos engineering principles to test their IoT cloud platform for weaknesses. This testing will help them improve system reliability and make the platform more robust. The open source µChaos software (used to simulate faults in smart sensors) for the ZephyrOS real-time operating system (commonly used in small IoT devices like wearables, smart home controllers, etc) lets engineers perform fault injection tests. These tests replicate failures and help engineers build tough devices.

The proactive approach of chaos engineering helps organisations to detect system vulnerabilities. This enables them to verify their systems’ ability to operate during failures and recover functionality after breakdowns. Chaos engineering for IoT is difficult but essential for building reliable edge devices.

Bringing chaos engineering to IoT and edge devices

As we saw earlier, the practice of chaos engineering in IoT systems requires engineers to create intentional failures at the edge. The testing process checks whether systems can stay stable when unexpected failures happen. Engineers perform tests by making sensors give unrealistic readings, like -200 degrees Celsius, to see if smart HVAC systems (heating, ventilation and air conditioning systems used for climate control in buildings) handle these situations properly. They also reduce network traffic to test whether industrial equipment can recover automatically. They sometimes restart wearable health trackers during important data transfers. They watch to see if the devices recover while keeping important information safe. Engineers also run power drain tests to quickly deplete batteries. This shows whether users get shutdown warnings before the device stops suddenly.

Researchers at the University of California, Berkeley, employed Raspberry Pi clusters to create sensor networks that replicated the operational behaviour of edge systems under failure conditions. They noticed that application data reporting errors occurred when network packet loss reached 10% of total packets. The implementation of automated recovery scripts reduced system errors by nearly 30%. In another example, engineers at an industrial site performed tests that duplicated sensor breakdowns on their small factory production area. The edge computing modules operated continuously because they maintained system operation while critical data streams recovered within five seconds.

The testing of IoT devices needs certain methods to handle distinct challenges. Inadequate testing procedures often lead to permanent damage to devices and create safety hazards. The placement of multiple devices in inaccessible locations complicates physical maintenance operations. An uncontrolled chaos test in a smart home system could lead to a failed smart lock that prevents homeowners from accessing their property. The restart of glucose monitors during measurement operations can create dangerous situations for patients who use these devices. Industrial networks can shut down completely when they encounter any level of system disruption.

The implementation of chaos engineering for IoT systems requires both strategic planning and innovative solutions. Engineers should perform system vulnerability tests, which ensure operational safety and reliability for real world deployment. The risk assessment process needs tested and accurate methods to protect both system devices and their users from harm.

Table 2: Chaos engineering overview

Lack of open source tools for chaos engineering in IoT systems

Over time, many open source tools have been created to make chaos engineering easier in cloud systems. These tools work well for cloud-based servers and containers but do not work as effectively on Internet of Things (IoT) devices, such as sensors, smart cameras and small robots. This is because most IoT devices do not use Kubernetes or Docker systems. Their operating systems are small or run directly on chip-level firmware rather than standard systems.

Most IoT devices are powered by microcontrollers with only a few megabytes of memory. This memory is much smaller than even what basic smartphones possess, which makes it hard to run chaos experiments as they require a lot of space to simulate failures.

However, IoT system testing methods are evolving through research activities. The use of Raspberry Pi clusters is an economical solution for testing. Engineers use these systems to duplicate actual IoT systems, which allows them to monitor device performance under various operational scenarios. They perform tests that duplicate network breakdowns, device shutdowns and sensor failures to study system responses.

Through GitHub, users can access small projects that demonstrate Arduino board hardware fault testing. The experiments enable developers to create IoT systems that maintain operational reliability through fault occurrences.

Currently, there is no open source framework specifically for chaos engineering in IoT devices. Without such ‘chaos agents’ on microcontrollers, it is difficult to run controlled system failures, create sensor errors and then automatically recover the system. A framework that includes pre-built test protocols for different IoT platforms would make experiments much easier. A specialised toolkit for edge devices would allow safe testing without damaging actual hardware.

The development of this toolkit would introduce substantial improvements to the current field of operations. A specific chaos engineering system for IoT devices would transform edge device development and maintenance practices just as Kubernetes transformed cloud infrastructure management. The framework would improve the operational safety and reliability of smart home equipment and industrial sensors, and enable them to operate effectively under actual operational conditions.

Challenges, risks and ethical questions

While chaos engineering helps improve IoT system reliability, it also comes with security risks. In addition, it raises multiple ethical concerns. Unlike cloud servers, IoT devices can be permanently damaged if something goes wrong. For example, a faulty firmware update or a poorly timed test can make a device completely unusable.

Real-world events demonstrate the high level of danger involved. The 2021 software update for Tesla vehicles resulted in short-term failures of the autopilot system. The system failure generated safety hazards for drivers operating their vehicles. The operation of medical devices including insulin pumps and pacemakers depends on dependable firmware systems. Software failures in medical devices could threaten thousands of patients throughout the world.

Safety is one of the most important concerns in IoT chaos engineering. Testing connected cars, medical implants and industrial robots can be dangerous to humans. Running these tests in real production environments is usually restricted. Even simulated tests can create potential security risks if not managed carefully.

The IoT ecosystem is also very complex, which adds to the difficulty. IoT devices use a wide range of hardware, operating systems and communication protocols. This variety makes it hard to design chaos tools that work across all systems. The lack of a single unified platform means experiments are more difficult. Each fragmented system increases the chance of unforeseen failures.

Personal data protection remains another important concern. The execution of chaos experiments results in accidental exposure of confidential information. For example, the testing of faulty smart home cameras and health monitors may lead to privacy breaches as these devices store private video footage and individual health information. The identification of legal and ethical obligations for these system failures becomes extremely challenging. The responsibility for these failures lies with engineers who run tests, the companies that operate devices, and the developers who built testing tools.

Organisations need to maintain ethical standards when they use chaos engineering to safeguard their IoT systems. Engineers who want to perform IoT chaos testing need to follow established safety protocols. The IEEE’s ‘Ethically Aligned Design’ for autonomous systems together with the European Union’s AI Act serve as guidelines for conducting safe experimental work. Researchers can perform device testing without unacceptable risks by using safety protocols and rollback mechanisms, and controlled testing environments that combine these frameworks.

The implementation of chaos engineering in IoT systems demands detailed planning, robust safety protocols and ethical supervision to succeed.

Table 3: Open source chaos engineering tools and IoT challenges

Towards a resilient IoT future

Open source communities are actively working to enhance IoT device dependability through their contributions of testing tools, platforms and research projects. An open chaos toolkit for IoT (e.g., software to simulate failures on smart sensors, wearables or industrial controllers) would enable basic resilience tests on devices. The system could operate through a structure that mirrors Linux operating system accessibility (common OS for IoT gateways and edge devices) and Kubernetes cloud system management (used to manage IoT services and applications at scale).

Multiple initiatives are currently trying to achieve this goal. The EdgeX Foundry (framework for connecting IoT devices from different vendors) and Eclipse IoT platforms (open source projects for managing IoT applications and devices) serve as frameworks that enable different devices to operate harmoniously together. Through GitHub, developers can share and test microcontrollers and embedded systems by running chaos simulation experiments. The built-in testing capabilities of Fledge (industrial IoT data platform for edge devices) and Zephyr RTOS (real-time operating system used in small IoT devices) platforms demonstrate successful methods to enhance edge device reliability. The development of IoT system reliability across device networks can improve through collaborative development approaches used by the community.

The tools need benchmarks and standards support to operate correctly. The IoT device certification process evaluates how devices operate while also testing their ability to function through disruptions. The certification process for smart sensors (such as temperature or motion sensors) requires them to maintain data accuracy at 30% network packet loss rates. A wearable device needs to restore its full operational capabilities within a five-second time frame after the system restarts. Users can verify device performance through such standardised benchmarks.

Teaching people about chaos engineering is just as important as developing tools. Most IoT developers focus on adding new features rather than preparing for failures. Open source initiatives, along with tutorials, workshops and hackathons, help train developers to run safe reliability tests and evaluate their devices’ performance under stress.