Title: Distributed Virtual Time-Based Synchronization for Simulation of Cyber-Physical Systems
Our world today increasingly relies on the orchestration of digital and physical systems to ensure the successful operations of many complex and critical infrastructures. Simulation-based testbeds are useful tools for engineering those cyber-physical systems and evaluating their efficiency, security, and resilience. In this article, we present a cyber-physical system testing platform combining distributed physical computing and networking hardware and simulation models. A core component is the distributed virtual time system that enables the efficient synchronization of virtual clocks among distributed embedded Linux devices. Virtual clocks also enable high-fidelity experimentation by interrupting real and emulated cyber-physical applications to inject offline simulation data. We design and implement two modes of the distributed virtual time: periodic mode for scheduling repetitive events like sensor device measurements, and dynamic mode for on-demand interrupt-based synchronization. We also analyze the performance of both approaches to synchronization including overhead, accuracy, and error introduced from each approach. By interconnecting the embedded devices’ general purpose IO pins, they can coordinate and synchronize with low overhead, under 50 microseconds for eight processes across four embedded Linux devices. Finally, we demonstrate the usability of our testbed and the differences between both approaches in a power grid control application.  more » « less
Award ID(s):
1730488
PAR ID:
10298991
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
ACM Transactions on Modeling and Computer Simulation
Volume:
31
Issue:
2
ISSN:
1049-3301
Page Range / eLocation ID:
1 to 24
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Modern embedded and cyber-physical systems are ubiquitous. Many critical cyber-physical systems have real-time requirements (e.g., avionics, automobiles, power grids, manufacturing systems, industrial control systems, etc.). Recent developments and new functionality require real-time embedded devices to be connected to the Internet. This gives rise to the real-time Internet-of-things (RT-IoT) that promises a better user experience through stronger connectivity and efficient use of next-generation embedded devices. However, RT-IoT are also increasingly becoming targets for cyber-attacks, which is exacerbated by this increased connectivity. This paper gives an introduction to RT-IoT systems, an outlook of current approaches and possible research challenges towards secure RT-IoT frameworks. 
    more » « less
  2. Reconnaissance is critical for adversaries to prepare attacks causing physical damage in industrial control systems (ICS) like smart power grids. Disrupting reconnaissance is challenging. The state-of-the-art moving target defense (MTD) techniques based on mimicking and simulating system behaviors do not consider the physical infrastructure of power grids and can be easily identified. To overcome these challenges, we propose physical function virtualization (PFV) that “hooks” network interactions with real physical devices and uses these real devices to build lightweight virtual nodes that follow the actual implementation of network stacks, system invariants, and physical state variations in the real devices. On top of PFV, we propose DefRec, a defense mechanism that significantly increases the effort required for an adversary to infer the knowledge of power grids’ cyber-physical infrastructures. By randomizing communications and crafting decoy data for virtual nodes, DefRec can mislead adversaries into designing damage-free attacks. We implement PFV and DefRec in the ONOS network operating system and evaluate them in a cyber-physical testbed, using real devices from different vendors and HP physical switches to simulate six power grids. The experimental results show that with negligible overhead, PFV can accurately follow the behavior of real devices. DefRec can delay adversaries’ reconnaissance for more than 100 years by adding a number of virtual nodes less than or equal to 20% of the number of real devices. 
    more » « less
  3. Distributed systems and blockchain consensus protocols rely on a clock synchronization algorithm to resolve any conflicts related to the state of data or resources. Faulty and adversarial peers make it challenging to achieve clock synchronization in a peer-to-peer distributed system. Clock-synchronization protocols such as the Network Time Protocol (NTP) and the Precision Time Protocol (PTP) are effective but only in a partially or fully trusted network. They fail to synchronize clocks in the presence of misbehaving peers who try to rewind or fast-forward time or introduce deadlocks purposely.This paper introduces BFT-Metronome, a Byzantine fault-tolerant clock synchronization algorithm for peer-to-peer distributed systems. The novel BFT-Metronome algorithm combines a sensor-fusion algorithm with a statistical outlier detection algorithm to enable participating peers to determine the most likely skew based on broadcast timestamps. The algorithm is designed to withstand up to N3−1 misbehaving participants in a network of N total participants, and the independently calculated offset is observed to be within 10 ms of precision vs. 40-100 ms in peer-to-peer distributed systems.The BFT-Metronome clock synchronization algorithm can also be applied to blockchain consensus protocols (such as Proof-of-Stake) that rely on weakly synchronized clocks. This is shown by integrating BFT-Metronome with the Lightweight Mining Algorithm to achieve consensus-round synchronization and timeouts. Then, generalization to other protocols is argued qualitatively. It can also be applied to large distributed systems to secure against insider threats during clock synchronization. 
    more » « less
  4. Reconnaissance is critical for adversaries to prepare attacks causing physical damage in industrial control systems (ICS) like smart power grids. Disrupting the reconnaissance is challenging. The state-of-the-art moving target defense (MTD) techniques based on mimicking and simulating system behaviors do not consider the physical infrastructure of power grids and can be easily identified. To overcome those challenges, we propose physical function virtualization (PFV) that ``hooks'' network interactions with real physical devices and uses them to build lightweight virtual nodes following the actual implementation of network stacks, system invariants, and physical state variations of real devices. On top of PFV, we propose DefRec, a defense mechanism that significantly increases the reconnaissance efforts for adversaries to obtain the knowledge of power grids' cyber-physical infrastructures. By randomizing communications and crafting decoy data for the virtual physical nodes, DefRec can mislead adversaries into designing damage-free attacks. We implement PFV and DefRec in the ONOS network operating system and evaluate them in a cyber-physical testbed, which uses real devices from different vendors and HP physical switches to simulate six power grids. The experiment results show that with negligible overhead, PFV can accurately follow the behavior of real devices. DefRec can significantly delay passive attacks for at least five months and isolate proactive attacks with less than $$10^{-30}$$ false negatives. 
    more » « less
  5. Several cyber-physical systems use real-time restart-based embedded systems with the Simplex architecture to provide safety guarantees against system faults. Some approaches have been developed to protect such systems from security violations too, but none of these approaches can prevent an adversary from modifying the operating system or application code to execute an attack that persists even after a reboot. In this work, we present a secure boot mechanism to restore real-time restart-based embedded systems into a secure computing environment after every restart. We analyze the delay introduced by the proposed security feature and present preliminary results to demonstrate the viability of our approach using an open-source bootloader and real-time operating system. 
    more » « less