Research Area 1: Scene-Graph Representations of Road Scenes for Autonomous Vehicle Safety
Autonomous vehicles (AVs) are expected to revolutionize transportation by reducing traffic and improving road safety. In 2019 alone, over 36,000 traffic fatalities occurred in the United States (US). Many AV methodologies using deep convolutional (CNN) and temporal (LSTM) networks and have been proposed; however, these methods are limited in their understanding of road scenes due to their inability to explicitly capture structural and semantic information, such as the relationships between road participants. To address this limitation, we propose using scene-graphs to represent road scenes, with nodes representing road objects and edges representing the relations between objects. By capturing the relationships between vehicles, pedestrians, and other entities explicitly, our methodology outperforms state of the art CNN+LSTM and Convolutional LSTM methods at subjective risk assessment and collision prediction. Additionally, our methodology is better able to transfer knowledge gained from synthetic datasets to real-world driving datasets. We also show that our methodology is faster and more energy efficient than the state-of-the-art methods.
Research Area 2: Digital Twin for Manufacturing Systems
A digital twin is the virtual representation of a physical system (its physical twin). The concept of the digital twin was first used by NASA to describe a digital replica of physical systems in space maintained for diagnosis and prognosis. Digital twin consists of large historical context and performance data and utilizes the direct (through inbuilt sensors) and indirect (through latent variable analysis) sensing to provide the near real-time representation of the physical system. Moreover, it consists of various models (for simulation, monitoring, control, optimization, etc.,) in a hierarchical manner (consisting of a representation of the system, process, component, etc.,) which can provide the blueprint of the whole system. Our research works involve a wide range of applications of the Digital Twin including Manufacturing, Autonomous Vehicles, Hardware Security, and IoT Security.
In one of our works, we contribute with an Internet-of-Things (IoT) based methodology to build digital twins using an indirect medium, such as side-channels, which can localize anomalous faults and infer the quality of the products being manufactured while keeping itself up-to-date. We achieve this by exploring and utilizing the side-channels emissions, such as acoustics, power, magnetic, of the system that unintentionally reveals the cyber and physical state of the system. This is a new project started in our lab from the beginning of 2018. Our first paper is published at the ACM/IEEE International Conference on Internet-of-Things Design and Implementation (IoTDI’19).
- A short video presentation on our current work on QUILT: Quality Inference from Living Digital Twins in IoT-Enabled Manufacturing Systems is given below:
Research Area 3: Machine Learning on Graphs and its Applications
Graph-structured data is becoming more and more abundant in several application contexts such as social networks, protein structures, or even software program analysis. To address these non-Euclidean graphs, it requires specific machine learning methods, well-known as graph embedding approaches, to first represent the data on the euclidean space that preserves the structural information of the graphs. Graph Embeddings are the transformation of property graphs to a vector or a set of vectors. Embedding should capture the graph topology, vertex-to-vertex relationship, and other relevant information about graphs, subgraphs, and vertices.
Our first work was Structural Graph Convolutional Neural Networks (SGCNN) for Engineering which focused on modeling engineering data as knowledge graphs. As shown in Figure 1, SGCNN presents a graph learning algorithm to classify engineering data artifacts – represented in the form of graphs – according to their structure and neighborhood features and is proven to be successful in classifying engineer artifact’s functions with a 91% precision. This work was published in the ACM/IEEE International Conference on Computer-Aided Design (ICCAD’18), San Diego, CA, November 2018.
Following this, AICPS has participated in two DARPA projects, Deep Code Curator (DCC) and Assured MicroPatching (AMP) projects. The DCC focused on representing a paper related to deep learning research by considering all the modalities (text, image, code). Also, it aims at detecting the similarities between publications using the graphical representations of the code. On the other hand, the AMP project will start in Aug. 2020 and will focus on automatic software repairing on the legacy software binaries that incorporate graph transformation techniques.
In addition to these two projects, we are also working on knowledge graph modeling and graph learning applications in the field of autonomous vehicle and hardware security. As to knowledge graph embedding, we have also released a knowledge graph embedding library called pykg2vec on GitHub, while the library paper is currently under the review in the Journal of Machine Learning.
Research Area 4: Neuroscience-Inspired Algorithms for Predictive Maintenance
In manufacturing, one of the greatest expenses is machine downtime due to component failures. To try and reduce downtime, we investigate methods of detecting early symptoms of failure so that repairs can be scheduled efficiently. This paradigm is known as predictive maintenance.
Many methods for predictive maintenance have been proposed using statistical methods, classical ML techniques, and deep learning. However, we are investigating the utility of an algorithm inspired by the structure of the human neocortex, called Hierarchical Temporal Memory (HTM), originally proposed by Hawkins and Blakeslee in On Intelligence. The structure of HTMs draws heavily upon neuroscience in an attempt to emulate the human brain’s ability to identify both short and long patterns efficiently, filter noise, learn continuously, and adapt to changing conditions over time. These characteristics make HTM an ideal candidate for anomaly detection and predictive maintenance tasks.
We have applied HTMs to several manufacturing applications including anomaly detection in roller-element bearing vibration data and 3D printer vibration data. Our results show that HTMs are highly noise resistant, application-agnostic, and able to outperform existing anomaly detection methods. Following is the neocortical architecture of the HTM used in the predictive maintenance:
Research Area 5: Neuroscience-Inspired Algorithms for Power Grid Anomaly Detection and State Estimation
In contrast to the one-directional fossil-fuel-powered power grid of the past, the modern power grid is much more dynamic and heterogenous. This so called “Smart Grid” must support EV charging, increased demand, consumer-grade solar, distributed power distribution, variable supply from renewables, etc. All of these new technologies severely challenge existing grid infrastructure and require advanced load-balancing, synchronization, and management techniques. Phasor measurement units (PMUs) are used to measure the frequency of the power grid and enable corrections by grid operators to ensure grid stability. To avoid grid collapse and blackouts, anomalies in the power grid must be detected quickly, necessitating good anomaly detection methods. Additionally, advanced load-prediction algorithms can help grid operators determine when to start up or shut down power generation facilities to reduce wastage.
To tackle these problems, we leverage Hierarchical Temporal Memory (HTM) as mentioned in Research Area 3. HTM is especially well suited for anomaly detection in live PMU data due to its noise resilience, continuous learning capability, and efficiency. We found that HTMs demonstrated equivalent or better performance than many state-of-the-art methods at this task. Additionally, we successfully demonstrated the ability of HTMs to predict future grid loads from past observations. The following figure is the proposed architecture for anomaly detection and simultaneous data prediction using the PMU data:
Research Area 6: Neuroscience-Inspired Algorithms for Hardware Trojan Detection
Since 2007, the use of side-channel measurements for detecting hardware Trojan (HT) has been extensively studied. However, the majority of works either rely on a golden chip, or they rely on methods that are not robust against subtle acceptable changes that would occur over the life-cycle of an integrated circuit (IC). In this paper, we propose using a brain-inspired architecture called Hierarchical Temporal Memory (HTM) for HT detection. Similar to the human brain, our proposed solution is resilient against natural changes that might happen in the side-channel measurements while being able to accurately detect abnormal behavior of the chip when the HT gets triggered. The following figure is the test bed for power side-channel data collection for hardware trojan detection:
