skip to main content

Title: Detecting Temporal Dependencies in Data
Organizations collect data from various sources, and these datasets may have characteristics that are unknown. Selecting the appropriate statistical and machine learning algorithm for data analytical purposes benefits from understanding these characteristics, such as if it contains temporal attributes or not. This paper presents a theoretical basis for automatically determining the presence of temporal data in a dataset given no prior knowledge about its attributes. We use a method to classify an attribute as temporal, non-temporal, or hidden temporal. A hidden (grouping) temporal attribute can only be treated as temporal if its values are categorized in groups. Our method uses a Ljung-Box test for autocorrelation as well as a set of metrics we proposed based on the classification statistics. Our approach detects all temporal and hidden temporal attributes in 15 datasets from various domains.
Authors:
; ; ;
Editors:
Pirk, Holger; Heinis, Thomas
Award ID(s):
1931363
Publication Date:
NSF-PAR ID:
10352321
Journal Name:
Proceedings of the British International Conference on Databases
Page Range or eLocation-ID:
29-39
Sponsoring Org:
National Science Foundation
More Like this
  1. Pirk, Holger ; Heinis, Thomas (Ed.)
    Organizations collect data from various sources, and these datasets may have characteristics that are unknown. Selecting the appropriate statistical and machine learning algorithm for data analytical purposes benefits from understanding these characteristics, such as if it contains temporal attributes or not. This paper presents a theoretical basis for automatically determining the presence of temporal data in a dataset given no prior knowledge about its attributes. We use a method to classify an attribute as temporal, non-temporal, or hidden temporal. A hidden (grouping) temporal attribute can only be treated as temporal if its values are categorized in groups. Our method uses a Ljung-Box test for autocorrelation as well as a set of metrics we proposed based on the classification statistics. Our approach detects all temporal and hidden temporal attributes in 15 datasets from various domains.
  2. Subspace clustering algorithms are used for understanding the cluster structure that explains the patterns prevalent in the dataset well. These methods are extensively used for data-exploration tasks in various areas of Natural Sciences. However, most of these methods fail to handle confounding attributes in the dataset. For datasets where a data sample represent multiple attributes, naively applying any clustering approach can result in undesired output. To this end, we propose a novel framework for jointly removing confounding attributes while learning to cluster data points in individual subspaces. Assuming we have label information about these confounding attributes, we regularize the clustering method by adversarially learning to minimize the mutual information between the data representation and the confounding attribute labels. Our experimental result on synthetic and real-world datasets demonstrate the effectiveness of our approach.
  3. In the past decade, the amount of attributed network data has skyrocketed, and the problem of identifying their underlying group structures has received significant attention. By leveraging both attribute and link information, recent state-of-the-art network clustering methods have achieved significant improvements on relatively clean datasets. However, the noisy nature of real-world attributed networks has long been overlooked, which leads to degraded performance facing missing or inaccurate attributes and links. In this work, we overcome such weaknesses by marrying the strengths of clustering and embedding on attributed networks. Specifically, we propose GRACE (GRAph Clustering with Embedding propagation), to simultaneously learn network representations and identify network clusters in an end-to-end manner. It employs deep denoise autoencoders to generate robust network embeddings from node attributes, propagates the embeddings in the network to capture node interactions, and detects clusters based on the stable state of embedding propagation. To provide more insight, we further analyze GRACE in a theoretical manner and find its underlying connections with two canonical approaches for network modeling. Extensive experiments on six real-world attributed networks demonstrate the superiority of GRACE over various baselines from the state-of-the-art. Remarkably, GRACE improves the averaged performance of the strongest baseline from 0.43 to 0.52, yieldingmore »a 21% relative improvement. Controlled experiments and case studies further verify our intuitions and demonstrate the ability of GRACE to handle noisy information in real-world attributed networks.« less
  4. This work proposes an Adaptive Fuzzy Prediction (AFP) method for the attenuation time series in Commercial Microwave links (CMLs). Time-series forecasting models regularly rely on the assumption that the entire data set follows the same Data Generating Process (DGP). However, the signals in wireless microwave links are severely affected by the varying weather conditions in the channel. Consequently, the attenuation time series might change its characteristics significantly at different periods. We suggest an adaptive framework to better employ the training data by grouping sequences with related temporal patterns to consider the non-stationary nature of the signals. The focus in this work is two-folded. The first is to explore the integration of static data of the CMLs as exogenous variables for the attenuation time series models to adopt diverse link characteristics. This extension allows to include various attenuation datasets obtained from additional CMLs in the training process and dramatically increasing available training data. The second is to develop an adaptive framework for short-term attenuation forecasting by employing an unsupervised fuzzy clustering procedure and supervised learning models. We empirically analyzed our framework for model and data-driven approaches with Recurrent Neural Network (RNN) and Autoregressive Integrated Moving Average (ARIMA) variations. We evaluate themore »proposed extensions on real-world measurements collected from 4G backhaul networks, considering dataset availability and the accuracy for 60 seconds prediction. We show that our framework can significantly improve conventional models’ accuracy and that incorporating data from various CMLs is essential to the AFP framework. The proposed methods have been shown to enhance the forecasting model’s performance by 30 − 40%, depending on the specific model and the data availability.« less
  5. This work proposes an Adaptive Fuzzy Prediction (AFP) method for the attenuation time series in Commercial Microwave links (CMLs). Time-series forecasting models regularly rely on the assumption that the entire data set follows the same Data Generating Process (DGP). However, the signals in wireless microwave links are severely affected by the varying weather conditions in the channel. Consequently, the attenuation time series might change its characteristics significantly at different periods. We suggest an adaptive framework to better employ the training data by grouping sequences with related temporal patterns to consider the non-stationary nature of the signals. The focus in this work is two-folded. The first is to explore the integration of static data of the CMLs as exogenous variables for the attenuation time series models to adopt diverse link characteristics. This extension allows to include various attenuation datasets obtained from additional CMLs in the training process and dramatically increasing available training data. The second is to develop an adaptive framework for short-term attenuation forecasting by employing an unsupervised fuzzy clustering procedure and supervised learning models. We empirically analyzed our framework for model and data-driven approaches with Recurrent Neural Network (RNN) and Autoregressive Integrated Moving Average (ARIMA) variations. We evaluate themore »proposed extensions on real-world measurements collected from 4G backhaul networks, considering dataset availability and the accuracy for 60 seconds prediction. We show that our framework can significantly improve conventional models’ accuracy and that incorporating data from various CMLs is essential to the AFP framework. The proposed methods have been shown to enhance the forecasting model’s performance by 30 − 40%, depending on the specific model and the data availability.« less