skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Analysis of Causal and Non-Causal Convolution Networks for Time Series Classification
Applications of neural networks like MLPs and ResNets in temporal data mining has led to improvements on the problem of time series classification. Recently, a new class of networks called Temporal Convolution Networks (TCNs) have been proposed for various time series tasks. Instead of time invariant convolutions they use temporally causal convolutions, this makes them more constrained than ResNets but surprisingly good at generalization. This raises an important question: How does a network with causal convolution solve these tasks when compared to a network with acausal convolutions? As the first attempt at answering these questions, we analyze different architectures through a lens of representational subspace similarity. We demonstrate that the evolution of input representations in the layers of TCNs is markedly different from ResNets and MLPs. We find that acausal networks are prone to form groupings of similar layers and TCNs on the other hand learn representations that are much more diverse throughout the network. Next, we study the convergence properties of internal layers across different architecture families and discover that the behaviour of layers inside Acausal network is more homogeneous when compared to TCNs. Our extensive empirical studies offer new insights into internal mechanisms of convolution networks in the domain of time series analysis and may assist practitioners gaining deeper understanding of each network.  more » « less
Award ID(s):
2112650
PAR ID:
10591677
Author(s) / Creator(s):
; ; ; ;
Publisher / Repository:
Society for Industrial and Applied Mathematics
Date Published:
ISBN:
978-1-61197-803-2
Page Range / eLocation ID:
797 to 805
Subject(s) / Keyword(s):
time series classification, model representation analysis, subspace clustering, transparent AI
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ; ; ; (, International Conference on Computer Aided Verification)
    null (Ed.)
    This paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for up-sampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks. 
    more » « less
  2. ; ; ; ; ; ; (, WSDM'21, The Fourteenth ACM International Conference on Web Search and Data Mining, March 2021)
    Liane Lewin-Eytan, David Carmel (Ed.)
    Graph convolutional networks (GCNs), aiming to obtain node embeddings by integrating high-order neighborhood information through stacked graph convolution layers, have demonstrated great power in many network analysis tasks such as node classification and link prediction. However, a fundamental weakness of GCNs, that is, topological limitations, including over-smoothing and local homophily of topology, limits their ability to represent networks. Existing studies for solving these topological limitations typically focus only on the convolution of features on network topology, which inevitably relies heavily on network structures. Moreover, most networks are text-rich, so it is important to integrate not only document-level information, but also the local text information which is particularly significant while often ignored by the existing methods. To solve these limitations, we propose BiTe-GCN, a novel GCN architecture modeling via bidirectional convolution of topology and features on text-rich networks. Specifically, we first transform the original text-rich network into an augmented bi-typed heterogeneous network, capturing both the global document-level information and the local text-sequence information from texts.We then introduce discriminative convolution mechanisms, which performs convolution on this augmented bi-typed network, realizing the convolutions of topology and features altogether in the same system, and learning different contributions of these two parts (i.e., network part and text part), automatically for the given learning objectives. Extensive experiments on text-rich networks demonstrate that our new architecture outperforms the state-of-the-arts by a breakout improvement. Moreover, this architecture can also be applied to several e-commerce search scenes such as JD searching, and experiments on JD dataset show the superiority of the proposed architecture over the baseline methods. 
    more » « less
  3. ; ; ; ; ; ; ; (, Communications Biology)
    Abstract While it is commonly accepted that ecosystem dynamics are nonlinear, what is often not acknowledged is that nonlinearity implies scale-dependence. With the increasing availability of high-resolution ecological time series, there is a growing need to understand how scale and resolution in the data affect the construction and interpretation of causal networks—specifically, networks mapping how changes in one variable drive changes in others as part of a shared dynamic system (“dynamic causation”). We use Convergent Cross Mapping (CCM), a method specifically designed to measure dynamic causation, to study the effects of varying temporal and taxonomic/functional resolution in data when constructing ecological causal networks. As the system is viewed at different scales relationships will appear and disappear. The relationship between data resolution and interaction presence is not random: the temporal scale at which a relationship is uncovered identifies a biologically relevant scale that drives changes in population abundance. Further, causal relationships between taxonomic aggregates (low-resolution) are shown to be influenced by the number of interactions between their component species (high-resolution). Because no single level of resolution captures all the causal links in a system, a more complete understanding requires multiple levels when constructing causal networks. 
    more » « less
  4. ; ; ; (, CVPR 2018)
    Deep neural networks are increasingly used on mobile devices, where computational resources are limited. In this paper we develop CondenseNet, a novel network architec- ture with unprecedented efficiency. It combines dense con- nectivity between layers with a mechanism to remove un- used connections. The dense connectivity facilitates feature re-use in the network, whereas learned group convolution- s remove connections between layers for which this feature re-use is superfluous. At test time, our model can be imple- mented using standard grouped convolutions—allowing for efficient computation in practice. Our experiments demon- strate that CondenseNets are much more efficient than state- of-the-art compact convolutional networks such as Mo- bileNets and ShuffleNets. 
    more » « less
  5. (, CVPR)
    Deep neural networks are increasingly used on mobile devices, where computational resources are limited. In this paper we develop CondenseNet, a novel network architec- ture with unprecedented efficiency. It combines dense con- nectivity between layers with a mechanism to remove un- used connections. The dense connectivity facilitates feature re-use in the network, whereas learned group convolution- s remove connections between layers for which this feature re-use is superfluous. At test time, our model can be imple- mented using standard grouped convolutions—allowing for efficient computation in practice. Our experiments demon- strate that CondenseNets are much more efficient than state- of-the-art compact convolutional networks such as Mo- bileNets and ShuffleNets. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email