Search for: All records

In this paper, we study the gradient descent dynamics for jointly training both layers of a one- hidden-layer ReLU network to fit a linear target function. Concretely, we consider a realizable setting where inputs are drawn i.i.d. from a Gaussian distribution and labels follow a planted linear model. This stylized framework captures salient features of end-to-end training in inverse prob- lems and certain auto-encoder models. Despite its apparent simplicity, the dynamics remain poorly understood, in part because the loss landscape contains multiple non-strict saddle points, making it unclear why gradient descent from random initialization reliably escapes bad stationary regions. We provide a detailed characterization of the optimization landscape and prove that gradient de- scent from a moderately small random initialization-simultaneously training both layers-converges to a global minimizer at a linear rate with order-wise optimal sample complexity. Our analysis tracks the trajectory through three phases: an alignment phase in which hidden weights progres- sively align with the planted direction while the output weights maintain the correct sign pattern; a growth phase in which the norms of both layers increase while preserving alignment; and a local re- finement phase in which the aligned neurons rapidly converge to the planted direction, yielding fast local convergence. To rigorously show that GD avoids non-strict saddles, we develop trajectory- level control arguments for the end-to-end dynamics. In addition, we establish novel uniform con- centration results that hold along the entire trajectory, and are essential for obtaining order-wise optimal sample complexity. We corroborate our theory with extensive experiments across a range of configurations. 

https://www.nsta.org/connected-science-learning/connected-science-learning-march-april-2022/data-driven-science-vlogging 

Abstract Design thinking is essential to the success of a design process as it helps achieve the design goal by guiding design decision-making. Therefore, fundamentally understanding design thinking is vital for improving design methods, tools and theories. However, interpreting design thinking is challenging because it is a cognitive process that is hidden and intangible. In this paper, we represent design thinking as an intermediate layer between human designers’ thought processes and their design behaviors. To do so, this paper first identifies five design behaviors based on the current design theories. These behaviors include design action preference, one-step sequential behavior, contextual behavior, long-term sequential behavior, and reflective thinking behavior. Next, we develop computational methods to characterize each of the design behaviors. Particularly, we use design action distribution, first-order Markov chain, Doc2Vec, bi-directional LSTM autoencoder, and time gap distribution to characterize the five design behaviors. The characterization of the design behaviors through embedding techniques is essentially a latent representation of the design thinking, and we refer to it as design embeddings. After obtaining the embedding, an X-mean clustering algorithm is adopted to each of the embeddings to cluster designers. The approach is applied to data collected from a high school solar system design challenge. The clustering results show that designers follow several design patterns according to the corresponding behavior, which corroborates the effectiveness of using design embedding for design behavior clustering. The extraction of design embedding based on the proposed approach can be useful in other design research, such as inferring design decisions, predicting design performance, and identifying design actions identification. 

Engineering design is often used to teach science, but not-yet leads to solid learning gains. We examined the relationship between science learning and engineering design using text mining. Association rule mining was applied to texts written during design to extract the relationships between solar-energy concepts and solar design performance. Findings suggest that students test concept-related factors’ effects on design outcomes to learn concepts and eliminate misconceptions. These findings have implications for future instructional design.