A robot’s deployment environment often involves perceptual changes that differ from what it has experienced during training. Standard practices such as data augmentation attempt to bridge this gap by augmenting source images in an effort to extend the support of the training distribution to better cover what the agent might experience at test time. In many cases, however, it is impossible to know test-time distribution- shift a priori, making these schemes infeasible. In this paper, we introduce a general approach, called Invariance through Latent Alignment (ILA), that improves the test-time performance of a visuomotor control policy in deployment environments with unknown perceptual variations. ILA performs unsupervised adaptation at deployment-time by matching the distribution of latent features on the target domain to the agent’s prior experience, without relying on paired data. Although simple, we show that this idea leads to surprising improvements on a variety of challenging adaptation scenarios, including changes in lighting conditions, the content in the scene, and camera poses. We present results on calibrated control benchmarks in simulation—the distractor control suite—and a physical robot under a sim-to-real setup. Video and code available at: https:
//invariance-through-latent-alignment.github.io
more »
« less
Robust Finger Interactions with COTS Smartwatches via Unsupervised Siamese Adaptation
Wearable devices like smartwatches and smart wristbands have gained substantial popularity in recent years. However, their small interfaces create inconvenience and limit computing functionality. To fill this gap, we propose ViWatch, which enables robust finger interactions under deployment variations, and relies on a single IMU sensor that is ubiquitous in COTS smartwatches. To this end, we design an unsupervised Siamese adversarial learning method. We built a real-time system on commodity smartwatches and tested it with over one hundred volunteers. Results show that the system accuracy is about 97% over a week. In addition, it is resistant to deployment variations such as different hand shapes, finger activity strengths, and smartwatch positions on the wrist. We also developed a number of mobile applications using our interactive system and conducted a user study where all participants preferred our unsupervised approach to supervised calibration. The demonstration of ViWatch is shown at https://youtu.be/N5-ggvy2qfI.
more »
« less
- Award ID(s):
- 1822935
- PAR ID:
- 10494111
- Publisher / Repository:
- ACM
- Date Published:
- Journal Name:
- Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
- ISBN:
- 9798400701320
- Page Range / eLocation ID:
- 1 to 14
- Format(s):
- Medium: X
- Location:
- San Francisco CA USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
BY-kinases constitute a protein tyrosine kinase family that encodes unique catalytic domains that deviate from those of eukaryotic kinases resembling P-loop nucleotide triphosphatases (NTPases) instead. We have used computational and supporting biochemical approaches using the catalytic domain of the Escherichia coli BY-kinase, Wzc, to illustrate mechanistic divergences between BY-kinases and NTPases despite their deployment of similar catalytic motifs. In NTPases, the “arginine finger” drives the reactive conformation of ATP while also displacing its solvation shell, thereby making favorable enthalpic and entropic contributions toward βγ-bond cleavage. In BY-kinases, the reactive state of ATP is enabled by ATP·Mg 2+ -induced global conformational transitions coupled to the conformation of the Walker-A lysine. While the BY-kinase arginine finger does promote the desolvation of ATP, it does so indirectly by generating an ordered active site in combination with other structural elements. Bacteria, using these mechanistic variations, have thus repurposed an ancient fold to phosphorylate on tyrosine.more » « less
-
null (Ed.)Gesture recognition has become increasingly important in human-computer interaction and can support different applications such as smart home, VR, and gaming. Traditional approaches usually rely on dedicated sensors that are worn by the user or cameras that require line of sight. In this paper, we present fine-grained finger gesture recognition by using commodity WiFi without requiring user to wear any sensors. Our system takes advantages of the fine-grained Channel State Information available from commodity WiFi devices and the prevalence of WiFi network infrastructures. It senses and identifies subtle movements of finger gestures by examining the unique patterns exhibited in the detailed CSI. We devise environmental noise removal mechanism to mitigate the effect of signal dynamic due to the environment changes. Moreover, we propose to capture the intrinsic gesture behavior to deal with individual diversity and gesture inconsistency. Lastly, we utilize multiple WiFi links and larger bandwidth at 5GHz to achieve finger gesture recognition under multi-user scenario. Our experimental evaluation in different environments demonstrates that our system can achieve over 90% recognition accuracy and is robust to both environment changes and individual diversity. Results also show that our system can provide accurate gesture recognition under different scenarios.more » « less
-
Finger-Fitts law [6] is a variant of Fitts’ law which accounts for the finger ambiguity in touch pointing. In this paper we investigated two research questions related to Finger-Fitts law: (1) Should Finger-Fitts law use nominal target width W or effect target width We to model MT? and (2) should Finger-Fitts law use a pre-defined value (denoted by σa) or a free parameter (denoted by c) to represent the absolute ambiguity caused by finger touch? Our investigation on two touch pointing datasets showed that there are cases where using nominal width has stronger model fitness, and also cases where using effective width is better. Regarding the representation of finger ambiguity, using a free parameter c to represent the ambiguity of finger touch always leads to stronger model fitness than using the pre-defined σa, after controlling for overfitting. It indicates that viewing the finger ambiguity as an empirically determined parameter has more flexibility to capture the ambiguity of finger touch involved in the study. Overall, our research advances the understanding on how to model Finger touch input with Finger-Fitts law.more » « less
-
null (Ed.)Abstract Refraction networking is a next-generation censorship circumvention approach that locates proxy functionality in the network itself, at participating ISPs or other network operators. Following years of research and development and a brief pilot, we established the world’s first production deployment of a Refraction Networking system. Our deployment uses a highperformance implementation of the TapDance protocol and is enabled as a transport in the popular circumvention app Psiphon. It uses TapDance stations at four physical uplink locations of a mid-sized ISP, Merit Network, with an aggregate bandwidth of 140 Gbps. By the end of 2019, our system was enabled as a transport option in 559,000 installations of Psiphon, and it served upwards of 33,000 unique users per month. This paper reports on our experience building the deployment and operating it for the first year. We describe how we overcame engineering challenges, present detailed performance metrics, and analyze how our system has responded to dynamic censor behavior. Finally, we review lessons learned from operating this unique artifact and discuss prospects for further scaling Refraction Networking to meet the needs of censored users.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3586183.3606794
