skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Doing Remote Controlled Studies with Humans: Tales from the COVID Trenches
How should empirical researchers conduct controlled, remote “lab” studies in the uncontrolled, noisy conditions of each participant's own home? Volatility in participant home environments, hardware, internet connection, and surrounding distractions takes the “controlled” out of controlled studies. This paper recounts our in-the-trenches mitigations for designing and conducting two complex controlled studies under COVID, in which participants, from home, interactively localized faults in an AI system. The studies with our COVID-era mitigations in 5 categories-Privacy/Security, Data Collection, Control, Technology Issues, Payment-ultimately produced crisp results beyond what we thought possible under such uncontrolled circumstances.  more » « less
Award ID(s):
2042324 1901031
PAR ID:
10310087
Author(s) / Creator(s):
; ; ; ; ; ; ; ; ; ; ; ;
Date Published:
Journal Name:
IEEE/ACM 13th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    In the present article, we follow up our recent work on the experimental assessment of two data-driven nonlinear system identification methodologies. The first methodology constructs a single nonlinear-mode model from periodic vibration data obtained under phase-controlled harmonic excitation. The second methodology constructs a state-space model with polynomial nonlinear terms from vibration data obtained under uncontrolled broadband random excitation. The conclusions drawn from our previous work (experimental) were limited by uncertainties inherent to the specimen, instrumentation, and signal processing. To avoid these uncertainties in the present work, we pursued a completely numerical approach based on synthetic measurement data obtained from simulated experiments. Three benchmarks are considered, which feature geometric, unilateral contact, and dry friction nonlinearity, respectively. As in our previous work, we assessed the prediction accuracy of the identified models with a focus on the regime near a particular resonance. This way, we confirmed our findings on the strengths and weaknesses of the two methodologies and derive several new findings: First, the state-space method struggles even for polynomial nonlinearities if the training data is chaotic. Second, the polynomial state-space models can reach high accuracy only in a rather limited range of vibration levels for systems with non-polynomial nonlinearities. Such cases demonstrate the sensitivity to training data inherent in the method, as model errors are inevitable here. Third, although the excitation does not perfectly isolate the nonlinear mode (exciter-structure interaction, uncontrolled higher harmonics, local instead of distributed excitation), the modal properties are identified with high accuracy. 
    more » « less
  2. In this paper, we present the methods, challenges, and lessons from conducting a moderated, remote, at-home study of an Augmented Reality (AR) application that overlays omic information in users’ kitchens. Due to the COVID-19 pandemic, our team adapted to remote studies, which have presented unique experiences and discussions. We explore ways that could lower barriers for researchers to conduct remote Mixed Reality (MR) studies and assume greater control over a remote study. We argue that remote studies conducted in study participants’ personal spaces can lead to more insightful and nuanced results, but participants’ privacy and issues related to equity should be considered and protected. 
    more » « less
  3. Abstract During the Coronavirus Disease 2019 (COVID-19) epidemic, many health professionals used social media to promote preventative health behaviors. We conducted a randomized controlled trial of the effect of a Facebook advertising campaign consisting of short videos recorded by doctors and nurses to encourage users to stay at home for the Thanksgiving and Christmas holidays ( NCT04644328 and AEARCTR-0006821 ). We randomly assigned counties to high intensity ( n  = 410 (386) at Thanksgiving (Christmas)) or low intensity ( n  = 410 (381)). The intervention was delivered to a large fraction of Facebook subscribers in 75% and 25% of randomly assigned zip codes in high- and low-intensity counties, respectively. In total, 6,998 (6,716) zip codes were included, and 11,954,109 (23,302,290) users were reached at Thanksgiving (Christmas). The first two primary outcomes were holiday travel and fraction leaving home, both measured using mobile phone location data of Facebook users. Average distance traveled in high-intensity counties decreased by −0.993 percentage points (95% confidence interval (CI): –1.616, −0.371; P = 0.002) for the 3 days before each holiday compared to low-intensity counties. The fraction of people who left home on the holiday was not significantly affected (adjusted difference: 0.030; 95% CI: −0.361, 0.420; P = 0.881). The third primary outcome was COVID-19 infections recorded at the zip code level in the 2-week period starting 5 days after the holiday. Infections declined by 3.5% (adjusted 95% CI: −6.2%, −0.7%; P = 0.013) in intervention compared to control zip codes. Social media messages recorded by health professionals before the winter holidays in the United States led to a significant reduction in holiday travel and subsequent COVID-19 infections. 
    more » « less
  4. Retrofitting urban watersheds with wireless sensing and control technologies will enable the next generation of autonomous water systems. While many studies have highlighted the benefits of real-time controlled gray infrastructure, few have evaluated real-time controlled green infrastructure. Motivated by a controlled bioretention site where phosphorus is a major runoff pollutant, phosphorus removal was simulated over a range of influent concentrations and storm conditions for three scenarios: a passive, uncontrolled bioretention cell (baseline), a real-time controlled cell (autonomous upgrade), and a cell with soil amendments (passive upgrade). Results suggest the autonomous upgrade matched the pollutant treatment performance of the baseline scenario in half the spatial footprint. The autonomous upgrade also matched the performance of the passive upgrade; suggesting real-time control may provide a ‘digital’ alternative to existing, passive upgrades. These findings may help site- and cost-constrained stormwater managers meet their water quality goals. 
    more » « less
  5. null (Ed.)
    This paper proposes and analyzes a stochastic Susceptible-Exposed-Infected-Removed (SEIR) spreading model on networks. Imagine a nursing home housing 28 seniors and 7 staff workers, in which one of the staff has tested positive for COVID-19. Unfortunately, the results of this test are 3 days late and the infected person had not been quarantining while waiting for their test results. What is now the individual risk to the different people living in this nursing home? If the home has access to two rapid COVID-19 viral tests, who should they be given to and why? In order to answer questions like this, we need to study stochastic models rather than deterministic ones. Unlike the vast majority of works that analyze various deterministic models, stochastic models are required when analyzing the risk of COVID-19 to individual people rather than tracking aggregate numbers in a given region. More specifically, this paper compares the results provided by analyzing stochastic and deterministic models and investigating when it is suitable to use the different models. In particular, we show why it is not suitable to use deterministic models when analyzing the spread in small communities and how these questions can be better addressed using stochastic ones. Finally, we show the added complications that arise due to the relatively long incubation period of COVID-19, and how it can be addressed. A simulated case study of the spread of COVID-19 in a 35-person nursing home is used to help illustrate our results. 
    more » « less