More Like this

1. Enhancing Collective Estimates by Aggregating Cardinal and Ordinal Inputs

   Ryan Kemmer, Yeawon Yoo (October 2020, Proceedings of the Eigth AAAI Conference on Human Computation and Crowdsourcing)

   There are many factors that affect the quality of data received from crowdsourcing, including cognitive biases, varying levels of expertise, and varying subjective scales. This work investigates how the elicitation and integration of multiple modalities of input can enhance the quality of collective estimations. We create a crowdsourced experiment where participants are asked to estimate the number of dots within images in two ways: ordinal (ranking) and cardinal (numerical) estimates. We run our study with 300 participants and test how the efficiency of crowdsourced computation is affected when asking participants to provide ordinal and/or cardinal inputs and how the accuracy of the aggregated outcome is affected when using a variety of aggregation methods. First, we find that more accurate ordinal and cardinal estimations can be achieved by prompting participants to provide both cardinal and ordinal information. Second, we present how accurate collective numerical estimates can be achieved with significantly fewer people when aggregating individual preferences using optimization-based consensus aggregation models. Interestingly, we also find that aggregating cardinal information may yield more accurate ordinal estimates. 

   more » « less
2. Elicitation and aggregation of multimodal estimates improve wisdom of crowd effects on ordering tasks

   https://doi.org/10.1038/s41598-024-52176-3  

   Yoo, Yeawon; Escobedo, Adolfo R.; Kemmer, Ryan; Chiou, Erin (February 2024, Scientific Reports)

   Abstract We present a wisdom of crowds study where participants are asked to order a small set of images based on the number of dots they contain and then to guess the respective number of dots in each image. We test two input elicitation interfaces—one elicits the two modalities of estimates jointly and the other independently. We show that the latter interface yields higher quality estimates, even though the multimodal estimates tend to be more self-contradictory. The inputs are aggregated via optimization and voting-rule based methods to estimate the true ordering of a larger universal set of images. We demonstrate that the quality of collective estimates from the simpler yet more computationally-efficient voting methods is comparable to that achieved by the more complex optimization model. Lastly, we find that using multiple modalities of estimates from one group yields better collective estimates compared to mixing numerical estimates from one group with the ordinal estimates from a different group. 

   more » « less
3. Perception of and Response to a Haptic Device as a Function of Signal Complexity

   Alvarez Valdivia, Antonio; Blumenschein, Laura H. (July 2023, IEEE World Haptics Conference)

   Haptics devices have been developed in a wide range of form factors, actuation methods, and degrees of freedom, often with the goal of communicating information. While work has investigated the maximum rate and quantity of information that can be transferred through haptics, these measures often do not inform how humans will use the devices. In this work, we measure the differences between perception and use as it relates to signal complexity. Using an inflatable soft haptic display with four independently actuated pouches, we provide navigation directions to participants. The haptic device operates in three modalities, in increasing order of signal complexity: Cardinal, Ordinal, and Continuous. We first measure participants’ accuracy in perceiving continuous signals generated by the device, showing average errors below 5 deg. Participants then used the haptic device in each operating mode to guide an object towards a target in a 2-dimensional plane. Our results indicate that human’s use of haptic signals often lags significantly behind the displayed signal and is less accurate than their static perception. Additionally signal complexity was correlated with path efficiency but inversely correlated with movement speed, showing that even simple design changes create complex tradeoffs. 

   more » « less
4. Minimax Bounds for Generalized Pairwise Comparisons

   Lee, Kuan-Yun; Courtade, Thomas A (July 2021, 2021 International Conference on Machine Learning (ICML) Workshop on Information-Theoretic Methods for Rigorous, Responsible, and Reliable Machine Learning)

   null (Ed.)

   We introduce the General Pairwise Model (GPM), a general parametric framework for pairwise comparison. Under the umbrella of the exponential family, the GPM unifies many pop- ular models with discrete observations, including the Thurstone (Case V), Berry-Terry-Luce (BTL) and Ordinal Models, along with models with continuous observations, such as the Gaussian Pairwise Cardinal Model. Using information theoretic techniques, we establish minimax lower bounds with tight topological dependence. When applied as a special case to the Ordinal Model, our results uniformly improve upon previously known lower bounds and confirms one direction of a conjecture put forth by Shah et al. (2016). Performance guarantees of the MLE for a broad class of GPMs with subgaussian assumptions are given and compared against our lower bounds, showing that in many natural settings the MLE is optimal up to constants. Matching lower and upper bounds (up to constants) are achieved by the Gaussian Pairwise Cardinal Model, suggesting that our lower bounds are best-possible under the few assumptions we adopt. 

   more » « less
5. Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning

   https://doi.org/10.3389/frai.2022.848056  

   Yasmin, Romena; Hassan, Md Mahmudulla; Grassel, Joshua T.; Bhogaraju, Harika; Escobedo, Adolfo R.; Fuentes, Olac (June 2022, Frontiers in Artificial Intelligence)

   This work investigates how different forms of input elicitation obtained from crowdsourcing can be utilized to improve the quality of inferred labels for image classification tasks, where an image must be labeled as either positive or negative depending on the presence/absence of a specified object. Five types of input elicitation methods are tested: binary classification (positive or negative); the ( x, y )-coordinate of the position participants believe a target object is located; level of confidence in binary response (on a scale from 0 to 100%); what participants believe the majority of the other participants' binary classification is; and participant's perceived difficulty level of the task (on a discrete scale). We design two crowdsourcing studies to test the performance of a variety of input elicitation methods and utilize data from over 300 participants. Various existing voting and machine learning (ML) methods are applied to make the best use of these inputs. In an effort to assess their performance on classification tasks of varying difficulty, a systematic synthetic image generation process is developed. Each generated image combines items from the MPEG-7 Core Experiment CE-Shape-1 Test Set into a single image using multiple parameters (e.g., density, transparency, etc.) and may or may not contain a target object. The difficulty of these images is validated by the performance of an automated image classification method. Experiment results suggest that more accurate results can be achieved with smaller training datasets when both the crowdsourced binary classification labels and the average of the self-reported confidence values in these labels are used as features for the ML classifiers. Moreover, when a relatively larger properly annotated dataset is available, in some cases augmenting these ML algorithms with the results (i.e., probability of outcome) from an automated classifier can achieve even higher performance than what can be obtained by using any one of the individual classifiers. Lastly, supplementary analysis of the collected data demonstrates that other performance metrics of interest, namely reduced false-negative rates, can be prioritized through special modifications of the proposed aggregation methods. 

   more » « less