skip to main content

Title: Governing crowdsourcing for unconstrained innovation problems
Abstract Research Summary

Theories of crowdsourced search suggest that firms should limit the search space from which solutions to the problem may be drawn by constraining the problem definition. In turn, problems that are not or cannot be constrained should be tackled through other means of innovation. We propose that unconstrained problems can be crowdsourced, but firms need to govern the crowds differently. Specifically, we hypothesize that firms should govern crowds for solving unconstrained problems by instructing them not just to solve the problem but also to help (re)define the problem by offering their problem frames and integrating others' frames. We find evidence for this interaction hypothesis in a field study of over a thousand participants in 20 different crowdsourcing events with interventions for the different governance approaches.

Managerial Summary

Unconstrained innovation problems, which require finding a new product and a new market at the same time, are thought to be difficult to solve via crowdsourcing. We propose and test a governance approach for problem‐finding, that is, (re)defining the firm's original problem statement by instructing crowds to make their problem frames explicit (by posting them) and to integrate others’ problem frames into their solution ideas. In doing so, we provide guidance for firms hoping to use crowdsourcing for both unconstrained and constrained problems. For constrained problems, as widely known, firms should govern for problem‐solving only; for unconstrained problems, they should govern for problem‐finding and problem‐solving. Both forms of governance are “light‐touch,” requiring only minimal intervention in the form of instructions for the crowd at the beginning of the crowdsourcing event.

more » « less
Author(s) / Creator(s):
 ;  ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Strategic Management Journal
Medium: X Size: p. 2783-2817
p. 2783-2817
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Background

    Engineering students inconsistently apply equilibrium when solving problems in statics, but few studies have explored why. Visual cognition studies suggest that features of the visual representations we use to teach students influence what domain knowledge they use to solve problems. However, few studies have explored how visual representations influence what problem‐solving strategies and domain knowledge students of different levels of expertise use when solving problems that require them to create and coordinate multiple representations.


    This study addressed the following research question: How do students with different levels of expertise coordinate their problem‐solving strategies, problem‐solving heuristics, and representation features when sketching their shear force and bending moment diagrams?


    We conducted think‐aloud interviews while students sketched shear force and bending moment diagrams. These interviews were subsequently analyzed using the constant comparative method to examine the effect of representations on students' problem‐solving approaches.


    Three themes emerged from the data: Students used heuristics that are based on perceptually salient features to sketch their shear force and bending moment diagrams; students across levels of expertise rely on theobject translationheuristic rather than equilibrium problem‐solving schema to sketch and reason through their shear force and bending moment diagrams, and domain knowledge aids students' ability to resolve conflicting heuristics. Our findings suggest that students primarily rely on heuristics triggered by representation features they notice.


    Students engaged with shear force and bending moment diagrams not as a way to describe systems that are not accelerating but as a series of representations that “should go to zero” or arrows that make things “not zero.”

    more » « less
  2. null (Ed.)
    Crowdsourcing provides an efficient label collection schema for supervised machine learning. However, to control annotation cost, each instance in the crowdsourced data is typically annotated by a small number of annotators. This creates a sparsity issue and limits the quality of machine learning models trained on such data. In this paper, we study how to handle sparsity in crowdsourced data using data augmentation. Specifically, we propose to directly learn a classifier by augmenting the raw sparse annotations. We implement two principles of high-quality augmentation using Generative Adversarial Networks: 1) the generated annotations should follow the distribution of authentic ones, which is measured by a discriminator; 2) the generated annotations should have high mutual information with the ground-truth labels, which is measured by an auxiliary network. Extensive experiments and comparisons against an array of state-of-the-art learning from crowds methods on three real-world datasets proved the effectiveness of our data augmentation framework. It shows the potential of our algorithm for low-budget crowdsourcing in general. 
    more » « less
  3. Abstract

    Problem solving is a central focus of mathematics teaching and learning. If teachers are expected to support students' problem‐solving development, then it reasons that teachers should also be able to solve problems aligned to grade level content standards. The purpose of this validation study is twofold: (1) to present evidence supporting the use of the Problem Solving Measures Grades 3–5 with preservice teachers (PSTs), and (2) to examine PSTs' abilities to solve problems aligned to grades 3–5 academic content standards. This study used Rasch measurement techniques to support psychometric analysis of the Problem Solving Measures when used with PSTs. Results indicate the Problem Solving Measures are appropriate for use with PSTs, and PSTs' performance on the Problem Solving Measures differed between first‐year PSTs and end‐of‐program PSTs. Implications include program evaluation and the potential benefits of using K‐12 student‐level assessments as measures of PSTs' content knowledge.

    more » « less
  4. As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification. 
    more » « less
  5. Abstract

    Both sketching and self‐explanation are widely believed to be effective for problem‐solving in science learning. However, it is unclear which aspects of these strategies promote learning and how they might interact. Compared to a read‐only baseline, we examined the impact of instructing 11‐year‐old students to solve science problems to sketch, self‐explain, or both. Problems were either high spatial or low spatial. We coded elements and relations in their sketches and think‐alouds. Self‐explanation led to greater accuracy on problems, but only when used alone. Sketching had no effect, although showing more elements and relations in sketches was associated with higher accuracy and occurred more often for high spatial problems. Students may be more comfortable with self‐explanation than with sketching and require encouragement or specific instruction to use sketching routinely for the benefits hypothesized to be associated with sketching to appear.

    more » « less