skip to main content

Explore Research Products in the NSF-PAR

This content will become publicly available on June 1, 2024

Title: Mitigating Skewed Bidding for Conference Paper Assignment
The explosion of conference paper submissions in AI and related fields has underscored the need to improve many aspects of the peer review process, especially the matching of papers and reviewers. Recent work argues that the key to improve this matching is to modify aspects of the bidding phase itself, to ensure that the set of bids over papers is balanced, and in particular to avoid orphan papers, i.e., those papers that receive no bids. In an attempt to understand and mitigate this problem, we have developed a flexible bidding platform to test adaptations to the bidding process. Using this platform, we performed a field experiment during the bidding phase of a medium-size international workshop that compared two bidding methods. We further examined via controlled experiments on Amazon Mechanical Turk various factors that affect bidding, in particular the order in which papers are presented [11, 17]; and information on paper demand [33]. Our results suggest that several simple adaptations, that can be added to any existing platform, may significantly reduce the skew in bids, thereby improving the allocation for both reviewers and conference organizers.  more » « less
Award ID(s):
2134857
NSF-PAR ID:
10480839
Author(s) / Creator(s):
; ; ;
Publisher / Repository:
ACM Digital Library
Date Published:
Journal Name:
Proceedings of the 22nd International Conference on Autonomous Agents and Multiagent Systems
Format(s):
Medium: X
Location:
London, UK
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ( , Uncertainty in artificial intelligence)
    null (Ed.)
    A number of applications involve sequential arrival of users, and require showing each user an ordering of items. A prime example is the bidding process in conference peer review where reviewers enter the system sequentially, each reviewer needs to be shown the list of submitted papers, and the reviewer then ``bids'' to review some papers. The order of the papers shown has a significant impact on the bids due to primacy effects. In deciding on the ordering of the list of papers to show, there are two competing goals: (i) obtaining sufficiently many bids for each paper, and (ii) satisfying reviewers by showing them relevant items. In this paper, we develop a framework to study this problem in a principled manner. We present an algorithm called SUPER*, inspired by the A* algorithm, for this goal. Theoretically, we show a local optimality guarantee of our algorithm and prove that popular baselines are considerably suboptimal. Moreover, under a community model for the similarities, we prove that SUPER* is near-optimal whereas the popular baselines are considerably suboptimal. In experiments on real data from ICLR 2018 and synthetic data, we find that SUPER* considerably outperforms baselines deployed in existing systems, consistently reducing the number of papers with fewer than requisite bids by 50-75% or more, and is also robust to various real world complexities. 
    more » « less
  2. ( , Proceedings of Machine Learning Research)
    Most computer science conferences rely on paper bidding to assign reviewers to papers. Although paper bidding enables high-quality assignments in days of unprecedented submission numbers, it also opens the door for dishonest reviewers to adversarially influence paper reviewing assignments. Anecdotal evidence suggests that some reviewers bid on papers by "friends" or colluding authors, even though these papers are outside their area of expertise, and recommend them for acceptance without considering the merit of the work. In this paper, we study the efficacy of such bid manipulation attacks and find that, indeed, they can jeopardize the integrity of the review process. We develop a novel approach for paper bidding and assignment that is much more robust against such attacks. We show empirically that our approach provides robustness even when dishonest reviewers collude, have full knowledge of the assignment system’s internal workings, and have access to the system’s inputs. In addition to being more robust, the quality of our paper review assignments is comparable to that of current, non-robust assignment approaches. 
    more » « less
  3. ( , Proceedings of Machine Learning Research)
    Most computer science conferences rely on paper bidding to assign reviewers to papers. Although paper bidding enables high-quality assignments in days of unprecedented submission numbers, it also opens the door for dishonest reviewers to adversarially influence paper reviewing assignments. Anecdotal evidence suggests that some reviewers bid on papers by “friends” or colluding authors, even though these papers are outside their area of expertise, and recommend them for acceptance without considering the merit of the work. In this paper, we study the efficacy of such bid manipulation attacks and find that, indeed, they can jeopardize the integrity of the review process. We develop a novel approach for paper bidding and assignment that is much more robust against such attacks. We show empirically that our approach provides robustness even when dishonest reviewers collude, have full knowledge of the assignment system’s internal workings, and have access to the system’s inputs. In addition to being more robust, the quality of our paper review assignments is comparable to that of current, non-robust assignment approaches. 
    more » « less
  4. ; ; ; ; ( , 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid))
    In the Internet of Things (loT) era, edge computing is a promising paradigm to improve the quality of service for latency sensitive applications by filling gaps between the loT devices and the cloud infrastructure. Highly geo-distributed edge computing resources that are managed by independent and competing service providers pose new challenges in terms of resource allocation and effective resource sharing to achieve a globally efficient resource allocation. In this paper, we propose a novel blockchain-based model for allocating computing resources in an edge computing platform that allows service providers to establish resource sharing contracts with edge infrastructure providers apriori using smart contracts in Ethereum. The smart contract in the proposed model acts as the auctioneer and replaces the trusted third-party to handle the auction. The blockchain-based auctioning protocol increases the transparency of the auction-based resource allocation for the participating edge service and infrastructure providers. The design of sealed bids and bid revealing methods in the proposed protocol make it possible for the participating bidders to place their bids without revealing their true valuation of the goods. The truthful auction design and the utility-aware bidding strategies incorporated in the proposed model enables the edge service providers and edge infrastructure providers to maximize their utilities. We implement a prototype of the model on a real blockchain test bed and our extensive experiments demonstrate the effectiveness, scalability and performance efficiency of the proposed approach. 
    more » « less
  5. ; ( , IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT))
    null (Ed.)
    Convergence bidding, has been adopted in recent years by most Independent System Operators (ISOs) in the United States as a relatively new market mechanism to enhance market efficiency. Convergence bidding affects many aspects of the operation of the electricity markets and there is currently a gap in the literature on understanding how the market participants strategically select their convergence bids in practice. To address this open Problem, in this paper, we study three years of real-world market data from the California ISO energy market. First, we provide a data - driven overview of all submitted convergence bids (CBs) and analyze the performance of each individual convergence bidder based on the number of their submitted CBs, the number of locations that they placed the CBs, the percentage of submitted supply or demand and CBs, the amount of cleared CBs, and their gained profit or loss. Next, we scrutinize the bidding strategies of the 13 largest market players that account for 75 % of all CBs in. the California ISO market. We identify quantitative features to characterize and distinguish their different convergence bidding strategies. This analysis results in revealing three different classes of CB strategies that are used in practice. We identify the differences between. these strategic bidding classes and compare their advantages and disadvantages. We also explain how some of the most active market participants are using bidding strategies that do not any of the strategic bidding methods that currently exist in the literature. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email