In decision-making scenarios, like scholarship distribution, academic evaluation, and product recommendations, stakeholders often * e-mail:hshrestha@wpi.edu † e-mail:kcachel@wpi.edu ‡ e-mail:malkhathlan@wpi.edu § e-mail:rundenst@wpi.edu ¶ e-mail:ltharrison@wpi.edu seek to create a consensus ranking by combining diverse perspectives. This complex process [2] requires careful consideration of different viewpoints and aggregating all preferences. Moreover, individual rankings may exhibit biases, potentially favoring certain groups over others [3]. For example, biases favoring male candidates over female ones in hiring can raise concerns about fairness in the ranking process.
To address these limitations, we designed FairSpace, an interactive visualization system for exploring large datasets of rankings and iteratively constructing fair consensus rankings. We use dimensional reduction techniques to visualize similar rankings in terms of utility and fairness and hierarchically construct a fair consensus ranking.
We begin with the data model used in FairFuse [4], which defines a list of candidates ranked by a set of rankers. Each candidate has a protected attribute such as race or gender, defining groups such as White, Black, Asian, etc. For demonstrating this system, we incorporate the fairness metrics Favored Pair Representation (FPR) and Attribute Pair Representation (ARP) to quantify the fairness score of rankings [1]. Similarly, we use the Kendall Tau distance to measure utility. Finally, we apply the Fair-Copeland Algorithm [1] for the auto-generation of fair consensus rankings by applying a certain fairness threshold. However, these metrics and algorithms can be replaced with others depending on the scenarios and requirements.
In addition to the existing tasks from FairFuse [4], we add new tasks considering the possibility of an increased number of rankings, such as: A) Identifying similar rankings and forming local clusters to simplify comparison, B) Supporting comparisons between clusters in terms of fairness and agreement, C) Supporting comparisons of individual rankings with their local cluster, D) Constructing a global consensus through a hierarchical approach, E) Constructing a global fair consensus ranking and analyzing its agreement with local consensuses as well as individual rankings.
FairSpace is designed to analyze and establish a fair consensus when dealing with a large number of rankings. The system comprises several views that offer a holistic perspective by presenting all the rankings collectively, as well as individual rankings. Additionally, we integrate visualizations to illustrate fairness metrics [4] and fair consensus ranking generation process.
To provide a holistic view of the data with respect to protected groups, FairSpace uses distribution plots as shown in the sidebar of Fig. 1A. When a ranking or a cluster of rankings is selected, this view is updated with an overlay of groups' distribution of the selected ranking(s). Alongside the distribution plot, we display the ratio of candidates by the groups in the protected attribute to provide the idea of majority and minority groups in the candidates pool.
FairSpace uses a dimensional reduction technique, specifically Multidimensional Scaling (MDS), to visualize the large number of rankings. However, this can be switched with other techniques such as t-SNE if required. These views represent the similarity of rankings in utility space, fairness space, and combination of both. This enables users to form cluster of similar rankings and analyze the clusters as a whole instead of going through individual rankings.
Using the Cluster Views (see Fig. 1), the decision maker can identify similar rankings in terms of both the fairness and utility metrics. These similar rankings can be selected by drawing a lasso around them, eventually allowing comparison of these rankings and generating their local consensus. The generated local consensus ranking is projected back to the cluster views enabling comparison between multiple local consensuses. Additionally, we incorporate the Group Fairness View from FairFuse [4] directly into the Cluster Views. Having the local consensus and the Group Fairness View within the Cluster Views allows the decision-maker to quickly grasp the underlying rankings without having to review each individual ranking. Any rankings in this view, when selected, are displayed in the Rank Comparison View Fig. 1E. The Unified Cluster View (Fig. 1B) allows decision-makers to visualize differences between clusters or between a cluster and a ranking using the Fair Divergence View (Fig. 2). Utilizing its heatmap, users can identify candidates ranked differently, such as how Cluster 2 compares to Cluster 1.
Finally, Cluster Views enable decision-makers to select local consensuses and construct a global consensus ranking. When building a global consensus, we consider the base rankings of the local consensuses. To emphasize the hierarchy, we use lines and opacity-lower levels in the tree have lower opacity, and vice versa.
The Rank Comparison View (Fig. 1E) displays any selected ranking(s) from the Cluster Views enabling detailed comparison. Each candidate's name along with the color associated with the protected attribute is displayed, which was found to be effective in FairFuse [5]. This view utilizes Parallel Coordinates Plot design for comparison of candidates between the rankings.
At the top of each ranking, we have the Group Fairness View which displays the fairness scores (FPR and ARP) [1] of the ranking. The colored dots represent a group, like White, Asian, Black, etc. The dots above 0.5 FPR line represent advantaged groups and viceversa. This view is also displayed on the Cluster Views for quick comparison. Additionally, the Group Fairness View is overlayed with a fairness slider. This slider can used to modify the ranking or a local consensus ranking to build a fairer ranking, essentially controlling the fairness threshold for the resulting fair ranking. This modified ranking is then projected back to the Cluster Views to provide feedback on how close/far it is from other rankings. This feedback can drive an iterative approach to build a global consensus that is fair and representative of all rankings.
The process of constructing a consensus ranking from a large set of individual rankings is complex, especially to mitigate biases. To address this challenge, we introduced FairSpace, a system to visualize and interact with many rankings. By facilitating the comparison of group distributions, fairness scores, and similarity scores, FairSpace empowers exploratory analysis of rankings, clusters of rankings, and consensus rankings. Through the integration of algorithms for generating fair consensus rankings and enabling manual ranking adjustments, along with a hierarchical approach for constructing a fair global consensus, FairSpace offers an iterative refinement approach to building a fair consensus from vast sets of rankings.
In the future, we plan to conduct a user study to validate the system and understand how people might use the system with large datasets. Additionally, we plan to expand this system to accommodate multiple protected attributes, provide support for partial rankings, and explore potential ways of integrating fairness in more realistic, uncertain contexts such as incomplete rankings.