An official website of the United States government
Most research studies on deep learning (DL) applied to the physical layer of wireless communication do not put forward the critical role of the accuracy-generalization trade-off in developing and evaluating practical algorithms. To highlight the disadvantage of this common practice, we revisit a data decoding example from one of the first papers introducing DL-based end-to-end wireless communication systems to the research community and promoting the use of artificial intelligence (AI)/DL for the wireless physical layer. We then put forward two key trade-offs in designing DL models for communication, namely, accuracy versus generalization and compression versus latency. We discuss their relevance in the context of wireless communications use cases using emerging DL models, including large language models (LLMs). Finally, we summarize our proposed evaluation guidelines to enhance the research impact of DL on wireless communications. These guidelines are an attempt to reconcile the empirical nature of DL research with the rigorous requirement metrics of wireless communications systems.
more »
« less
Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts
Presenting a predictive model's performance is a communication bottleneck that threatens collaborations between data scientists and subject matter experts. Accuracy and error metrics alone fail to tell the whole story of a model – its risks, strengths, and limitations – making it difficult for subject matter experts to feel confident in their decision to use a model. As a result, models may fail in unexpected ways or go entirely unused, as subject matter experts disregard poorly presented models in favor of familiar, yet arguably substandard methods. In this paper, we describe an iterative study conducted with both subject matter experts and data scientists to understand the gaps in communication between these two groups. We find that, while the two groups share common goals of understanding the data and predictions of the model, friction can stem from unfamiliar terms, metrics, and visualizations – limiting the transfer of knowledge to SMEs and discouraging clarifying questions being asked during presentations. Based on our findings, we derive a set of communication guidelines that use visualization as a common medium for communicating the strengths and weaknesses of a model. We provide a demonstration of our guidelines in a regression modeling scenario and elicit feedback on their use from subject matter experts. From our demonstration, subject matter experts were more comfortable discussing a model's performance, more aware of the trade-offs for the presented model, and better equipped to assess the model's risks – ultimately informing and contextualizing the model's use beyond text and numbers.
more »
« less
- PAR ID:
- 10412965
- Date Published:
- Journal Name:
- IEEE Transactions on Visualization and Computer Graphics
- ISSN:
- 1077-2626
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The formation of social groups is defined by the interactions among the group members. Studying this group formation process can be useful in understanding the status of members, decision-making behaviors, spread of knowledge and diseases, and much more. A defining characteristic of these groups is the pecking order or hierarchy the members form which help groups work towards their goals. One area of social science deals with understanding the formation and maintenance of these hierarchies, and in our work we provide social scientists with a visual analytics tool - PeckVis - to aid this process. While online social groups or social networks have been studied deeply and lead to a variety of analyses and visualization tools, the study of smaller groups in the field of social science lacks the support of suitable tools. Domain experts believe that visualizing their data can save them time as well as reveal findings they may have failed to observe. We worked alongside domain experts to build an interactive visual analytics system to investigate social hierarchies. Our system can discover patterns and relationships between the members of a group as well as compare different groups. The results are presented to the user in the form of an interactive visual analytics dashboard. We demonstrate that domain experts were able to effectively use our tool to analyze animal behavior data.more » « less
-
Data Science is one of the fastest growing fields with unmet demand from employers. Many academic institutions have taken on the task of creating programs to meet both current and future needs and demands. Data science, as a field, integrates aspects of computer science, statistics, and subject matter expertise which encourages cross-disciplinary conversations and collaboration. In this talk, we present results from a broad survey of instructors of introductory college-level data science courses for undergraduates. In addition, we explore the alignment of these findings with the recommendations of various professional organizations. We conducted a national survey on topics covered in introductory, college-level data science courses. With responses from computer scientists, statisticians, and allied fields, these results represent a wide array of instructors of data science. The survey identifies topics commonly covered, the amount of time spent on each, common and divergent definitions of data science, and course materials used. These results will be presented. We will then discuss the alignment of these results through a rigorous review and synthesis of recommendations from various professional organizations. These include Association for Computing Machinery's Computing Competencies for Undergraduate Data Science Curricula[1], the National Academies of Science, Engineering, and Medicine’s Data Science for Undergraduates: Opportunities and Options[2], the Park City Math Institute's report Curriculum Guidelines for Undergraduate Programs in Data Science[3], and the American Statistical Association’s Two-Year College Data Science Summit Final Report[4] and Curriculum Guidelines for Undergraduate Programs in Statistical Science[5]. We will also explore alignment with ABET’s accreditation of data science.[6]more » « less
-
Failure analysis and defect detection are crucial processes in industries, governments, and societies to mitigate the risks associated with defective microelectronics. The accurate identification of faulty parts is vital for preventing potential damages. However, traditional manual and automated defect detection approaches face challenges due to the scarcity of ground truth data from defective parts. This limitation hampers the effectiveness of subject matter experts and machine learning models in recognizing and classifying new instances of defects. To address this issue, we propose a synthetic data augmentation workflow that generates virtual defective parts, effectively overcoming the data scarcity problem and enabling the creation of large datasets at a low cost. Our approach enhances defect detection capabilities, empowering industries and governments to improve the quality and reliability of electronic devices.more » « less
-
The “Accessible Oceans” pilot project aims to inclusively design auditory displays that support perception and understanding of ocean data in informal learning environments (ILEs). The project’s multi-disciplinary team includes expertise from all related fields — ocean scientists, dataset experts, a sound designer with specialization in data sonification, and a learning sciences researcher. In addition, the PI is blind and provides a crucial perspective in our research. We describe the sound design of informative sonifications and respective auditory displays based on iterative design with user input at each stage, including from blind and low-vision (BLV) students, their teachers, and subject-matter experts. We discuss the importance of framing data sonifications through an auditory presentation of contextual information. We also report on our latest auditory display evaluation using Auditory Interface UX Scale (BUZZ) surveys at three ILE test sites. These responses further affirm our auditory display design developments. We include access to the auditory displays media and lessons learned over the course of this multi-year NSF-funded Advancing Informal Stem Learning (AISL) grant https://accessibleoceans.whoi.edu/more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1109/TVCG.2023.3259341
