The success of ChatGPT is reshaping the landscape of the entire IT industry. The large language model (LLM) powering ChatGPT is experiencing rapid development, marked by enhanced features, improved accuracy, and reduced latency. Due to the execution overhead of LLMs, prevailing commercial LLM products typically manage user queries on remote servers. However, the escalating volume of user queries and the growing complexity of LLMs have led to servers becoming bottlenecks, compromising the quality of service (QoS). To address this challenge, a potential solution is to shift LLM inference services to edge devices, a strategy currently being explored by industry leaders such as Apple, Google, Qualcomm, Samsung, and others. Beyond alleviating the computational strain on servers and enhancing system scalability, deploying LLMs at the edge offers additional advantages. These include real-time responses even in the absence of network connectivity and improved privacy protection for customized or personal LLMs. This article delves into the challenges and potential bottlenecks currently hindering the effective deployment of LLMs on edge devices. Through deploying the LLaMa-2 7B model with INT4 quantization on diverse edge devices and systematically analyzing experimental results, we identify insufficient memory and/or computing resources on traditional edge devices as the primary obstacles. Based on our observation and empirical analysis, we further provide insights and design guidance for the next generation of edge devices and systems from both hardware and software directions
more »
« less
Poster: Revealing Hidden Secrets: Decoding DNS PTR records with Large Language Models
Geolocating network devices is essential for various research areas. Yet, despite notable advancements, it continues to be one of the most challenging issues for experimentalists. An approach for geolocating that has proved effective is leveraging geolocating hints in PTR records associated with network devices. We argue that Large Language Models (LLMs), rather than humans, are better equipped to identify patterns in DNS PTR records, and significantly scale the coverage of tools like Hoiho. We introduce an approach that leverages LLMs to classify PTR records, and generate regular expressions for these classes, and hint-to-location mapping. We present preliminary results showing the applicability of using LLMs as a scalable approach to leverage PTR records for infrastructure geolocation.
more »
« less
- Award ID(s):
- 2246475
- PAR ID:
- 10534921
- Publisher / Repository:
- Proceedings of the ACM SIGCOMM 2024 Conference: Posters and Demos
- Date Published:
- ISBN:
- 9798400707179
- Subject(s) / Keyword(s):
- Internet Measurement, Natural Language Processing, Large Lan- guage Models, Internet Geolocation, RIPEAtlas
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
As Blockchain technology become more understood in recent years and its capability to solve enterprise business use cases become evident, technologist have been exploring Blockchain technology to solve use cases that have been daunting industries for years. Unlike existing technologies, one of the key features of blockchain technology is its unparalleled capability to provide, traceability, accountability and immutable records that can be accessed at any point in time. One application area of interest for blockchain is securing heterogenous networks. This paper explores the security challenges in a heterogonous network of IoT devices and whether blockchain can be a viable solution. Using an experimental approach, we explore the possibility of using blockchain technology to secure IoT devices, validate IoT device transactions, and establish a chain of trust to secure an IoT device mesh network, as well as investigate the plausibility of using immutable transactions for forensic analysis.more » « less
-
Accurate mapping of Autonomous Systems (ASes) to their owner organizations is fundamental for understanding the structure and dynamics of the Internet. However, as AS numbers have traditionally been delegated in an ad-hoc manner and organizational ownership has evolved over time, many organizations have registered resources under different names. Traditionally, researchers have relied on datasets like AS2Org, which map ASNs to organizations primarily using WHOIS records, but WHOIS inconsistencies often lead to missed and false relationships. We propose a new approach by leveraging the Resource Public Key Infrastructure (RPKI) to map ASNs to their managing organization. Our methodology combines multiple data sources: WHOIS records to extract organization names, RPKI certificates to identify potential siblings, and Large Language Models (LLMs) to find evidence not visible in WHOIS records currently. This integrated approach enables a more robust and accurate mapping of ASNs to organizations, notably improving inferences for 14% of multi-ASN clusters.more » « less
-
Uncertainty decomposition refers to the task of decomposing the total uncertainty of a model into data (aleatoric) uncertainty, resulting from the inherent complexity or ambiguity of the data, and model (epistemic) uncertainty, resulting from the lack of knowledge in the model. Performing uncertainty decomposition for large language models (LLMs) is an important step toward improving the reliability, trustworthiness, and interpretability of LLMs, but this research task is very challenging and remains unresolved. The existing canonical method, Bayesian Neural Network (BNN), cannot be applied to LLMs, because BNN requires training and ensembling multiple variants of models, which is infeasible or prohibitively expensive for LLMs. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarifications ensemble, which bypasses the need to train new models. Rather than ensembling models with different parameters, our approach generates a set of clarifications for the input, feeds them into the fixed LLMs, and ensembles the corresponding predictions. We show that our framework shares a symmetric decomposition structure with BNN. Empirical evaluations demonstrate that the proposed framework provides accurate and reliable uncertainty quantification on various tasks. Code will be made publicly available at https://github.com/UCSB-NLP-Chang/llm_uncertainty .more » « less
-
The process of matching patients with suitable clinical trials is essential for advancing medical research and providing optimal care. However, current approaches face challenges such as data standardization, ethical considerations, and a lack of interoperability between Electronic Health Records (EHRs) and clinical trial criteria. In this paper, we explore the potential of large language models (LLMs) to address these challenges by leveraging their advanced natural language generation capabilities to improve compatibility between EHRs and clinical trial descriptions. We propose an innovative privacy-aware data augmentation approach for LLM-based patient-trial matching (LLM-PTM), which balances the benefits of LLMs while ensuring the security and confidentiality of sensitive patient data. Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%. Additionally, we present case studies to further illustrate the effectiveness of our approach and provide a deeper understanding of its underlying principles.more » « less
An official website of the United States government

