More Like this

1. Investigating The Reproducibility of NPM Packages

   https://doi.org/10.1109/ICSME46990.2020.00071  

   Goswami, Pronnoy; Gupta, Saksham; Li, Zhiyuan; Meng, Na; Yao, Daphne (September 2020, 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME))

   null (Ed.)
2. A Large Scale Analysis of Semantic Versioning in NPM

   https://doi.org/10.1109/MSR59073.2023.00073  

   Pinckney, Donald; Cassano, Federico; Guha, Arjun; Bell, Jonathan (May 2023, IEEE)
3. A Large Scale Analysis of Semantic Versioning in NPM

   Donald Pinckney; Federico Cassano; Arjun Guha; Jonathan Bell (May 2023, IEEE International Working Conference on Mining Software Repositories)

   The NPM package repository contains over two million packages and serves tens of billions of downloads per-week. Nearly every single JavaScript application uses the NPM package manager to install packages from the NPM repository. NPM relies on a “semantic versioning” (‘semver’) scheme to maintain a healthy ecosystem, where bug-fixes are reliably delivered to downstream packages as quickly as possible, while breaking changes require manual intervention by downstream package maintainers. In order to understand how developers use semver, we build a dataset containing every version of every package on NPM and analyze the flow of updates throughout the ecosystem. We build a time-travelling dependency resolver for NPM, which allows us to determine precisely which versions of each dependency would have been resolved at different times. We segment our analysis to allow for a direct analysis of security-relevant updates (those that introduce or patch vulnerabilities) in comparison to the rest of the ecosystem. We find that when developers use semver correctly, critical updates such as security patches can flow quite rapidly to downstream dependencies in the majority of cases (90.09%), but this does not always occur, due to developers’ imperfect use of both semver version constraints and semver version number increments. Our findings have implications for developers and researchers alike. We make our infrastructure and dataset publicly available under an open source license. 

   more » « less
4. Leveraging Large Language Models to Detect npm Malicious Packages

   Zahan, Nusrat; Burckhardt, Philipp; Lysenko, Mikola; Aboukhadijeh, Feross; Williams, Laurie (April 2025, IEEE)

   Existing malicious code detection techniques demand the integration of multiple tools to detect different malware patterns, often suffering from high misclassification rates. Therefore, malicious code detection techniques could be enhanced by adopting advanced, more automated approaches to achieve high accuracy and a low misclassification rate. The goal of this study is to aid security analysts in detecting malicious packages by empirically studying the effectiveness of Large Language Models (LLMs) in detecting malicious code. We present SocketAI, a malicious code review workflow to detect malicious code. To evaluate the effectiveness SocketAI, we leverage a benchmark dataset of 5,115 npm packages, of which 2,180 packages have malicious code. We conducted a baseline comparison of GPT-3 and GPT-4 models with the state-of-the-art CodeQL static analysis tool, using 39 custom CodeQL rules developed in prior research to detect malicious Javascript code. We also compare the effectiveness of static analysis as a pre-screener with SocketAI workflow, measuring the number of files that need to be analyzed and the associated costs. Additionally, we performed a qualitative study to understand the types of malicious packages detected or missed by our workflow. Our baseline comparison demonstrates a 16% and 9% improvement over static analysis in precision and F1 scores, respectively. GPT-4 achieves higher accuracy with 99% precision and 97% F1 scores, while GPT-3 offers a more cost-effective balance at 91% precision and 94% F1 scores. Prescreening files with a static analyzer reduces the number of files requiring LLM analysis by 77.9% and decreases costs by 60.9% for GPT-3 and 76.1% for GPT-4. Our qualitative analysis identified data theft, execution of arbitrary code, and suspicious domain categories as the top detected malicious packages. 

   more » « less
5. Artifact For A Large Scale Analysis of Semantic Versioning in NPM

   https://doi.org/10.5281/zenodo.7552551  

   Pinckney, Donald; Cassano, Federico; Guha, Arjun; Bell, Jonathan (January 2023, Zenodo)

   This is the artifact for: A Large Scale Analysis of Semantic Versioning in NPM. The artifact contains: A full scrape of all metadata from NPM (package / version information, dependencies, etc.) as of October 31, 2022.A copy of our code, which includes the software for scraping metadata and package tarball (code) data, as well as all analysis scripts that are needed to replicate the figures from the paper. 

   more » « less