More Like this

1. Custos Secrets: a Service for Managing User-Provided Resource Credential Secrets for Science Gateways

   https://doi.org/10.1145/3491418.3535177  

   Ranawaka, Isuru; Goonasekara, Nuwan; Afgan, Enis; Basney, Jim; Marru, Suresh; Pierce, Marlon (July 2022, Practice and Experience in Advanced Research Computing (PEARC '22))

   Jupyter Notebooks are an enormously popular tool for creating and narrating computational research projects. They also have enormous potential for creating reproducible scientific research artifacts. Capturing the complete state of a notebook has additional benefits; for instance, the notebook execution may be split between local and remote resources, where the latter may have more powerful processing capabilities or store large or access-limited data. There are several challenges for making notebooks fully reproducible when examined in detail. The notebook code must be replicated entirely, and the underlying Python runtime environments must be identical. More subtle problems arise in replicating referenced data, external library dependencies, and runtime variable states. This paper presents solutions to these problems using Juptyer’s standard extension mechanisms to create an archivable system state for a running notebook. We show that the overhead for these additional mechanisms, which involve interacting with the underlying Linux kernel, does not introduce substantial execution time overheads, demonstrating the approach’s feasibility. 

   more » « less
2. Elevating Jupyter Notebook Maintenance Tooling by Identifying and Extracting Notebook Structures

   https://doi.org/10.1109/ICSME55016.2022.00047  

   Jiang, Yuan; Kastner, Christian; Zhou, Shurui (October 2022, 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME))

   Data analysis is an exploratory, interactive, and often collaborative process. Computational notebooks have become a popular tool to support this process, among others because of their ability to interleave code, narrative text, and results. However, notebooks in practice are often criticized as hard to maintain and being of low code quality, including problems such as unused or duplicated code and out-of-order code execution. Data scientists can benefit from better tool support when maintaining and evolving notebooks. We argue that central to such tool support is identifying the structure of notebooks. We present a lightweight and accurate approach to extract notebook structure and outline several ways such structure can be used to improve maintenance tooling for notebooks, including navigation and finding alternatives. 

   more » « less
3. Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code

   https://doi.org/10.32473/flairs.v34i1.128560  

   Oli, Priti; Banjade, Rabin; Tamang, Lasang Jimba; Rus, Vasile (May 2021, The International FLAIRS Conference Proceedings)

   We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they have the advantage of no need for expert involvement to define features. 

   more » « less
4. Exploring Organization of Computational Notebook Cells in 2D Space

   https://doi.org/10.1109/VL/HCC53370.2022.9833128  

   Harden, Jesse; Christman, Elizabeth; Kirshenbaum, Nurit; Wenskovitch, John; Leigh, Jason; North, Chris (October 2022, 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC))

   Representing branching and comparative analyses in computational notebooks is complicated by the 1-dimensional (1D), top-down list arrangement of cells. Given the ubiquity of these and other non-linear features, their importance to analysis and narrative, and the struggles current 1D computational notebooks have, enabling organization of computational notebook cells in 2 dimensions (2D) may prove valuable. We investigated whether and how users would organize cells in such a “2D Computational Notebook” through a user study and gathered feedback from participants through a follow-up survey and optional interviews. Through the user study, we found 3 main design patterns for arranging notebook cells in 2D: Linear, Multi-Column, and Workboard. Through the survey and interviews, we found that users see potential value in 2D Computational Notebooks for branching and comparative analyses, but the expansion from 1D to 2D may necessitate additional navigational and organizational aids. 

   more » « less
5. Advanced Searches of the Protein Data Bank using Python in Jupyter Notebooks

   Greever, Victoria; Rack, Anna; Pick, Murphy; Schoneman, Lee; Craig, Paul A (April 2025, ASBMB)

   The Protein Data Bank (PDB) holds an extensive amount of information, and can be a vital tool when performing background research for biochemical work. In an attempt to make the information in the PDB more accessible, the RCSB Search API was employed within Jupyter Notebooks to create more customizable and user-friendly tools with Python code. Areas of focus include searches targeting ligands with specific characteristics, searches for FDA Approved Drugs, as well as sequence searches, used to search for entries based on different sequence characteristics. This code has been built into Jupyter Notebook templates that include examples of these searches as well as annotated code that users can customize to more efficiently run advanced searches on the PDB and download structure and small molecule files returned by the search. These notebooks also walk users through different ways to organize or utilize the returns from advanced searches. Future plans include increasing the amount and type of information available from a search, improved ease of access for visualizing and downloading search results, and expanding the scope of our notebooks to cover more types of searches. This research was supported by NSF-IUSE award number 2142033. 

   more » « less