More Like this

1. Software Environments in Binder Containers

   https://doi.org/10.5281/zenodo.4891790  

   Shaffer, Tim ; Chard, Kyle ; Thain, Douglas ( January 2021 )

   Abstract

   <p>Binder is a publicly accessible online service for executing interactive notebooks based on Git repositories. Binder dynamically builds and deploys containers following a recipe stored in the repository, then gives the user a browser-based notebook interface. The Binder group periodically releases a log of container launches from the public Binder service. Archives of launch records are available here. These records do not include identifiable information like IP addresses, but do give the source repo being launched along with some other metadata. The main content of this dataset is in the <code>binder.sqlite</code> file. This SQLite database includes launch records from 2018-11-03 to 2021-06-06 in the <code>events</code> table, which has the following schema.</p> <code>CREATE TABLE events( version INTEGER, timestamp TEXT, provider TEXT, spec TEXT, origin TEXT, ref TEXT, guessed_ref TEXT ); CREATE INDEX idx_timestamp ON events(timestamp); </code> <ul><li><code>version</code> indicates the version of the record as assigned by Binder. The <code>origin</code> field became available with version 3, and the <code>ref</code> field with version 4. Older records where this information was not recorded will have the corresponding fields set to null.</li><li><code>timestamp</code> is the ISO timestamp of the launch</li><li><code>provider</code> gives the type of source repo being launched (&#34;GitHub&#34; is by far the most common). The rest of the explanations assume GitHub, other providers may differ.</li><li><code>spec</code> gives the particular branch/release/commit being built. It consists of <code>&lt;github-id&gt;/&lt;repo&gt;/&lt;branch&gt;</code>.</li><li><code>origin</code> indicates which backend was used. Each has its own storage, compute, etc. so this info might be important for evaluating caching and performance. Note that only recent records include this field. May be null.</li><li><code>ref</code> specifies the git commit that was actually used, rather than the named branch referenced by <code>spec</code>. Note that this was not recorded from the beginning, so only the more recent entries include it. May be null.</li><li>For records where <code>ref</code> is not available, we attempted to clone the named reference given by <code>spec</code> rather than the specific commit (see below). The <code>guessed_ref</code> field records the commit found at the time of cloning. If the branch was updated since the container was launched, this will not be the exact version that was used, and instead will refer to whatever was available at the time (early 2021). Depending on the application, this might still be useful information. Selecting only records with version 4 (or non-null <code>ref</code>) will exclude these guessed commits. May be null.</li></ul> <p>The Binder launch dataset identifies the source repos that were used, but doesn&#39;t give any indication of their contents. We crawled GitHub to get the actual specification files in the repos which were fed into repo2docker when preparing the notebook environments, as well as filesystem metadata of the repos. Some repos were deleted/made private at some point, and were thus skipped. This is indicated by the absence of any row for the given commit (or absence of both <code>ref</code> and <code>guessed_ref</code> in the <code>events</code> table). The schema is as follows.</p> <code>CREATE TABLE spec_files ( ref TEXT NOT NULL PRIMARY KEY, ls TEXT, runtime BLOB, apt BLOB, conda BLOB, pip BLOB, pipfile BLOB, julia BLOB, r BLOB, nix BLOB, docker BLOB, setup BLOB, postbuild BLOB, start BLOB );</code> <p>Here <code>ref</code> corresponds to <code>ref</code> and/or <code>guessed_ref</code> from the <code>events</code> table. For each repo, we collected spec files into the following fields (see the repo2docker docs for details on what these are). The records in the database are simply the verbatim file contents, with no parsing or further processing performed.</p> <ul><li><code>runtime</code>: <code>runtime.txt</code></li><li><code>apt</code>: <code>apt.txt</code></li><li><code>conda</code>: <code>environment.yml</code></li><li><code>pip</code>: <code>requirements.txt</code></li><li><code>pipfile</code>: <code>Pipfile.lock</code> or <code>Pipfile</code></li><li><code>julia</code>: <code>Project.toml</code> or <code>REQUIRE</code></li><li><code>r</code>: <code>install.R</code></li><li><code>nix</code>: <code>default.nix</code></li><li><code>docker</code>: <code>Dockerfile</code></li><li><code>setup</code>: <code>setup.py</code></li><li><code>postbuild</code>: <code>postBuild</code></li><li><code>start</code>: <code>start</code></li></ul> <p>The <code>ls</code> field gives a metadata listing of the repo contents (excluding the <code>.git</code> directory). This field is JSON encoded with the following structure based on JSON types:</p> <ul><li>Object: filesystem directory. Keys are file names within it. Values are the contents, which can be regular files, symlinks, or subdirectories.</li><li>String: symlink. The string value gives the link target.</li><li>Number: regular file. The number value gives the file size in bytes.</li></ul> <code>CREATE TABLE clean_specs ( ref TEXT NOT NULL PRIMARY KEY, conda_channels TEXT, conda_packages TEXT, pip_packages TEXT, apt_packages TEXT );</code> <p>The <code>clean_specs</code> table provides parsed and validated specifications for some of the specification files (currently Pip, Conda, and APT packages). Each column gives either a JSON encoded list of package requirements, or null. APT packages have been validated using a regex adapted from the repo2docker source. Pip packages have been parsed and normalized using the Requirement class from the pkg_resources package of setuptools. Conda packages have been parsed and normalized using the <code>conda.models.match_spec.MatchSpec</code> class included with the library form of Conda (distinct from the command line tool). Users might want to use these parsers when working with the package data, as the specifications can become fairly complex.</p> <p>The <code>missing</code> table gives the repos that were not accessible, and <code>event_logs</code> records which log files have already been added. These tables are used for updating the dataset and should not be of interest to users.</p>
   More>>
2. Scaling databases and file apis with programmable ceph object storage

   LeFevre, Jeff Maltzahn ( February 2020 , USENIX VAULT'20)

   The Skyhook Data Management project (SkyhookDM.com) at the Center for Research in Open Source Software (cross.ucsc.edu) at UC Santa Cruz implements customized extensions through Ceph's object class interface that enables offloading database operations to the storage system. In our previous Vault '19 talk, we showed how SkyhookDM can transparently scale out databases. The SkyhookDM Ceph extensions are an example of our 'programmable storage' research efforts at UCSC, and can be accessed through commonly available external/foreign table database interfaces. Utilizing fast in-memory serialization libraries such as Google Flatbuffers and Apache Arrow, SkyhookDM currently implements common database functions such as SELECT, PROJECT, AGGREGATE, and indexing inside Ceph, along with lower-level data manipulations such as transforming data from row to column formats on RADOS servers. In this talk, we will present three of our latest developments on the SkyhookDM project since Vault '19. First, SkyhookDM can be used to also offload operations of access libraries that support plugins for backends, such as HDF5 and its Virtual Object Layer. Second, in addition to row-oriented data format using Google's Flatbuffers, we have added support for column-oriented data formats using the Apache Arrow library within our Ceph extensions. Third, we added dynamic switching between row andmore »column data formats within Ceph objects, a first step towards physical design management in storage systems, similar to physical design tuning in database systems.« less
3. Optimal Hashing in External Memory

   Conway, Alex Farach-Colton ( January 2018 , Leibniz international proceedings in informatics)

   Hash tables are a ubiquitous class of dictionary data structures. However, standard hash table implementations do not translate well into the external memory model, because they do not incorporate locality for insertions. Iacono and Pătraşu established an update/query tradeoff curve for external-hash tables: a hash table that performs insertions in O(λ/B) amortized IOs requires Ω(logλ N) expected IOs for queries, where N is the number of items that can be stored in the data structure, B is the size of a memory transfer, M is the size of memory, and λ is a tuning parameter. They provide a complicated hashing data structure, which we call the IP hash table, that meets this curve for λ that is Ω(loglogM +logM N). In this paper, we present a simpler external-memory hash table, the Bundle of Arrays Hash Table (BOA), that is optimal for a narrower range of λ. The simplicity of BOAs allows them to be readily modified to achieve the following results: A new external-memory data structure, the Bundle of Trees Hash Table (BOT), that matches the performance of the IP hash table, while retaining some of the simplicity of the BOAs. The Cache-Oblivious Bundle of Trees Hash Table (COBOT), themore »first cache-oblivious hash table. This data structure matches the optimality of BOTs and IP hash tables over the same range of λ.« less
4. TaxonWorks: Character state matrices and identification tools

   https://doi.org/10.3897/biss.5.75554  

   Dmitriev, Dmitry ; Yoder, Matthew ( September 2021 , Biodiversity Information Science and Standards)

   TaxonWorks is an integrated web-based application for practicing taxonomists and biodiversity specialists. It is focused on promoting collaboration between researchers and developers. TaxonWorks has a modular structure that enables various components of the application to target specific needs and requirements of different groups of users. Specific areas of interest may include nomenclature-related tasks (Yoder and Dmitriev 2021) designed to help assemble and validate scientific name checklists of a target group of organisms; and collection management tasks, including interfaces to create, filter, and edit collecting events, collection objects, and loans. This presentation focuses on matrix-related tools integrated into TaxonWorks. A matrix, which could either be used for phylogenetic analysis or to build an identification key, is structured as a table where columns represent numerous characters that could be used to describe a set of entities, taxa or specimens (presented as rows of the table). Each cell of the table may contain observations for specific character/entity combinations. TaxonWorks does not generate a table for each a particular matrix—all observations are stored as graphs. This structure allows building of a matrix of an unlimited size as well as reuse of individual observations in multiple matrices. For matrix columns, TaxonWorks supports a variety ofmore »different kinds of characters or descriptors: qualitative, presence/absence, quantitative, sample, gene, free text, and media. Each character may have specific properties, for example a qualitative descriptor may have numerous characters states, and a quantitative descriptor may have a measurement unit defined. For an entity in a matrix row, TaxonWorks supports either collection objects (specimens) or taxa as Operational Taxonomic Units (OTU). OTUs could either be linked to nomenclature or be stand alone entities (e.g., representing undescribed species). The matrix, once built, could serve several purposes. A matrix based on qualitative and quantitative characters could be used to build an interactive key (Fig. 1), construct standardized natural language descriptions for each entity, and determine a diagnosis (a minimal set of characters that separate one entity from all others). It could also be exported and used for phylogenetic analysis or to build an interactive key in an external application. TaxonWorks supports export files in several formats, including Nexus, TNT, NeXML. Application Programming Interfaces (API) are also available. A matrix based on media descriptors could be used as a pictorial identification tool (Fig. 2).« less
5. Typed Table Transformations

   Erwig, Martin ( January 2018 , Int. Workshop on Software Engineering Methods in Spreadsheets)

   Spreadsheet tables are often labeled, and these labels effectively constitute types for the data in the table. In such cases tables can be considered to be built from typed data where the placement of values within the table is controlled by the types used for rows and columns. We present a new approach to the transformations of spreadsheet tables that is based on transformations of row and column types. We illustrate the basic idea of type-based table construction and transformation and lay out a series of research questions that should be addressed in future work.