Computational experiments comprise a large spectrum of scientific computations, from large-scale simulations, to justin-time analytics, or performance studies. Different types of such experiments place different requirements on the experimental container in which they execute from the perspective of isolation (shared, multi-tenant, or completely isolated environment), levels of access, configurability, interactivity, and performance. Within this spectrum, computer science systems experiments are perhaps the most challenging to provide for as many -though not all -often require high level of configurability and access (bare metal reconfigurability with specialized network configuration capabilities), strong performance isolation, interactive access, and the ability to experiment at high performance.
To adequately address this level of challenge, traditional testbeds that provided experimental capabilities for computer science research have generally been configured by technologies developed in-house [1], [2]. The Chameleon testbed [3] broke with this pattern by adapting a mainstream open source cloud technology, OpenStack, to provide similar capabilities. This implementation strategy carries a range of practical benefits -such as familiar interfaces for users and operators, or the opportunity to leverage contributions from a large development community -but it also creates the potential to contribute back, and thus influence the debate on the best cloud configuration to support computer science research. With commercial cloud providers, such as Amazon Web Services (AWS), increasingly willing to offer more flexibility in the form of e.g., bare metal instances [4], and NSF's investment in the CloudBank [5] initiative to make commercial clouds available to the systems research community, this debate is gaining in importance, and could result in significant broadening of the opportunities for computer science experimentation.
The ability to leverage commercial cloud resources could open up significant opportunities for research. With their greater geographic dispersion, greater diversity of resources, and greater scales -though perhaps lesser customization to research needs -the commercial clouds provide an interesting offering to supplement specialized testbeds like Chameleon.
In [3] we articulate the specific extensions required to adapt clouds to the needs of systems research; in this paper we examine the question of what experiments could leverage both research and commercial clouds and how such experiments should be configured to control wide-area network performance to enable repeatably. Specifically, we describe our approach to constructing such an experiment in the context of the experimental workflow stages [6] of an experiment distributed over the Chameleon testbed and AWS cloud accessed via CloudBank.
The specific contributions of the paper are as follows:
pabilities, in particular cost. The rest of the paper is organized as follows. In Section 2 we describe tools and methods for constructing multi-cloud experiments. In Section 3, we show how those tools can be used to construct a non-trivial networking experiment. We follow by a discussion of cloud capabilities and cost, discuss related work, and conclude in Section 6.
Computer science experiments are typically enacted in stages [6] beginning with experiment design, identifying suitable resources, allocating and configuring them, running the experiment itself, and finally analysis. We analyze these stages below, and compare and contrast approaches used for experiment development on Chameleon and commercial resources available via CloudBank.
Both Chameleon and CloudBank provide user access via federated login -Chameleon via Globus Auth [7], CloudBank via CILogon [8] -so that once logged into one system a user can use the other without entering a password again. To use system resources, a user needs to be associated with a project that has active allocations. In Chameleon those allocations consist of Service Units (SUs) which represent one hour of wall clock time on a mainstream server. To request an allocation, users are first certified for PI eligibility according to infrastructure-specific policies [9], [10]. In Chameleon they can then propose a project to be awarded an allocation. In CloudBank, verified PIs are allocated pre-determined funds which represent direct funding for a specific project to be spent on one or more public clouds. Within each system, the PI or their delegate can manage allocations to track spending and grant membership in the project to others.
In the first step of the experimental workflow, users can browse resources available for experimentation but their descriptions differ among the systems. Chameleon resource descriptions are fine-grained so that users can browse the exact node configurations including processor types, cache hierarchies, I/O device types, rack placement, etc. Commercial providers typically represent their resources as "instance types" (see e.g., [11]) and provide general information about vCPUs/CPUs, memory, storage, network, and much less detailed information about processors. Further, information about resources in Chameleon is being kept rigorously up-to-date -component upgrades might affect experiments sensitive to hardware such as e.g., power management or performance variability -the testbed is then versioned so that experimenters can verify this information at a glance and users are offered a suite of verification tools for sanity checks. Commercial clouds do not track the hardware evolution and do not provide such versioning though Chameleon's verification tools, based on standard Linux commands (such as e.g., "biosdecode" and "dmidecode" to get BIOS settings) can be used to mitigate these shortcomings to some extent.
Once selected, resources can be allocated. A notable difference between Chameleon and commercial resources is the support for advance reservations which allow Chameleon users to reserve availability of resources ranging from nodes, to networks, and IP addresses for a specific time in the future [12]. In contrast, commercial clouds rely primarily on on-demand availability; vehicles such as "reserved instances" [13], [14] represent a billing discount, while "capacity reservations" based on an up-front payment that reserve capacity for specific type of instance [15], [16] work in a similar way to allocation on Chameleon. Another significant difference is how resources are described: Chameleon allows users to specify resources at different levels: from model-level descriptions (e.g., "I need four nodes with at least 2GB per core") to indicating a specific node in the system, essential for experiments that require control of hardware variability. In commercial clouds the only way to describe resources to be allocated is the instance type; while this provides a high-level description, it might map to different types of resources, without the user being able to control the mapping [3].
In both Chameleon and commercial clouds allocated resources are configured by deploying disk images to create bare metal or virtual machine instances; since many users will want to use consistent configuration across a distributed experiment it is useful to consider how portable those images are. Most cloud platforms have specific requirements for the format of a disk image (e.g., RAW or QCOW2 [17]), its disk layout (e.g., whole disk or partition image), or the environment included in the image (e.g., cloud-init for injecting SSH keys or a DHCP client configured on specific interfaces) making images incompatible between various providers. They all however support the same general structure (e.g., the SSH key injection pattern via cloud-init) that can be leveraged by tools, such as OpenStack qemu-image convert [18] and AWS VM Import/Export [19] to convert between images; thus, to use a Chameleon image on AWS, it has to be converted to RAW format (if not RAW already) and then converted to AMI [20] using AWS VM Import. This common structure and the resulting portability options are an important consequence of the decision to configure a research testbed as a cloud [3]. The fact that Chameleon is not only a cloud, but an OpenStack cloud in particular, facilitates things even further: for example, the metadata service in OpenStack supports EC2-compatible API [21] which means that the images designed for EC2 will work with OpenStack directly. That said, some aspects of portability may be complicated by issues higher in the stack: for example, Chameleon includes utilities for system verification and snapshotting on its base images that will not be present on images imported from AWS. We provide a more general discussion of image conversion tools, including tools like Packer [22], in [23].
Orchestration, which allows users to deploy configurations consisting of multiple interdependent images , networks, and other cloud services automatically, is a related issue and provides a further demonstration of the advantages of configuring a CS research testbed as a cloud. OpenStack Heat [24] and CloudFormation [25], used for orchestration in Chameleon and AWS respectively, both use declarative languages (YAML or instance types were evaluated. Each instance used the imported CentOS7 image and was hosted on a VM or bare metal server with the hardware properties and stated networking bandwidth limitations in Table I. The AWS stated networking limitations ranged from "Low to Moderate" to "100 Gbps". The CloudConnect circuit limited the maximum bandwidth to 5 Gbps. To configure Chameleon nodes, we used the Centos7 image provided by the testbed. To achieve a consistent configuration on AWS, the same image was converted from QCOW2 to RAW type, then uploaded to AWS S3, and then converted to AMI using AWS VM Import using one of the methods described in Section II. All hosts were tuned by using ESnet's recommendations [36] for large wide area data transfers as in Table II.
The experiment used iperf3 [37] to test the TCP bandwidth achievable between the baremetal host on Chameleon and each of the targeted AWS instance types over the deployed direct connect circuit. Each instance type was tested using 30 second tests with both single and multiple streams in each direction. Each test was executed 15 times. The average, minimum, and maximum egress and ingress bandwidth are shown in Figure 2.
Other studies have shown that achieving repeatable networking results on public clouds is difficult or even impossible [38]. Although using direct connections eliminates some of the unpredictability of the public Internet it does not change the way public clouds control network performance.
Figure 2 shows the bandwidth achieved between Chameleon and the AWS instances. Each group of columns shows the bandwidth achieved by an AWS instance type. The smallest two instance types were described by AWS as having 'low' and 'moderate' bandwidth limitations. In the experiments, these limitations were effectively 1 Gbps in either direction. The 'low' bandwidth instance achieved less consistent performance and appeared to be the subject of periodic rate limiting. Continuous transfers with the 'low' bandwidth node would result in extremely low rate limiting of approximately 65 Mbps. Each iperf3 test was executed for 30 seconds. In order to avoid artificial rate limiting, the tests were performed 5 minutes apart. Applications requiring continuous 1 Gbps performance should opt for at least 'moderate' network performance.
The CloudConnect circuit across Internet2 was limited to 5 Gbps and the remainder of the instance types should have been able to fully utilize the circuit. Their AWS network limitations ranged from 5 Gbps to 100 Gbps. The experiment attempted to maximize the bandwidth using 1, 2, 4, and 8 parallel network streams. Ideally, the the maximum bandwidth could be achieved with a single TCP stream. However, wide area TCP data transfers are adversely affected by small numbers of dropped packets. The data shows that all instance types achieved network performance near the maximum 5 Gbps. However, the single stream performance increased for higher bandwidth instance types even though the instance's network limitation was well beyond the CloudConnect circuit. Further, the m5dn.2xlarge virtual machine performed as well as the c5n.metal node for egress and slightly better for ingress even though the bare metal node had more memory, more compute cores, and a higher bandwidth limitation.
The reason that a single stream could not achieve the maximum bandwidth on virtual instances that had greater than or equal to 5 Gbps networks was that the DirectConnect configuration studied resulted in more dropped packets for instances with lower network limitations. The dropped packets occurred in both directions and were typically seen in small bursts. As a result, maximum AWS egress and ingress bandwidth could only be achieved with multiple TCP streams or on instances with over provisioned networks. In addition, the average single stream egress bandwidth is higher than the ingress bandwidth for each instance type but the minimum single stream ingress bandwidth is higher than the minimum egress bandwidth. This means that single stream ingress flows are more often affected by small bursts of dropped packets while single stream egress flows are more likely to experience large bursts of dropped packets.
Combining commercial and research clouds significantly expands the resource portfolio available to science, increasing opportunities to build more complex experimental topologies, deploy them on more sites, provide access to more diversity, as well as more scale. While bare metal offerings are few, information about resources and their evolution is not always available, and the user typically has no control over the specific architecture their experiment will map to, not all experiments require these features as exemplified by our case study.
Overall our experiences configuring an experiment spanning Chameleon and commercial clouds available via CloudBank demonstrated that configuring a research testbed as a cloud brought the unexpected benefit of improved portability. While digital artifacts the users produce, such as images and orchestration templates, are not always directly compatible (though we noted ready-made conversion tools for some scenarios), they represent a similar structure, captured by such tools as e.g., use of cloud-init for SSH key injection. Converting them to achieve consistent configuration on experiment resources deployed on Chameleon and AWS is thus a relatively simple matter of updating well-known qualities rather than rebuilding them from scratch. In addition to convenience, this also addresses an issue of consistency and ultimately also reproducibility of how exactly a configuration can be or was repeated in a different setting. Similarly, configuring networking relied on the existence of similar concepts at edge on both ends of the connection Another issue distinguishing research and commercial clouds is the cost to the scientific community. To create a rough comparison of cost we first paired the resource types available on Chameleon with AWS instances that are comparable or less powerful to create a conservative estimate. For example, our Haswell compute nodes [39] were paired with "c5d.metal" -but our GPU P100 nodes were paired with "p3.8xlarge" even though "p3.8xlarge" are not bare metal as AWS does not offer comparable bare metal resources. We then took actual usage numbers for each Chameleon resource type, computed the number of hours they were used over the first 5 years of the project, and multiplied it by the price of the corresponding AWS resource. The total number of node hours over all the resource types were 5,676,114, representing an estimated cost of $30,353,133 ($49,172,075 including overhead which would normally be applied to this type of purchase though is waived through CloudBank usage). In comparison, the total funding received for Chameleon over this period of time is projected to be $16.6M. Further, our rough estimate did not include the Chameleon KVM cloud, data download or storage, any of the special features or specialized services (like BYOC [40] used in the experiment), or startup costs, all of which would have made the Chameleon resources even more cost-effective. On the other hand, Chameleon provides significantly lesser availability than commercial clouds (a factor partially mitigated by advance reservations described in Section II) and probably lesser reliability; while those are of course desirable to the research application, increasing them would also increase the cost. Thus, this is another area where research and commercial clouds provide different offerings at a different cost.
Investigating methods of building an integrated environment across multiple clouds is almost as old as cloud computing itself [41]. In particular, works like [23], [42], [43] investigate issues of resource management, image portability, and orchestration; our focus is different in that we investigate this issue from the perspective of providing an experimental container for computer science systems research which among others includes management of low-level networking resources, and consider the issue of cost.
Multiple projects also sought to characterize the networking performance of research versus commercial clouds [44], [45]. Our approach pushes this line of investigation further by focusing on low-level networking services like DirectConnect and shows a measure of similarity in the problems.
In this paper, we present a recipe and a set of tools, expressed as a Jupyter notebook, that allows experimenters to construct a class of experiments distributed over research and commercial clouds, specifically the Chameleon testbed and commercial clouds available via the CloudBank project. In doing so, we discuss the different capabilities of these two different types of clouds and show how they can be used to construct experiments spanning both.
Our observations show that the decision to configure Chameleon as an enhanced cloud pays dividends in the context of such experiments. Specifically, similar structure of disk images and orchestration templates used in experiments of this type facilitates portability and maintaining a consistent experimental environment. Further, similarities between the ExoGENI stitchport and direct connects implemented by commercial clouds allow experimenters to leverage the same concepts when constructing wide-area circuits.
Results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation. This work was also partially supported by the U.S. Department of Energy, Office of Science, under contract number DE-AC02-06CH11357.