<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>RT-Bench: A Long Overdue Update</title></titleStmt>
			<publicationStmt>
				<publisher>19th Workshop on Operating Systems Platforms for Embedded Real-Time Applications (OSPERT 2025)</publisher>
				<date>07/07/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10621334</idno>
					<idno type="doi"></idno>
					
					<author>Mattia Nicolella</author><author>Denis Hoornaert</author><author>Renato Mancuso</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[RT-Bench is a framework and community project that aims to establish a unified set of benchmarks with a homogeneous launch and result reporting interface, and with a simple build system. RT-Bench targets academic researchers and industry practitioners interested in understanding the performance characteristics of embedded/real-time systems when tested over realistic use-case applications. To facilitate real-time systems research, RT-Bench is designed from the ground up to include a set of fundamental capabilities such as periodic execution, selectable OS scheduler, and native and multi-architecture performance counters support, to name a few.RT-Bench has undergone continuous improvements and extensions. This paper reviews the most recent additions and features of the framework. Most prominently, these include heap migration, synchronized benchmark release, and experimental support for multi-threaded applications. This contribution includes a tutorial session with template benchmarks to showcase the new features and illustrate the process of integrating new benchmark suites.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head>I. INTRODUCTION</head><p>For practitioners and academics, benchmarking constitutes an essential step in testing and validating their systems regardless of the application domain. Naturally, many domainspecific benchmark suites have emerged over time as a result of independent efforts. On the one hand, this organic growth has offered a broad and diversified portfolio of benchmarks. On the other hand, however, it has caused an inherent fragmentation w.r.t. the set of features, launch/command interfaces, and supported metrics. This lack of a de facto standard primarily hinders reproducibility, as well as productivity and adoption, since practitioners must manually adapt each suite of interest to their needs via a repetitive and time-consuming process.</p><p>Since its release <ref type="bibr">[1]</ref>, RT-Bench <ref type="bibr">[2]</ref> has aimed to reduce these frictions through a rich standardized interface that enables compatible benchmarks to tap into its feature set seamlessly. For instance, the framework allows users to swiftly leverage common timing features, such as periodic execution and reporting of elapsed time, in all the included benchmarks. Not only does RT-Bench focus on features relevant to real-time system benchmarking, but it also provides native support for resource profiling, which has proven helpful in understanding run-time resource requirements and pinpointing performance bottlenecks in more general settings.</p><p>The framework also strives to combine user-friendliness, requiring minimal effort when adapting to new benchmarks.</p><p>To that end, RT-Bench's contributors have maintained an extensive documentation [3] of the framework, ranging from the supported benchmark suites to the framework's APIs.</p><p>This article and its associated tutorial session aim to reiterate RT-Bench's core concepts, mechanisms, and goals while formally introducing the newly ratified features.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>II. WHAT'S NEW?</head><p>Since its first release in 2022 <ref type="bibr">[1]</ref>, RT-Bench has known a relative success within the real-time community <ref type="bibr">[4]</ref>- <ref type="bibr">[10]</ref> <ref type="foot">foot_0</ref> ; calling for many improvements, feature additions, feature revision, bug fixes and code consolidation. This section describes the most prominent changes brought to RT-Bench.</p><p>A. Continuous Back-to-back Executions.</p><p>One of the first features introduced in RT-Bench in an effort to more realistically reflect the behavior of real-time applications was the enforcement of periodic execution. As RT-Bench increased in adoption, however, it was brought to our attention that many scenarios and use cases exist where this mode of operation is not ideal and, in fact, undesirable.</p><p>Typically, these correspond to cases where the role of the benchmark is to create pressure on the system's resources (i.e., bandwidth from IsolBench <ref type="bibr">[11]</ref>). As such, the intermittent activity of interfering workloads that comes from periodic releases and missed deadlines (and consequent job skipping, showed in Fig. <ref type="figure">1</ref>, top part) creates "pressure gaps" that can make empirical worst-cases harder to observe and reliably reproduce. Such gaps are also inconvenient when attempting to collect stable readings of the performance counters.</p><p>Hence, a "continuous back-to-back execution" mode has been introduced (Fig. <ref type="figure">1</ref>, bottom part). It can be enabled simply by omitting a period (-p), ensuring retro-compatibility with existing RT-Bench command options. Note that nothing else changes as the benchmark execution routine is still executed in a loop until a SIGINT is received or until the specified number of tasks instances (-t) has been completed.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B. Heap Migration</head><p>The push toward heterogeneous System-on-Chips is not just confined to processing elements. Modern embedded platforms, in particular, also tend to feature a diversified array of onand off-chip memories with different sizes and temporal characteristics. On Linux-based systems, these memories are often treated as reserved memory nodes. Thus, unless system designers add ad-hoc support, these nodes remain natively inaccessible to standard user-space applications.</p><p>To enable the study of benchmarks' memory affinity, RT-Bench integrates a user-friendly heap migration mechanism. When heap migration is enabled, RT-Bench switches to a custom heap allocator over an internally allocated memory region. The user can control the type of allocation commandline parameters. Specifically, when enabling heap migration (-H), users can indicate the target memory in two ways: <ref type="bibr">(1)</ref> by providing a physical memory address; or (2) by providing a file pathname.</p><p>In the former case, RT-Bench will perform a non-cacheable memory mapping using /dev/mem and will use said mapping as the applications' heap memory region. In the latter case, RT-Bench will memory-map the provided file. If this is a normal file (e.g., in a tempfs filesystem), the resulting memory region will be cacheable. This option is handy as reserved heterogeneous memory nodes can be exposed to user space by exporting them as block devices that can be memory-mapped. Kernel-level support for the creation of said interface, however, is out of the scope of RT-Bench. When using the (-H) option in either mode, the end-users are responsible for providing adequate access to the underlying memory (e.g., by ensuring it is a usable physical address/file) to avoid bus errors.</p><p>Note that a heap size limit parameter (-m) is required because RT-Bench must terminate the target benchmark if the working set size exceeds the size of target memory region.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C. External Synchronized Release</head><p>As RT-Bench runs atop of a rich full-fledged operating system (i.e., Linux), running several benchmarks concurrently to observe and study their interactions is already possible. However, no control over the release time of the workloads is usually possible. This makes the simulation of specific conditions, such as the worst-case job release pattern, hard to perform.</p><p>Thus, an initial release (offset) synchronization mechanism has been added to RT-Bench. This mechanism, shown in Fig. <ref type="figure">2</ref>, allows different benchmarks or different instances of the same benchmark to be released together via the newly introduced (-s[=TASKSET_NAME]) parameter. When the parameter is used, the released benchmarks behave as if they belong to the same task set with synchronized release offset. Specifying a task set name is optional. If an explicit name is not provided, RT-Bench will default to an implicit name.</p><p>Practically speaking, RT-Bench's synchronization unrolls in three phases. Initially, all workloads initialize themselves until they are ready to execute for a first iteration. There, they wait to receive a synchronization signal (i.e., SIGUSR1). This behavior allows users to add an arbitrary number of benchmarks to the set by simply starting them with the same (-s) parameter value. Once all the desired benchmarks have been launched, the user can start the full experiment by sending the SIGUSR1 signal to any one of the benchmarks in the set. Once any of the benchmarks receives the signal, said benchmark becomes the synchronization manager. The manager is endowed with the responsibility to (1) determine a synchronized release time in the near future and (2) send a signal to wake up all the other benchmarks (subordinates). Upon reception of the signal, each subordinate benchmark, as well as the manager, will configure an absolute release timer with the exact coordinate time determined earlier.</p><p>Note that the mechanism still involves the end-users as they must manually start the required benchmarks and choose a unique task set name (TASKSET_NAME). They also must send the first SIGUSR1 signal once all the desired benchmarks have been launched. Alternatively, a synch-helper tool is also provided which allows users to specify how many benchmarks to wait for before the start SIGUSR1 signal is sent.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D. Extended Reporting of Benchmark-specific Metrics</head><p>For some benchmarks, the default performance metrics reported by RT-Bench are only a subset of the information useful for the experiment at hand. Since its initial release, RT-Bench offers the possibility to report one benchmarkspecific metric for each benchmark. This feature is used, by the applications in the IsolBench suite <ref type="bibr">[11]</ref>, which also report measured memory latencies and bandwidths. However, in a number of cases, limiting benchmark-specific metric reporting to a single value was deemed too restrictive. For instance, a benchmark performing neural network inference for object identification might want to simultaneously report the number of detected objects and the corresponding confidence scores.</p><p>For this reason, the benchmark-specific reporting mechanism has been revamped. In this revision, benchmarkspecific metrics and metadata are stored in a struct named extra_measurement aggregating pairs of headers and metrics. At run-time, the struct is populated by the user-defined benchmark_log_data() routine with the desired metrics and their human-readable name. Accordingly, the EXTENDED_REPORT compilation flag has been deprecated.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E. Performance Monitoring Thread Improvements</head><p>Recall that RT-Bench included the option to spawn a performance monitoring thread PMThread. The PMThread runs in parallel with a given benchmark, and its goal is to perform high-frequency sampling of architectural performance counters to more accurately profile the benchmark under analysis.</p><p>An interesting challenge in the design of the PMThread is how to store the trace of performance samples efficiently. In the initial version, sample measurements 2 were stored in one contiguous container (i.e., a buffer) where each entry correto a timestamp in the execution. For a time bucket &#8710; t , the i th entry contains the performance measurements sampled at instant i&#215;&#8710; t of the benchmark run-time. Such pre-allocated container would limit the execution time of the benchmark and eventually lead to a buffer overflow.</p><p>The reworked implementation, instead, features support for a list of buffers that can hold these measurements. This is achieved dynamically during the execution phase as follows. Whenever a buffer is filled, a new buffer is allocated and added to the list, with a size that is twice as large as the previous buffer in the list. The approach limits the number of memory allocations during execution while not copying/moving any previously acquired data samples. Finally, any memory used to hold performance counters is not tracked by the memory watchdog, and thus, it does not count against the benchmark's memory limit (-m).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>F. Experimental Multi-thread Support</head><p>The feedback following the previous release of RT-Bench lamented the lack of support for multithreading. The absence was justified by several-feasible but tricky-implementation hurdles such as synchronization and core affinity assignment.</p><p>Since then, steady progress has led to the introduction of an experimental support for multithreading. It is now possible to easily use the framework with a multithreaded benchmark since new APIs to (1) spawn new threads and (2) define a parallel computation section with these threads are available. With these APIs the main execution thread can operate before, during and after the parallel section; with minimal changes to RT-Bench's execution logic as shown in Fig. <ref type="figure">3</ref>.</p><p>Note that, at the time of writing, multi-threading cannot be used in conjunction with other options. For instance, as the subsystem used for heap migration and Working Set Size (WSS) reports are not thread-safe, multi-threading can only be used provided no dynamic memory allocations are performed during the execution phases. Likewise, performance counter reports generated by the PMThread only cover the main thread.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>III. DEMONSTRATION SESSION OUTLINE</head><p>This section provides a step-by-step overview of the tutorial session accompanying this paper. The objective is to 2 i.e., a struct aggregating a selected set of PMCs such as L2_refills. demonstrate the user-friendliness of RT-Bench and provide the audience with key pointers to get started with their projects. 1. Building and Launching a Benchmark. Using a project template <ref type="bibr">[12]</ref> specifically made for this occasion, we will showcase how RT-Bench-adapted benchmarks can be compiled and executed. In particular, we will focus on the required command line parameters, scenario configuration via .json files, and where to find built binaries. This part will cover how to release a taskset, a heap migration example, and how to retrieve performance measurements, including an example using the PMThread. 2. Adding a Benchmark. This part will showcase how to extend RT-Bench with additional benchmarks <ref type="bibr">[13]</ref>, whether single-or multi-threaded. The inspection of the single-threaded benchmark from the template will serve as a stepping stone to revise RT-Bench's basics. Specifically, it will include how to split a main function into the three required harness functions (i.e., init, execute, and teardown) and tips on how to manage any inputs modified in place during execution. This part of the demo will also cover how to add custom logged metrics. Finally, the steps that must be undertaken to tap into the experimental multi-thread support will be shown by adapting the benchmark at hand.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>IV. CONCLUSION &amp; FUTURE WORK</head><p>RT-Bench benefits from a continued effort in maintaining the project and adding new features. Future work is directed towards (1) adding multithreaded benchmarks and improving the support for multithread execution, (2) supporting C++ benchmarks, and (3) the ability to define and release a taskset from a single .json specification. Other future work includes the construction of a RT-Bench database, which contains metrics of different benchmarks categorized by platform.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0"><p>For sake of transparency, cited papers exclude research items involving the authors of the original paper<ref type="bibr">[1]</ref> and their close collaborators.</p></note>
		</body>
		</text>
</TEI>
