The intriguing opportunities for real-time deep learning (DL) applications at the edge clash with the harsh reality that computation, communication, and accuracy are limited with most edge devices. A wide variety of video analytics workloads in surveillance, industry, UAVs, IoT, and AR could effectively leverage advances in DL if not for the severe resource constraints faced with edge computation [1]- [5]. Realtime workloads are especially sensitive to these constraints due to tight end-to-end latencies often imposed on DL inference results. Offloading these tasks can mitigate the challenges of running these workloads on edge devices [6].
To maximize throughput, DL inference latency needs to be minimized. However, edge devices, characterized by limited resources [7], may not be able to process frames with a reasonable latency or frame rate. With a typical frame rate of 30 frames per second, there are about 33 ms between frames. However, depending on the DL model and the computing resources of the edge device, inference for a single frame can take hundreds of milliseconds to process [8].
A compelling solution to this problem is offloading [6], [9], [10] where we can leverage the resources of a nearby edge node or server. An edge server, less restricted by resource constraints, can accelerate DL inference throughput, especially when equipped with a GPU [11]. However, a single device's video stream may under-utilize modern hardware and fragment expensive hardware resources [12]. Leveraging multi-tenancy by allowing many devices to offload can efficiently utilize the server.
Real-time and multi-tenant constraints make offloading less flexible. When there is continuous demand for inference results, quality of service (QoS) becomes sensitive to endto-end latency [13]. Multi-tenant offloading helps improve server resource utilization, but saturating server resources compromises QoS due to higher service latency [14]. Variable network conditions, device variability, and unpredictable system load make end-to-end offload latency unstable [15]. A reactive offloading policy for this domain must be developed and tuned to maximize frame rate without violating latency constraints [16].
We propose a novel system that addresses the challenges of multi-tenancy, constrained local resources, limited offloading availability, and variable network conditions for real-time DL workloads that operate on fast and concurrent video streams:
To address these constraints, we use the rate of end-to-end latency violations as a critical metric in determining the suitability of an individual edge device's offloading policy. With little to no timeouts, offloading can be scaled up (and subsequently, as the offloading rate approaches the source frame rate, the local inference rate can scale down). We propose a novel real-time feedback control system, FrameFeedback, to regulate the offload rate. FrameFeedback can provide stable control under nominal conditions and reconfigure policies when QoS violations occur [17].
Our work makes the following contributions. 1) We identify the factors impacting the QoS in real-time DL applications at the edge and under multi-tenancy. We demonstrate that these factors are not taken into account in state-of-the-art approaches for total or partial offloading. 2) We use end-to-end latency violation rate as a new metric for evaluating offloading policies.
TABLE I: Notation Notation Description Fs Source frame rate P Total inference processing rate P l Local processing rate Po Offloading rate L Maximum tolerable offloading latency T Rate of offloaded frames that time out Tn Rate of offloading timeouts due to network T l Rate of offloading timeouts due to server load
TABLE II: P l of our different Raspberry-Pis 3B Rev. 1.2 4B Rev. 1.2 4B Rev. 1.4 CPUs 4 4 4 Speed 1200 MHz 1500 MHz 1800 MHz Memory 909 Mi 3.7 Gi 7.6 Gi MobileNetV3Small P i 5.5 13 13.4 EfficientNetB0 P i 1.8 2.5 4.2
3) We present FrameFeedback, a novel closed-loop control system that uses system feedback to set an optimal offload policy under current system constraints.
FrameFeedback is available as open source software for Raspberry Pi and NVIDIA GPUsfoot_0 . 4) We evaluate FrameFeedback on a realistic edge system against baseline approaches and demonstrate its superiority under suboptimal server load and network conditions, where FrameFeedback outperforms a stateof-the-art system (DeepDecision) by more than a factor of two. The rest of this paper is organized as follows. Section V reviews related literature. We describe the details of the FrameFeedback system and controller in Sections II and III with evaluation following in Section IV. We discuss future work in Section VI.
In this section, we provide background for FrameFeedback and discuss the constraints that multi-tenant edge AI systems operate under. We also discuss our system configuration. We refer the reader to Figure 1 to visualize our system and the specific interactions between devices and servers. Table I provides the relevant notation.
We consider an environment that delivers AI services to users through user mobile edge devices, edge servers, and wireless networks. We seek resource allocation approaches that address the following combination of challenges and constraints:
1) Multi-Tenancy: We consider a system where users cannot have dedicated resources to offload AI tasks [18]. The combined load from all user devices also impacts the server's prediction time, so we cannot assume that our only source of offloading latency comes from networking. We also consider multiple classification workloads with different computational costs, latency, and quality requirements.
2) Constrained Local Resources: Edge devices are resource restricted, whether due to power limits, weak hardware, or lack of accelerators [19]. For the specific application domain that we study in this paper -object classification and identification from real-time video streams at high frame rates, Table II provides examples from our Raspberry-Pis running classification on frames of size 224×224. Referring to Table I, our system assumes that, in all user devices that capture video, P l < F s . In other words, the local processing rate, P l , is slower than the source frame rate F s .
3) Limited Offloading Availability: Specific workloads may saturate a server, thus causing QoS violations [20]. When the workload fully saturates the system, the system should respond by reducing offloading and distributing the available capacity fairly among clients. Our notation represents the load-induced time-out rate (and rejections) as T l .
4) Variable Network Conditions: Most edge devices connect to the network wirelessly. Movement and sources of interference can make connections unreliable. Bandwidth limits, packet loss, or other network delays cause complex end-to-end offload latency patterns that can be difficult to predict [21]. Our notation includes T n for network-induced timeouts that we consider in addition to T l .
5) Quality of Service: QoS signifies how well the edge device can complete time-sensitive tasks enabled by DL inference results. Offloading is necessary to maximize task processing time (P ), but our system assumptions cannot guarantee successful offloading. Therefore, when calculating P , the system must account for T and be careful not to offload excessively when T is high. Consider if P o = F s and T > F s -P l at some point. This means that the device attempts to offload all its frames, but the effective value of P is lower than P l . The system should never fail to react to this scenario, and the controller should always strive to keep P ≥ P l .
TABLE III: Top-1 Model Accuracy [27], [28] Model Top-1 Accuracy EfficientNetB0 77.1% EfficientNetB4 82.9% MobileNetV3Small 67.4% MobileNetV3Large 75.2%
We neither measure nor optimize power usage with our QoS model, but note that, in general, effective offloading leads to lower power usage on edge devices [6], [22]. Our experiments show that Raspberry Pi CPU usage drops from 50.2% to 22.3% on average when transitioning from local execution to offloading.
We consider two categories of delay: multi-tenant contention and networking delay. The number of connected clients and their respective offloading request rates dominate the system load at the edge server. As the load increases, the time to execute each batch of DL inference increases, and interprocess communication increases. ATOMS [23] considers many of these factors but at the cost of an exhaustively complex system model that requires many different subsystems, including a device-server clock synchronization and a resource reservation system, to operate.
One benefit of FrameFeedback is that it does not need to directly measure or understand the sources of the delay, since it properly reacts when the delay arises. We make two observations: First, an offloaded inference task is successful if its result returns before its deadline. On a per-frame basis, the only relevant measure of QoS is binary. Second, the controller's reaction to T depends entirely on T "s value, and any changes made to P o should be dependent on T .
In this work, we consider 250ms as a justifiable deadline for a real-world, real-time video processing system. This assumption aligns with recent work on video analytics [24], [25]. Furthermore, we consider pipelined offloading to overlap frame processing for higher throughput.
We use Image Classification models from the Keras [26] framework to model our system. We chose MobileNetV3 [27] and EfficientNets [28] because they are fast, well documented and applicable to many domains.
Table III presents details on the accuracy of our selected models. These accuracies may be lower than those of nonconstrained models, but are consistent with models previously used in edge computing [29].
One way to increase the accuracy of the model is to improve the information in the classified image. The default resolution with which all of these models have been pre-trained with is 224×224 except for EfficientNetB4, which accepts images with a resolution of 380× 380, and allows for a variable input size. Most modern cameras generate images with larger resolutions and resize them to fit as input to classification models, so using a larger resolution closer to the source could improve accuracy. When offloading images for classification, it is common to compress them [30], [31]. Using lighter compression can improve accuracy. Both techniques have a significant downside when offloading, because both increase the number of bytes per frame that need to be transferred.
In this section, we describe the FrameFeedback controller covering the foundations of the controller, showing our necessary modifications, and justifying the controller's settings. We adapt the basic theory of a closed-loop PID controller to show that a modified and tuned controller is a good choice for our problem domain.
A closed-loop or feedback controller applies continuous control to a system by measuring a process variable (P V ) and applying a correction such that P V converges to a set point (SP ). A Proportional-Integral-Derivative (PID) controller computes an error e(t) where:
And a control function:
where K P , K I , K D are the proportional, integral, and derivative coefficients, respectively [32].
FrameFeedback uses a discrete feedback controller to determine a suitable offloading rate for each edge device using measured information such as P o and T . When T is zero (no offloading latency violations or dropped requests), offloading will likely continue or increase.
1) PD Controller: For our controller, we observe that the integral term of conventional PID controllers is unnecessary. The characteristics of this term, the consideration of the past, and the variable external forces on P V , are not factors in our system. Since our controller's input is the average of T from the last few seconds, we already consider past data and show that P V does not experience variable forces.
Therefore, K I = 0, and we modify Equation 2 to:
Our goal of maximizing the processing rate by maximizing available offloading means that our error function e(t) needs to consider two cases:
1) If T is low (≤ 10% of F s ), e(t) needs to be positive 2) If T is high, e(t) needs to be negative These two cases are challenging to capture with a simple P V function. A balanced approach requires considering each case separately with a piecewise P V : Fig. 2: Offloading rate P o for controllers with different K P and K D coefficients. We introduce packet loss of 7% after 27 seconds.
In this approach, we set SP = F s , so e(t) is linear in both cases.
Our PD controller will drive the offloading rate to F s , and in the presence of timeouts exceeding 10% of F s , it will scale back the offloading. Note that e(t) = 0 when T = 0.1F s , so P o will stabilize to 0.1F s when offloading always fails. This does not negatively impact QoS when compared to not trying to offload at all, but it does provide a constant measurement of offloading availability. Therefore, when good conditions return, offloading will immediately begin to increase.
A traditional tuning approach, such as the Ziegler-Nichols method [33] offers good intuition for tuning our controller. Still, their exact method cannot be applied because it is designed for a full PID controller. Furthermore, due to the changing network and system load conditions, our P V function demonstrates more variance than those they considered.
The procedure to adjust our PD controller was to gradually increase K P until the controller sensitivity was high and the P V oscillated under constant conditions. Next, we increased K D to reduce the oscillations and stabilize the system. This follows from the wisdom that increasing K P increases sensitivity while degrading stability, and increasing K D decreases overshoot and improves stability [32].
To introduce an additional tuning layer into the controller, we limited u(t) to a specific range of updates. To make the
TABLE IV: PID Settings Variable Value K P 0.2 K I 0 K D 0.26 Update minimum -0.5 * Fs Update maximum 0.1 * Fs Measure Frequency 1
controller sensitive to when T is high, the minimum update to P o is -Fs 2 . However, we did not want the controller to react too quickly to increase P o (the offloading rate), so we imposed a maximum update of Fs 10 when increasing P o . These limits result in the controller improving QoS by reacting more forcefully to timeouts.
Table IV displays the settings of our controller. Figure 2 shows the effect of different settings on the responsiveness and stability of our controller. At first, we see the controllers' behavior under ideal network conditions, and once we introduce packet loss, we can observe how the settings affect stability.
We used the Raspberry-Pis detailed in Table II as our edge devices. For the collection of the data shown in Figures 2, 3 and 4, we use the three Raspberry-Pi's concurrently sending streaming requests to our edge server and evaluated their total inference throughput. We use MobileNetV3 for these tests because it produces the smoothest results. To help reproduce results, we only used the same device and model for data collection. TThe Raspberry Pi runs Debian Linux 11 (bullseye) on kernel 5.15.84-v8+. It has four Cortex-A72 arm64 CPUs, a heat sink, and a fan. We wrote the entire Pi code in Python and used Python 3.9.13. Our Keras models run through TensorFlow [34] 2.8.0, which we built specifically for the Pi's CPU architecture.
Our edge server runs on an Ubuntu 20.04.5 virtual machine inside of KVM. It has 16 EPYC 7251 CPUs. It has a Tesla V100 GPU running CUDA version 11.8 passed through from the host machine. We use Python 3.8.13 and Tensorflow version 2.9.1.
This configuration is realistic, as hardware and software are widely used in real-world deployments and have been used for evaluation in related work [23].
We did not observe significant differences in throughput or CPU usage when testing with an actual webcam versus sourcing frames from the ImageNet dataset. For convenience and reproducibility, we use ImageNet frames for evaluation.
Batching can help maximize throughput and hardware utilization, especially when using GPUs with Image Classification models, which are notorious for their low hardware usage [35]. We adaptively vary our batch size based on the request volume to take advantage of batching. ATOMS [23] follows the same approach. Since we have a constant stream of incoming frames, our batching scheme can be simple: construct a batch using all frames (to a limit) that arrived while executing the previous batch. We maintain a request queue that is filled during the execution of a batch, and we fill the next batch with the contents of this queue. As noted, we cannot allow the batch size to grow too large, so we impose a limit of 15 frames for each batch, while rejecting the rest in the queue.
In addition to FrameFeedback, we evaluate the following controllers:
1) Local Inference: The first of the baselines we compare our FrameFeedback controller with is the local execution only. This is an undesirable solution due to the low throughput and high power usage of computing Image Classification on Raspberry Pis.
2) Always Offload: This baseline is self-explanatory. At all times, we offload all frames to the edge server. Since we disregard any feedback, it is unlikely that this solution will be optimal unless the system conditions are perfect.
3) All-or-Nothing Intervals: Mimicking DeepDecision's [30] approach, we decide at each measurement step (1 second) whether to offload all frames in that interval or to classify frames locally. To make this decision, we follow DeepDecision's intuition and try to keep track of the system state by sending a heartbeat request to profile the latency. If the request is successful (returns before the deadline), we deem the conditions sufficient for offloading.
We can alter two main system variables to induce timeouts and measure throughput. Changes in the network and server load can induce timeouts in different patterns, and we can change both in different ways.
1) Network: Our primary method for degrading the network connection between Pi and the server is by limiting the bandwidth and introducing packet loss, which has a realistic impact on real edge networks [21]. To artificially inject rate limits and loss, we used NetEm [36], a Linux network emulation tool that allows us to fine-grained control over the emulated network. We could have also used NetEm to add a latency delay to packets, but we believe that rate and loss are better tools to induce timeouts as they are more indirect. Additionally, NetEm delay calculations may not be accurate [36].
To ensure that the rate of deadline violations is not affected by excessively high or low rate and loss values, we decided to limit the bandwidth between 1-10 kbps and set the loss rate to 7% for this experiment. It should be noted that this loss rate may be low for some wireless networks, which can experience packet loss rates in the tens of percentage points [37].
2) Server Load: The second method of injecting latency is by using a high load on the server. While measuring on one device, we can use other devices to put a higher demand on the server. This allows us to measure the system behavior under high multi-tenancy.
Queuing delays, interprocess communication, and increased batch sizes increase the latency of each request. At some level of request volume, batch sizes will saturate the system, leading to rejections. However, batch size limits are set per model, so we hit both model types when measuring controller response under server load.
Combined Network and Server Measurements: Combining both sources of end-to-end latency largely works additively to create more unsuccessful offload requests. Combining the two sources of additional latency can generate various timeout patterns of interest. Due to space considerations, we do not discuss these patterns further here.
TABLE V: Network Variables Time(s) Bandwidth (kbps) Loss (%) 0-30 10 0 30-45 4 0 45-60 1 0 60-90 10 0 90-105 10 7 105+ 4 7
To compare FrameFeedback to the baselines under ideal and degrading network conditions, we subjected each controller to the same tests. Generating a stream of 4,000 frames at 30 frames per second, we configure NetEm to alter the network conditions at predetermined intervals. Refer to Table V for specific values and to Figure 3 for data.
The first thing to address is that P , the successful inference rate, is noisy for all controllers. This is due to the discrete nature of the frames, which are generated at fixed intervals. Unpredictable and random timeout rates subtract from P , making it noisy even though an average trend is visible. TABLE VI: Server Load Configuration Time(s) Request Rate 0-10 0 10-20 90 20-35 120 35-50 135 50-60 150 60-75 130 75-90 120 90-100 90 100+ 0
We include P o for FrameFeedback to demonstrate its behavior. The dark blue dots represent P o + P l -T and represent the throughput.
Under very high or low network quality periods, Frame-Feedback and all-or-nothing intervals have equivalent throughput. However, we can see that under intermediate network conditions, FrameFeedback has a higher throughput because it can find an offload rate that the current conditions can support. Especially around 40 seconds and beyond 90 seconds, FrameFeedback has a better average P (between 50% and up to 3×) compared to the all-or-nothing approach. Clearly, the only-offloading strategy is suboptimal.
Our testing structure for measuring the effects of multitenancy and server load follows our network testing protocol. We generate a stream, at 30 frames per second, of 4,000 frames, but instead of using NetEm to degrade the network, we use other devices to inject request volume. Refer to Table VI for details.
Figure 4 contains the same structure as Figure 3 in that we show the total throughput P for each baseline controller, and for FrameFeedback, we show P and P o . Beginning with zero, we increase and then decrease the external offloading at identical intervals for each controller. Up until about 150 additional requests, our Pi can fit in some offloading when controlled by FrameFeedback. The other controllers have lower throughput due to their inability to adapt in a finegrained way.
In this Section, we review different approaches to real-time DL offloading at the edge.
Initial works on running DL-based video analytics workloads solely on edge devices show poor performance. Most combinations of devices and models result in inference latencies of up to 2000 milliseconds [38] for data sets like those explored in this paper. This prevents meaningful service when processing real-time video streams.
Some works rely completely on an external edge server to compute real-time DL results. In a single-tenant system, the work in [39] adapts the level of offload based on network conditions. Another single-tenant system presented in [8] describes an object tracking application where the detection stage uses off-loaded DL while computing frame-by-frame tracking locally. Multi-tenant offloading of multiple devices' video streams is explored in [40], but this work lacks exploration of QoS degradation due to networking or total system load.
Considering the constraints of a multi-tenant real-time offloading system, [23] describes a reservation, planning, and scheduling system called ATOMS. Since it requires computing time estimation and clock synchronization, the system is VI. complex but robust. ATOMS considers resource contention for CPU workloads and GPU-supported DL workloads, but it lacks rigorous evaluation under variable network conditions and is not an applicable solution for our system.
Adaptive partial offloading is a promising way to balance system constraints such as network bottlenecks and server load with local device processing. OsmoticGate [31] coordinates edge devices with edge nodes and can adaptively offload DL workloads to the cloud. Although local processing on edge devices is not actually performed, the constraints considered for edge node processing versus cloud offloading are similar to local versus edge node processing. OsmoticGate considers frame chunk size and rate for varying system conditions. DeepDecision [30] uses all-or-nothing adaptive offloading, where it can offload all frames from a video stream to an edge server if it detects suitable conditions. If latency or power thresholds are exceeded, DeepDecision can adjust the offloading policy or compute the results on the device. Neither OsmoticGate nor DeepDecision adequately describes a way to generate an offloading policy given the constraints of our realtime and multi-tenant edge video analytics.
We presented a system to improve the service of real-time video DL applications in a constrained and variable edge environment by adaptively adjusting offloading. FrameFeedback is an end-to-end latency-sensitive closed-loop feedback controller. When an edge server supports offloading for many resource-constrained devices, FrameFeedback's custom control function guides it toward maximum throughput. Frame-Feedback reacts to degradation in system-wide offloading capacity better when compared to baseline approaches.
https://github.com/mnj98/edge-inference.
Authorized licensed use limited to: to IEEExplore provided by University Libraries | Virginia Tech. Downloaded on March 10,2025 at 16:53:02 UTC from IEEE Xplore. Restrictions apply.