With the rapidly increasing computing capability and a huge consumer market, modern commodity smartphones have become a popular platform for the emerging computationally intensive virtual reality (VR) applications [29,43]. These applications can be seamlessly integrated with the recently released VR head mounted display (HMD) mounts, such as Google Cardboard [15], Google Daydream [16], Samsung Gear VR [38], DODOCase [37], and Archos VR Glasses [2]. Moreover, smartphone-based HMDs have enabled a brand new interface for presenting immersive video (a.k.a., 360degree video) content in the 360 degree of freedom controlled by a user's head movements. Such immersive video streaming provides users with an enriched viewing experience as if they were an integral part of the video and enables signicantly improved quality of experiences (QoE) as compared to the traditional 3D or high denition 2D videos [24].
However, the improved QoE provided by the immersive video comes with signicant costs, such as high bandwidth consumption and performance overhead while streaming the 360-degree video frames [4]. Since the emergence of immersive streaming applications, there have been many research eorts focusing on reducing the bandwidth consumption by employing view-based optimizations [3,18,34,35]. However, the community has not fully investigated the power perspective of immersive video streaming. Power consumption is a critical problem in immersive streaming for two key reasons. First, the smartphone-based HMDs are driven by power-constrained batteries. Second, intensive power consumption can accumulate heat that would signicantly impact the viewing experience of HMDs users due to the device's wearable nature. This, in essence, makes power consumption an integral part of the QoE.
Although power optimization techniques have been proposed for traditional 2D videos on smartphones [8,19,25,52,53] and wearable devices [23], these techniques cannot eectively reduce the energy consumption of immersive streaming on smartphone HMDs. This is mainly due to the unique workload and power prole of immersive streaming, described as follows. First, the volume of video data in immersive streaming is huge (i.e., 6X to 8X of the traditional video [40]), as the entire 360-degree frames must be transmitted and processed. This incurs signicantly higher power consumption of network and computation, thus leaving a large room for further optimization even after the traditional power optimization techniques are applied. Second, dierent from traditional video streaming, immersive streaming is a user-centric video application, as it grants the viewers full control over the view angles via head movements and generates the viewport from the 360-degree frame on the smartphone upon each movement. Consequently, frequent user movements would trigger non-trivial power consumption in sensing, computation and view generation, which is not considered by the traditional power optimization techniques. In summary, a new and customized power management mechanism is essential in achieving power eciency in immersive streaming.
In this work, we investigate the problem of reducing the power consumption in immersive streaming systems. To address the aforementioned challenges, we rst conduct a quantitative power measurement study (discussed in Section 3) of immersive streaming on commodity smartphones. Our measurements indicate that the VR view generation operation consumes signicant power and is the topmost power consumption source. Based on this observation, we design a quality-aware frame rate adaptation mechanism to reduce the power consumption. Our key idea is to reduce the frequency at which the VR views are generated, i.e., reducing the frame rate of immersive streaming dynamically. We consider the eect of frame rate reduction on the perceivable video quality by leveraging an objective and quantitative video quality metric called spatio-temporal video quality metric (STVQM) [33]. This metric correlates the perceivable video quality with the frame rate and has been proved to be consistent with the subjective quality metric (the mean opinion score (MOS) [45]). We further leverage one of the unique characteristics in immersive streaming, namely user-initiated view switching, in the power optimization mechanism by following two key design principles. (1) No frame rate reduction during xed view. The mechanism maintains the original frame rate when viewers are not switching views and only reduces the frame rate during view switching. The rationale behind this principle is that, during a view switching process, the viewer's attention is typically not at the view being switched but rather the view being switched to and, therefore, the reduced frame rate during switching has limited impact on the perceivable video quality. (2) Quality-aware frame rate selection during view switch. The mechanism selects the optimal frame rate to minimize power consumption under the video quality constraint based on the STVQM metric.
We incorporate the above two principles and implement a new frame rate adaptation mechanism called QuRate for smartphonebased immersive video streaming, which optimizes the power consumption in a quality-aware and user-centric manner. QuRate monitors the user movement pattern at runtime and determines the most power ecient frame rate while maintaining the perceivable video quality. Furthermore, to reduce the runtime performance and power overhead introduced by QuRate itself, we develop an oine/online hybrid execution model. In the oine phase, we build a frame rate library (FRL), which quanties the correlations among quality, frame rate, and head motion, through power/quality proling based on historical user data. In the online phase, the library FRL is used to determine the instant frame rate based on the dynamic head movement and the quality constraint. We evaluate the eectiveness of QuRate by using real user head movement data and measure the power consumption of immersive video streaming using ve commodity smartphones. Our evaluation results show that QuRate can extend the smartphone battery life by up to 1.24X while achieving satisfactory video quality based on a real user study.
To the best of our knowledge, QuRate is the rst power optimization framework for smartphone-based immersive video streaming that considers both user behavior and video content. To summarize, we have made the following contributions.
• We for the rst time identify the unique problem of power consumption ineciency in immersive video streaming based on an empirical power measurement study. The observed ineciency can be attributed to the unique characteristics of immersive streaming which are not considered by the traditional video power optimization techniques.
QuRate that addresses the aforementioned power ineciency problem for immersive streaming. QuRate takes into consideration both the unique user behavior and video content features in immersive streaming to achieve power-ecient frame rate adaptation with minimum video quality impact. • We evaluate and justify the signicant power savings and minimum video quality impact achieved by QuRate. Our comprehensive set of evaluations include empirical evaluations based on empirical user head movement traces from a publicly available dataset, as well as an IRB-approved user study.
Virtual reality technology can generate three-dimensional virtual environments emulating the physical world, which provides the users with an immersive experience [7]. It is widely used in many areas, such as gaming [36], healthcare [6], and entertainment videos [17,35]. In a typical VR setup, the user wears a HMD device that displays the specic view based on head movements, similar to what one would see in the physical world. Among all the VR applications, immersive video streaming has naturally become a hot spot because of the popularity of video streaming in the consumer entertainment market [17,35]. For example, there are currently millions of immersive videos available on YouTube, the number of which is rapidly growing on a daily basis [51]. In particular, immersive video is attractive in scenarios like live broadcasts of sports games, in which the viewers can switch their views based on their own preferences, as if they were watching the game in person in the stadium [30]. Figure 1 shows a typical end-to-end workow of an immersive video streaming system, following the ISO standard for Internet video streaming, namely Dynamic Adaptive Streaming over HTTP (DASH) [42]. The end-to-end system follows a client/server architecture. On the server side, the video packager partitions the source 360-degree video into DASH compliant segments [42], which are deployed on a web server for HTTP streaming. On the client side, the web browser eyes in observing moving object. Normally, when the velocity of an object is larger than 20 degree/second, the gain (i.e., the ratio between eye velocity and object velocity) can no longer maintain in the range of 0.9 to 1.0, which is required for the human vision system to observe the object clearly [14]. In this case, corrective saccades, a compensation mechanism that combines head and eyeball movements, is needed to realign the target. However, according to [44], the possibility of error in corrective saccades is 29% -79% depending on the environment, which means corrective saccades is highly unreliable and the eyes would still have blurred vision while viewing a fast moving object.
Based on the above evidence and the scientic discovery from the biology eld, reducing the frame rate of VR video in a reasonable range and while the user's view is fast switching would pose insignicant impact to the user experience, because the view is already blurred to begin with. This key observation serves as the basis of our frame rate reduction method for power optimization, which we present in details in the next subsections and further justify using subjective user studies in Section 5.7.
Frame Rate Adaptation. For a premium viewing experience, the frame rate of immersive video is typically 60 FPS. Since a large amount of computation must be conducted at the rendering of each video frame (e.g., read the viewer's orientation, locate the eld of view within the 360-degree frame, and generate the left and right views for the viewer's eyes), it leaves large room for power savings by reducing the frame rate (i.e., the frequency that the VR view is generated). However, since a reduced frame rate may signicantly impact the video quality, we only conduct such reduction while the user is switching views. Our intuitions are two-fold. First, the video scene during fast view switching will be low quality to begin with based on the discussions in Section 4.2.1; Second, the video quality during view switching is non-critical to the user experience, as it is an indication that user is interested in the new view. Taking a 360-degree soccer video as an example, the user would focus on a xed view, such as two players grabbing the soccer ball from each other. Then, when the ball is passed through a wide range, the user's attention will switch and track the ball until it reaches another xed view. During the switching, i.e., while both the user's orientation and the ball are in motion, the quality of the video and thus the frame rate is much less critical to the user's experience, which can be reduced without compromising the QoE.
Based on this observation, in QuRate, we maintain the original frame rate while the view is xed (i.e., motion speed below a noise threshold) and only reduce the frame rate when the user switches from the current view to a new view. The frame rate reduction mechanism is shown in Algorithm 1, which employs the Motion Detector to determine whether the frame rate should be reduced.
Despite its obvious eectiveness in power savings, it is well known that frame rate reduction would degrade the quality of the video if not well controlled. Therefore, we must quantitatively evaluate the quality loss due to frame rate reduction and develop a systematic approach to minimize it. As the rst step in achieving this NewViewPoint; 14: end goal, we adopt an objective video quality metric, namely spatiotemporal quality metric (STVQM) [33] to evaluate the quality of the immersive video under frame rate control, which considers the interactions between spatial and temporal quality perceptions:
where a and b are constants determined by a least-square non-linear tting using the subjective data, which leads to a = 0.028, b = 0.764; FR refers to frame rate; and SVQM (spatial video quality); TI (temporal information) and SI (spatial information) are calculated as [46]:
) In Equation (4), std space stands for the standard deviation of the pixels in one video frame, Sobel(F n ) refers to the pixels in the video frame at time point n after being ltered with a sobel lter [41]. M n (i, j) in Equation (3) refers to the pixel dierences between the frames in the user's view of time points n and n 1 at position (i, j). In addition, PSNR in Equation (2) refers to peak signal to noise ratio, which is a commonly used video quality metric [20]. All other constants are chosen by a least-square non-linear tting algorithm as described in [33], where s = 0.0356, t = 0.236, µ = 36.9, and s = 2.59.
The reason why we choose this metric is that it takes into account both the motion in the video and the frame rate being applied. The former (i.e., motion) matches well with the motion feature of the immersive video, which includes both the motion in the original video and that caused by user-initiated view switches. The latter (i.e., frame rate) matches well with the proposed approach based on frame rate control. Furthermore, according to [33], the STVQM metric has been clearly justied by the mean opinion scores from well organized subjective experiments.
Based on the STVQM metric and representative user head movement data (e.g., from [11]), we can calculate the quality-aware and power-ecient frame rate by rewriting Equation (1) as follows:
Following Equation (5), we can calculate the frame rate at the system runtime based on the quality requirement of the target video. However, we note that such an online frame rate calculation is infeasible due to the complexity of Equation ( 5), which requires the computations of TI, SI, and SV QM every time the video or user motion varies at runtime. According to [33] and [46], such computations involve pixel-level processing of one or multiple video frames, which by itself incurs non-trivial performance and power overhead and may oset the power saving goal of QuRate.
To address the challenge of the direct online mechanism, we develop an oine frame rate library, as presented in Figure 5, to facilitate power-ecient frame rate reduction at runtime. This library can be built using a dataset of user head movement data while watching immersive videos. In particular, for each user u i watching each video j , 1 i I , 1 j , and I and represents the number of users and videos in the dataset, respectively, we conduct the following three steps to build the frame rate library:
• Step 1, assign user u i 's movement data to an automatic view switching algorithm and play/record the VR video j with user u i 's movement; • Step 2, calculate the TI and SI values of the recorded video following Equations (3) and (4), as well as the SV QM value following Equation (2); and • Step 3, employ Equation (1) to calculate the STVQM value for video j at user u i 's view switching speed and all possible frame FR (e.g., 10, 20, ..., 60).
We repeat the above three steps for all the user-video pairs and obtain the following lookup table:
where FRL represents the frame rate library, which is not a closed form equation but presented as a lookup table obtained from the user/video dataset; is the user motion speed available in FRL that is closest to the instant motion speed of the target user; and Q is the objective video quality that the user aims to maintain. The generated FRL enables us to determine the power ecient frame rate for a new user. In particular, the parameters Q and are corresponding to the quality-aware and user-centric design principles in QuRate, respectively.
Based on the oine frame rate library in Equation ( 6), we develop the online algorithm for frame rate adaptation, as shown in Algorithm 2. The algorithm selects the best frame rate based on the current user's view switching speed, which is determined by QuRate through the sensors on the smartphone HMD.
During our experiments, we have noticed that manual power evaluation is a tedious process for each user-video pair. For example, for a one-minute video, we must spend at least one minute for the video playback and roughly another minute for preparing the test and collecting the results. In addition, the measurement noise is very common due to the complexity of the smartphone [8]. Other than that, the power measurement requires re-structuring the interconnection of the battery component, which increases the uncertainty. The experiment also needs to be paused frequently to cool down the system and avoid the inaccuracy caused by the generated heat. To overcome these challenges, we develop an analytical power model for the immersive video streaming system. This power model is based on the power measurement samples we have obtained and can be used to analyze the power consumption with the QuRate scheme. In this way, we can estimate the power consumption after only measuring the power once in the default case. This is helpful in tuning the power optimization framework (e.g., adjusting the threshold values).
Theoretically, when the frame rate is adjusted to a constant value, the average power consumption during the playback can be estimated using the following equation:
where P Est. refers to the estimated power consumption with the frame rate control, refers to the percentage of power consumed by view generation over the total power consumption, P Def . is the actual power consumption with the default frame rate FR Def . , and FR is the constant value that the frame rate is adjusted to. We further expand Equation (7) to consider the case that the frame rate is varying during the playback (i.e., after adopting the QuRate scheme), as shown below:
), (8) where n is the number of dierent frame rates, and i is the frequency of each frame rate FR i that appears during the video playback. In this way, we can estimate the power consumption after only measuring the power once in the default case. This is helpful in tuning the power optimization framework (e.g., adjusting the threshold values). In Section 5.4, we evaluate the accuracy of our predictive power model for immersive video streaming under varying frame rates.
We evaluate QuRate with the goal of understanding its eciency in power savings and the potential impact, if any, on the perceivable quality of the video. In particular, we rst measure and compare the power consumption in the cases with and without QuRate using empirical head movement data. Then, we evaluate and justify the power analytically model by comparing the modeled power results with the empirical measurements. Also, we conduct battery stress test to further verify the power evaluation results in empirical user settings. Last but not least, we carry out IRB-approved subjective QoE evaluations with human subjects involved, which proves the minimum impact QuRate poses on the perceivable video quality.
We adopt the same system setup (i.e., the power monitor and ve smartphones) as in Section 3 for our evaluation of QuRate. Also, based on the test videos described in Table 4 obtained from the publicly available head movement dataset [11], we select 21 out of 59 users who have watched the same set of 6 videos (referred to as Videos 1 to 6 hereafter based on Table 4). We calculate the switching speeds of the 21 users based on the and orientation coordinates provided by the dataset, as shown in Equation ( 9), where S i represents the switching speed of the orientation vector O i from time t t 1 to t i .
For each video, we rank the 21 users based on the average speed of each user watching all the 6 videos. In order to study the impact of the user's view switching speed, we select 4 representative users for each video to construct the oine frame rate library (e.g., for Video 1, we select User 8 ranked 19th, User 3 ranked 14th, User 7 ranked 8th, and User 6 ranked 5th), as shown in Table 5. In this process, our selection criterion is to cover high, medium, and low ranked user groups.
We build the oine frame rate library by calculating the STVQM values for all the 6 videos following Equation (1), as shown in Table 4, where the STVQM score refers to the quality of the video itself (i.e., without applying the users' movement). Then, we use the STVQM scores to categorize the motions of the 6 videos into slow, medium, and fast based on the understanding from [33], where a slower motion video obtains a lower STVQM score. Next, we apply the head movements of selected users from Table 5 to each video and calculate all the parameters (e.g., TI and SI) using Equations (1) to (2) with a TI and SI calculator [46] and a screen recorder [32] as described in Section 4.3. Finally, we plot 4 curves representing the frame rate library (i.e., Equation ( 6)) for each video to indicate the relationship between the video quality and the frame rate under dierent view switching speeds, as shown in Figure 6. Each curve in Figure 6 represents one user and thus indicates the behavior of one switching speed for the video. We observe that for each video, a faster switching speed requires lower frame rate at the same STVQM. This matches with our intuition that a fast switching view indicates the user's lack of interest in the current view, which allows us to reduce the frame rate while still maintaining the premium video quality.
For each video in Figure 6, we choose the video quality of users with the lowest switching speed at 60 FPS as the target video quality (e.g., we select the STVQM objective as 48 for Video 1). After applying the 4 users' switching speeds to Figure 6, we build the frame rate library to facilitate the online frame rate selection for an arbitrary new user, as shown in Table 4. We consider any switching speed slower than the slowest speed in Table 5 as a xed view, for which we apply the highest frame rate (i.e., 60 FPS). Based on our statistical analysis of the 21 users, the percentages of xed views in the 6 test videos are 36%, 33%, 37%, 37%, 32%, and 35%, which indicate large (more than 60%) room for power reduction.
Evaluation Method. We choose 10 users that are not involved in Table 5 for each video (i.e., Users 10 -19) as the test user set to evaluate the eectiveness of QuRate at the online stage. For these 10 users, we rst calculate their average switching speeds, e.g., the solid curve in Figure 7 shows the view switching speed of User 10 watching Video 1. Then, based on the frame rate library, we assign a frame rate to each second of the video, as presented by the dashed curve in Figure 7. For example, at the 30th second, if the switching speed of User 10 watching Video 1 is faster than the fast switching speed in the frame rate library, we choose the frame rate as 20 FPS.
Feasibility Evaluation. We conduct a feasibility evaluation to validate our hypothesis that users typically spend non-trivial amount of time in view switching and thus enable the opportunity for applying QuRate for power savings. Figure 8 summarizes the frequencies of view switches that are beyond the pre-dened threshold speed for frame rate reduction (i.e., considered as a view switch by QuRate), which are based on the public dataset [11]. We observe that the average frequency of view switching for all the 60 user/video combinations is 22.8%, with the highest of 68.1%, which indicates potential opportunities for power savings via QuRate. Furthermore, the switching frequencies demonstrate noticeable dependencies on individual users, which justies the necessity of the user-centric principle adopted by QuRate.
Power Evaluation and Comparison. In order to evaluate the performance of QuRate, we apply each user's head movement data to Algorithms 1 and 2. Then, we measure the power consumption and video quality of each user watching the videos with two other cases for comparison: (1) no frame rate reduction (i.e., the Default case); and (2) no QuRate for quality control (i.e., the Naive case). Figure 9 summarizes the average power consumption of 10 users (i.e., watching each video in the three cases on the LG V20 phone, where Naive means reducing the frame rate to the lowest value (i.e., at 10 FPS) without considering the quality impact. Figure 10 presents the runtime video quality (i.e., the STVQM value) of each case with User 10 watching the 6 videos. The standard deviations of the curves are 15.37 -15.85 (Default), 3.97 -3.90 (QuRate), and 4.20 -4.30 (Naive). Furthermore, we repeat the experiments with Users 10 -14 on Samsung S7, Moto G5, and LG G5, the results of which are shown in Figure 11. We observe that the Naive case saves the most power (27.57% to 43.89%) in our evaluations. However, it also results in the lowest video quality as shown in Figure 10. Also, the default frame rate achieves the highest video quality most of the time. Yet, it is highly unstable (i.e., the standard deviation can be up to 15.85) and consumes the highest power. After applying the QuRate scheme, the power consumption is reduced by a considerable amount (5.62% to 32.74%) with relatively consistent video qualities, as compared to the Default case. In addition, we notice that by using QuRate, the power consumption distribution is much larger than the other two approaches. We believe this is because QuRate is user motion related,