Zero-Latency strategies for video transmission using frame extrapolation
- Authors
- Publication Date
- Sep 11, 2024
- Source
- HAL
- Keywords
- Language
- English
- License
- Unknown
- External links
Abstract
The demand for seamless, high-quality video content delivery with minimal latency is paramount in today's applications such as sports broadcasting, videoconferencing, and remote system control. However, video delivery still faces challenges due to unpredictable nature of communication channels. The variations in channel characteristics can impact the quality of experience in terms of content quality and End-To-End latency - the time elapsed between video acquisition at the transmitter and its display at the receiver. The aim of this thesis is to address the issue of real time applications with unicast transmission from server to client such as remote control applications, while maintaining a good quality. We test the effectiveness of a recent deep learning technique for latency compensation in the video transmission scheme and its impact on video quality. This technique predicts future frames using available previous frames, allowing the end-user to display the images at the desired time. The results demonstrate the promise of extrapolation, especially for content with low temporal information. However, it still needs to be improved in terms of quality, long-term prediction, and extrapolation delay. Various studies focus on the integration of a hybrid digital-analog scheme to improve the perceptual quality, taking advantage of the strengths of both digital and analog methods. We study the effectiveness of low-latency hybrid scheme in term of reducing latency while maintaining high video quality. The results show that the hybrid scheme improves the quality of the received video in most cases. However, the extrapolation artifacts outweigh encoding artifacts and mask the advantages of hybrid schemes. Thus, the improvement in hybrid scheme performance relies on the enhancement of extrapolation. Moreover, HTTP Adaptive Streaming methods have proven their effectiveness in improving the quality of experience by dynamically adjusting the encoding rate based on channel conditions. However, most of these adaptation algorithms are implemented at the client level, which poses challenges in meeting latency requirements for real time applications. In addition, in real time application, videos are acquired, compressed, and transmitted from the device acting as the server. Therefore, client-driven rate adaptation approaches are not suitable due to the variability of the channel characteristics. Moreover, in these methods, the decision-making is done with a periodicity of the order of a second, which is not reactive enough when the server is moving, leading to significant delays. Therefore, it is important to use a finer adaptation granularity in order to reduce the End-To-End delay. We aim to control the End-To-End latency during video delivery while ensuring a high quality of experience. A frame-level encoder rate control at the transmitter side is combined with a frame extrapolation at the receiver side to compensate the End-To-End delays. Frame-level rate control enables the system to adapt to sudden variations of channel characteristics. Null apparent End-To-End delay can be reached at the price of some signal quality. To the best of our knowledge, state-of-the-art algorithms try to optimize the individual sources of delay in the video delivery scheme, but not to reduce the whole End-To-End latency and achieve zero latency. A model predictive control approach involving the buffer level at the transmitter and the throughput estimation is used to find the optimal value of encoding rate for each frame. It dynamically adjusts the trade-off between the encoding rate and the extrapolation horizon at the receiver, while predicting the impact of the encoding rate decision on future frames, thus providing the best quality of experience.
