Mob-FGSR: Frame Generation and Super Resolution for Mobile
Real-Time Rendering

 

Proceedings of Siggraph'2024, Denver, 28 July-1 August, USA, Pages 1–11

Sipeng Yang, Qingchuan Zhu, Junhao Zhuge, Qiang Qiu, Chen Li, Yuzhong Yan, Huihui Xu, Ling-qi Yan, Xiaogang Jin

 

Frame generation (FG) and super resolution (SR) results of our method. Requiring only color, depth, and motion vectors from frames rendered at times 0 (A) and 1 (C), our approach efficiently produces interpolated (B) / extrapolated (D) frames and their SR counterparts at desired times. The left large images illustrate an interpolated frame (upper), its SR result (middle), and the high-resolution temporal anti-aliasing (TAA) reference (bottom). The right images show enlarged views of the yellow-framed regions of the interpolated frame (I.), their SR results (II.), and TAA references (III.), demonstrating our method’s smooth motion estimation and high-quality SR results. (E) displays ×2 SR results of commercial applications FSR 1, FSR 2, and DLSS 2. Experiments show that our approach achieves results comparable to the best of these solutions. Notably, our models have short runtimes, processing FG at 720P in 2.2ms and FG & SR at 1080P in 2.3ms on a Snapdragon 8 Gen 3 processor.

Abstract

Recent advances in supersampling for frame generation and super-resolution improve real-time rendering performance significantly. However, because these methods rely heavily on the most recent features of high-end GPUs, they are impractical for mobile platforms, which are limited by lower GPU capabilities and a lack of dedicated optical flow estimation hardware. We propose Mob-FGSR, a novel lightweight supersampling framework tailored for mobile devices that integrates frame generation with super resolution to effectively improve real-time rendering performance. Our method introduces a splat-based motion vectors reconstruction method, which allows for accurate pixel-level motion estimation for both interpolation and extrapolation at desired times without the need for high-end GPUs or rendering data from generated frames. Subsequently, fast image generation models are designed to construct interpolated or extrapolated frames and improve resolution, providing users with a plethora of options. Our runtime models operate without the use of neural networks, ensuring their applicability to mobile devices. Extensive testing shows that our framework outperforms other lightweight solutions and rivals the performance of algorithms designed specifically for high-end GPUs. Our model's minimal runtime is confirmed by on-device testing, demonstrating its potential to benefit a wide range of mobile real-time rendering applications. More information and an Android demo can be found at: https://mob-fgsr.github.io.

Download

PDF, 37.8 MB Video, 321.0 MB