MobileSky: Real-time Sky Replacement for Mobile AR
IEEE Transactions on Visualization and Computer Graphics, 2023 (Accepted)
Xinjie Wang, Qingxuan Lv, Guo Chen, Jing Zhang, Zhiqiang Wei, Junyu Dong, Hongbo Fu, Zhipeng Zhu, Jingxin Liu, and Xiaogang Jin
(a) real-time sky replacement (b) with virtual blue sky with whales (c) with virtual dusk sky with balloons
Our MobileSky is able to naturally blend a stylized skybox (i.e., a pre-made panoramic sky video with virtual 3D objects) and camera feed into a visually pleasing AR scene in real time. This is enabled by real-time generation of a computed mask map based on the camera feed and IMU data for replacing sky regions at a real-time rate. We demonstrate: (a) An illustration of MobileSky application; (b) and (c) Two captured frames under different weather conditions and the composite examples with fantasy skies and imaginary objects.
Abstract
We present MobileSky,
the first automatic method for real-time high-quality sky replacement for
mobile AR applications. The primary challenge of this task is how to extract
sky regions in camera feed both quickly and accurately. While the problem of
sky replacement is not new, previous methods mainly concern extraction
quality rather than efficiency, limiting their application to our task. We
aim to provide higher quality, both spatially and temporally consistent sky
mask maps for all camera frames in real time. To this end, we develop a
novel framework that combines a new deep semantic network called FSNet with
novel post-processing refinement steps. By leveraging IMU data, we also
propose new sky-aware constraints such as temporal consistency, position
consistency, and color consistency to help refine the weakly classified part
of the segmentation output. Experiments show that our method achieves an
average of around 30 FPS on off-the-shelf smartphones and outperforms the
state-of-the-art sky replacement methods in terms of execution speed and
quality. In the meantime, our mask maps appear to be visually more stable
across frames. Our fast sky replacement method enables several applications,
such as AR advertising, art making, generating fantasy celestial objects,
visually learning about weather phenomena, and advanced video-based visual
effects. To facilitate future research, we also create a new video dataset
containing annotated sky regions with IMU data.