DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models
ACM Transactions on Graphics (Proc. Siggraph 2024), 2024, 43(4): Article No.: 39.
Yuqing Zhang, Yuan Liu, Zhiyu Xie, Lei Yang, Zhongyuan Liu, Mengzhou Yang, Runze Zhang, Qilong Kou, Cheng Lin, Wenping Wang, Xiaogang Jin
Using untextured meshes and textual descriptions as input (top row), our method generates high-quality appearances consisting of albedo, roughness, and metallic (middle row) that can be applied in modern graphics engines for photo-realistic rendering under any new illumination environments (bottom row).
Abstract
Recent advancements in
2D diffusion models allow appearance generation on untextured raw meshes.
These methods create RGB textures by distilling a 2D diffusion model, which
often contains unwanted baked-in shading effects and results in unrealistic
rendering effects in the downstream applications. Generating Physically
Based Rendering (PBR) materials instead of just RGB textures would be a
promising solution. However, directly distilling the PBR material parameters
from 2D diffusion models still suffers from incorrect material
decomposition, such as baked-in shading effects in albedo. We introduce
DreamMat, an innovative approach to resolve the aforementioned problem,
to generate high-quality PBR materials from text descriptions. We find out
that the main reason for the incorrect material distillation is that
large-scale 2D diffusion models are only trained to generate final shading
colors, resulting in insufficient constraints on material decomposition
during distillation. To tackle this problem, we first finetune a new
light-aware 2D diffusion model to condition on a given lighting environment
and generate the shading results on this specific lighting condition. Then,
by applying the same environment lights in the material distillation,
DreamMat can generate high-quality PBR materials that are not only
consistent with the given geometry but also free from any baked-in shading
effects in albedo. Extensive experiments demonstrate that the materials
produced through our methods exhibit greater visual appeal to users and
achieve significantly superior rendering quality compared to baseline
methods, which are preferable for downstream tasks such as game and film
production.