Abstract
Deep learning for image processing typically treats input imagery as pixels in some color space. This paper proposes instead to learn from program traces of procedural fragment shaders -- programs that generate images. At each pixel, we collect the intermediate values computed at program execution, and these data form the input to the learned model. We investigate this learning task for a variety of applications: our model can learn to predict a low-noise output image from shader programs that exhibit sampling noise; this model can also learn from a simplified shader program that approximates the reference solution with less computation, as well as learn the output of postprocessing filters like defocus blur and edge-aware sharpening. Finally we show that the idea of learning from program traces can even be applied to non-imagery simulations of flocks of boids. Our experiments on a variety of shaders show quantitatively and qualitatively that models learned from program traces outperform baseline models learned from RGB color augmented with hand-picked shader-specific features like normals, depth, and diffuse and specular color. We also conduct a series of analyses that show certain features are important within the trace: these coincide with intuitively important aspects of the program. The important features can be helpful for selecting a good subset of trace features for learning, and even learning from a small subset of the trace already outperforms the baselines. We finally show that multiple shaders can be learned together with a shared denoising network and a lightweight shader-specific encoder.
Links
Citation
Yuting Yang, Connelly Barnes and Adam Finkelstein.
"Learning from Shader Program Traces."
Eurographics, April 2022
, April, 2022.
Bibtex
@inproceedings{Yang:2022:LFS,
author = "Yuting Yang and Connelly Barnes and Adam Finkelstein",
title = "Learning from Shader Program Traces",
booktitle = "Eurographics",
year = "2022",
month = apr
}
