TY  - JOUR
A1  - Shekhar, Sumit
A1  - Reimann, Max
A1  - Mayer, Maximilian
A1  - Semmo, Amir
A1  - Pasewaldt, Sebastian
A1  - Döllner, Jürgen
A1  - Trapp, Matthias
T1  - Interactive photo editing on smartphones via intrinsic decomposition
JF  - Computer graphics forum : journal of the European Association for Computer Graphics
N2  - Intrinsic decomposition refers to the problem of estimating scene characteristics, such as albedo and shading, when one view or multiple views of a scene are provided. The inverse problem setting, where multiple unknowns are solved given a single known pixel-value, is highly under-constrained. When provided with correlating image and depth data, intrinsic scene decomposition can be facilitated using depth-based priors, which nowadays is easy to acquire with high-end smartphones by utilizing their depth sensors. In this work, we present a system for intrinsic decomposition of RGB-D images on smartphones and the algorithmic as well as design choices therein. Unlike state-of-the-art methods that assume only diffuse reflectance, we consider both diffuse and specular pixels. For this purpose, we present a novel specularity extraction algorithm based on a multi-scale intensity decomposition and chroma inpainting. At this, the diffuse component is further decomposed into albedo and shading components. We use an inertial proximal algorithm for non-convex optimization (iPiano) to ensure albedo sparsity. Our GPU-based visual processing is implemented on iOS via the Metal API and enables interactive performance on an iPhone 11 Pro. Further, a qualitative evaluation shows that we are able to obtain high-quality outputs. Furthermore, our proposed approach for specularity removal outperforms state-of-the-art approaches for real-world images, while our albedo and shading layer decomposition is faster than the prior work at a comparable output quality. Manifold applications such as recoloring, retexturing, relighting, appearance editing, and stylization are shown, each using the intrinsic layers obtained with our method and/or the corresponding depth data.
KW  - CCS Concepts
KW  - center dot Computing
KW  - methodologie
KW  - Image-based rendering
KW  - Image
KW  - processing
KW  - Computational photography
Y1  - 2021
U6  - https://doi.org/10.1111/cgf.142650
SN  - 0167-7055
SN  - 1467-8659
VL  - 40
SP  - 497
EP  - 510
PB  - Blackwell
CY  - Oxford
ER  -