TY - JOUR A1 - Reimann, Max A1 - Buchheim, Benito A1 - Semmo, Amir A1 - Döllner, Jürgen A1 - Trapp, Matthias T1 - Controlling strokes in fast neural style transfer using content transforms JF - The Visual Computer N2 - Fast style transfer methods have recently gained popularity in art-related applications as they make a generalized real-time stylization of images practicable. However, they are mostly limited to one-shot stylizations concerning the interactive adjustment of style elements. In particular, the expressive control over stroke sizes or stroke orientations remains an open challenge. To this end, we propose a novel stroke-adjustable fast style transfer network that enables simultaneous control over the stroke size and intensity, and allows a wider range of expressive editing than current approaches by utilizing the scale-variance of convolutional neural networks. Furthermore, we introduce a network-agnostic approach for style-element editing by applying reversible input transformations that can adjust strokes in the stylized output. At this, stroke orientations can be adjusted, and warping-based effects can be applied to stylistic elements, such as swirls or waves. To demonstrate the real-world applicability of our approach, we present StyleTune, a mobile app for interactive editing of neural style transfers at multiple levels of control. Our app allows stroke adjustments on a global and local level. It furthermore implements an on-device patch-based upsampling step that enables users to achieve results with high output fidelity and resolutions of more than 20 megapixels. Our approach allows users to art-direct their creations and achieve results that are not possible with current style transfer applications. Y1 - 2022 U6 - https://doi.org/10.1007/s00371-022-02518-x SN - 0178-2789 SN - 1432-2315 VL - 38 IS - 12 SP - 4019 EP - 4033 PB - Springer CY - New York ER - TY - JOUR A1 - Shekhar, Sumit A1 - Reimann, Max A1 - Mayer, Maximilian A1 - Semmo, Amir A1 - Pasewaldt, Sebastian A1 - Döllner, Jürgen A1 - Trapp, Matthias T1 - Interactive photo editing on smartphones via intrinsic decomposition JF - Computer graphics forum : journal of the European Association for Computer Graphics N2 - Intrinsic decomposition refers to the problem of estimating scene characteristics, such as albedo and shading, when one view or multiple views of a scene are provided. The inverse problem setting, where multiple unknowns are solved given a single known pixel-value, is highly under-constrained. When provided with correlating image and depth data, intrinsic scene decomposition can be facilitated using depth-based priors, which nowadays is easy to acquire with high-end smartphones by utilizing their depth sensors. In this work, we present a system for intrinsic decomposition of RGB-D images on smartphones and the algorithmic as well as design choices therein. Unlike state-of-the-art methods that assume only diffuse reflectance, we consider both diffuse and specular pixels. For this purpose, we present a novel specularity extraction algorithm based on a multi-scale intensity decomposition and chroma inpainting. At this, the diffuse component is further decomposed into albedo and shading components. We use an inertial proximal algorithm for non-convex optimization (iPiano) to ensure albedo sparsity. Our GPU-based visual processing is implemented on iOS via the Metal API and enables interactive performance on an iPhone 11 Pro. Further, a qualitative evaluation shows that we are able to obtain high-quality outputs. Furthermore, our proposed approach for specularity removal outperforms state-of-the-art approaches for real-world images, while our albedo and shading layer decomposition is faster than the prior work at a comparable output quality. Manifold applications such as recoloring, retexturing, relighting, appearance editing, and stylization are shown, each using the intrinsic layers obtained with our method and/or the corresponding depth data. KW - CCS Concepts KW - center dot Computing KW - methodologie KW - Image-based rendering KW - Image KW - processing KW - Computational photography Y1 - 2021 U6 - https://doi.org/10.1111/cgf.142650 SN - 0167-7055 SN - 1467-8659 VL - 40 SP - 497 EP - 510 PB - Blackwell CY - Oxford ER - TY - GEN A1 - Reimann, Max A1 - Klingbeil, Mandy A1 - Pasewaldt, Sebastian A1 - Semmo, Amir A1 - Trapp, Matthias A1 - Döllner, Jürgen Roland Friedrich ED - Sourin, A Sourina T1 - MaeSTrO: A Mobile App for Style Transfer Orchestration using Neural Networks T2 - International Conference on Cyberworlds (CW) N2 - Mobile expressive rendering gained increasing popularity among users seeking casual creativity by image stylization and supports the development of mobile artists as a new user group. In particular, neural style transfer has advanced as a core technology to emulate characteristics of manifold artistic styles. However, when it comes to creative expression, the technology still faces inherent limitations in providing low-level controls for localized image stylization. This work enhances state-of-the-art neural style transfer techniques by a generalized user interface with interactive tools to facilitate a creative and localized editing process. Thereby, we first propose a problem characterization representing trade-offs between visual quality, run-time performance, and user control. We then present MaeSTrO, a mobile app for orchestration of neural style transfer techniques using iterative, multi-style generative and adaptive neural networks that can be locally controlled by on-screen painting metaphors. At this, first user tests indicate different levels of satisfaction for the implemented techniques and interaction design. KW - non-photorealistic rendering KW - style transfer Y1 - 2018 SN - 978-1-5386-7315-7 U6 - https://doi.org/10.1109/CW.2018.00016 SP - 9 EP - 16 PB - IEEE CY - New York ER - TY - JOUR A1 - Reimann, Max A1 - Klingbeil, Mandy A1 - Pasewaldt, Sebastian A1 - Semmo, Amir A1 - Trapp, Matthias A1 - Döllner, Jürgen Roland Friedrich T1 - Locally controllable neural style transfer on mobile devices JF - The Visual Computer N2 - Mobile expressive rendering gained increasing popularity among users seeking casual creativity by image stylization and supports the development of mobile artists as a new user group. In particular, neural style transfer has advanced as a core technology to emulate characteristics of manifold artistic styles. However, when it comes to creative expression, the technology still faces inherent limitations in providing low-level controls for localized image stylization. In this work, we first propose a problem characterization of interactive style transfer representing a trade-off between visual quality, run-time performance, and user control. We then present MaeSTrO, a mobile app for orchestration of neural style transfer techniques using iterative, multi-style generative and adaptive neural networks that can be locally controlled by on-screen painting metaphors. At this, we enhance state-of-the-art neural style transfer techniques by mask-based loss terms that can be interactively parameterized by a generalized user interface to facilitate a creative and localized editing process. We report on a usability study and an online survey that demonstrate the ability of our app to transfer styles at improved semantic plausibility. KW - Non-photorealistic rendering KW - Style transfer KW - Neural networks KW - Mobile devices KW - Interactive control KW - Expressive rendering Y1 - 2019 U6 - https://doi.org/10.1007/s00371-019-01654-1 SN - 0178-2789 SN - 1432-2315 VL - 35 IS - 11 SP - 1531 EP - 1547 PB - Springer CY - New York ER - TY - BOOK A1 - Herbst, Eva‐Maria A1 - Maschler, Fabian A1 - Niephaus, Fabio A1 - Reimann, Max A1 - Steier, Julia A1 - Felgentreff, Tim A1 - Lincke, Jens A1 - Taeumel, Marcel A1 - Hirschfeld, Robert A1 - Witt, Carsten T1 - ecoControl T1 - ecoControl BT - Entwurf und Implementierung einer Software zur Optimierung heterogener Energiesysteme in Mehrfamilienhäusern BT - design and implementation of a prototype for optimizing heterogeneous energy systems in multi‐family residential buildings N2 - Eine dezentrale Energieversorgung ist ein erster Schritt in Richtung Energiewende. Dabei werden auch in Mehrfamilienhäusern vermehrt verschiedene Strom- und Wärmeerzeuger eingesetzt. Besonders in Deutschland kommen in diesem Zusammenhang Blockheizkraftwerke immer häufiger zum Einsatz, weil sie Gas sehr effizient in Strom und Wärme umwandeln können. Außerdem ermöglichen sie, im Zusammenspiel mit anderen Energiesystemen wie beispielsweise Photovoltaik-Anlagen, eine kontinuierliche und dezentrale Energieversorgung. Bei dem Betrieb von unterschiedlichen Energiesystemen ist es wünschenswert, dass die Systeme aufeinander abgestimmt arbeiten. Allerdings ist es bisher schwierig, heterogene Energiesysteme effizient miteinander zu betreiben. Dadurch bleiben Einsparungspotentiale ungenutzt. Eine zentrale Steuerung kann deshalb die Effizienz des Gesamtsystems verbessern. Mit ecoControl stellen wir einen erweiterbaren Prototypen vor, der die Kooperation von Energiesystemen optimiert und Umweltfaktoren miteinbezieht. Dazu stellt die Software eine einheitliche Bedienungsoberfläche zur Konfiguration aller Systeme zur Verfügung. Außerdem bietet sie die Möglichkeit, Optimierungsalgorithmen mit Hilfe einer Programmierschnittstelle zu entwickeln, zu testen und auszuführen. Innerhalb solcher Algorithmen können von ecoControl bereitgestellte Vorhersagen genutzt werden. Diese Vorhersagen basieren auf dem individuellen Verhalten von jedem Energiesystem, Wettervorhersagen und auf Prognosen des Energieverbrauchs. Mithilfe einer Simulation können Techniker unterschiedliche Konfigurationen und Optimierungen sofort ausprobieren, ohne diese über einen langen Zeitraum an realen Geräten testen zu müssen. ecoControl hilft darüber hinaus auch Hausverwaltungen und Vermietern bei der Verwaltung und Analyse der Energiekosten. Wir haben anhand von Fallbeispielen gezeigt, dass Optimierungsalgorithmen, welche die Nutzung von Wärmespeichern verbessern, die Effizienz des Gesamtsystems erheblich verbessern können. Schließlich kommen wir zu dem Schluss, dass ecoControl in einem nächsten Schritt unter echten Bedingungen getestet werden muss, sobald eine geeignete Hardwarekomponente verfügbar ist. Über diese Schnittstelle werden die Messwerte an ecoControl gesendet und Steuersignale an die Geräte weitergeleitet. N2 - The energy turnaround in Germany affects not only big industries but also smaller advocates who are interested in cost-efficient and regenerative energy supply. The observable signs of decentralized supply indicate that many individuals are eager to employ affordable energy devices, such as solar power systems, by themselves. Owners or managers of multi-family residential buildings, for example, install heterogeneous sets of devices that have to satisfy the varying demands of tenants. These devices are primarily influenced by environmental factors such as the weather. Independently, on-site cogeneration units are increasingly used to produce both electrical and thermal energy in a dependable and decentralized way. While having an arguably good efficiency on their own, such energy systems, however, are not built to cooperate in an heterogeneous installation. Hence they can negatively affect overall costs or impair the optimal ecological energy usage. We propose a centralized, extensible control platform that supports low-effort integration and efficient cooperation of heterogeneous energy production and storage units. Our prototype ecoControl shows that such a software system can be used to optimize the communication protocol of energy devices in multi-family residential buildings. In addition a simulation of the devices and forecasts of both energy supply and demand facilitate an advanced configuration of the system to enable an optimal drive. An intuitive user interface supports technicians, managers or owners to monitor and adjust the operation of installed devices to accommodate given conditions - even if not anticipated by the manufacturer. In several example cases, we illustrate how optimization algorithms can improve the use of heat storages to increase overall efficiency by a significant factor. Although further investigations with representative settings are needed, we argue that ecoControl can contribute to Germany's energy turnaround by projecting a novel perspective on the application of interdependent energy production and storage units. T3 - Technische Berichte des Hasso-Plattner-Instituts für Digital Engineering an der Universität Potsdam - 93 KW - Energiesparen KW - Prognosen KW - Effizienz KW - Optimierungen KW - Algorithmen KW - Blockheizkraftwerke KW - Mehrfamilienhäuser KW - energy savings KW - forecasts KW - efficiency KW - optimizations KW - algorithms KW - cogeneration units KW - multi-­family residential buildings Y1 - 2015 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:kobv:517-opus4-72147 SN - 978-3-86956-318-3 SN - 1613-5652 SN - 2191-1665 IS - 93 PB - Universitätsverlag Potsdam CY - Potsdam ER -