TY - JOUR A1 - Reimann, Max A1 - Buchheim, Benito A1 - Semmo, Amir A1 - Döllner, Jürgen A1 - Trapp, Matthias T1 - Controlling strokes in fast neural style transfer using content transforms JF - The Visual Computer N2 - Fast style transfer methods have recently gained popularity in art-related applications as they make a generalized real-time stylization of images practicable. However, they are mostly limited to one-shot stylizations concerning the interactive adjustment of style elements. In particular, the expressive control over stroke sizes or stroke orientations remains an open challenge. To this end, we propose a novel stroke-adjustable fast style transfer network that enables simultaneous control over the stroke size and intensity, and allows a wider range of expressive editing than current approaches by utilizing the scale-variance of convolutional neural networks. Furthermore, we introduce a network-agnostic approach for style-element editing by applying reversible input transformations that can adjust strokes in the stylized output. At this, stroke orientations can be adjusted, and warping-based effects can be applied to stylistic elements, such as swirls or waves. To demonstrate the real-world applicability of our approach, we present StyleTune, a mobile app for interactive editing of neural style transfers at multiple levels of control. Our app allows stroke adjustments on a global and local level. It furthermore implements an on-device patch-based upsampling step that enables users to achieve results with high output fidelity and resolutions of more than 20 megapixels. Our approach allows users to art-direct their creations and achieve results that are not possible with current style transfer applications. Y1 - 2022 U6 - https://doi.org/10.1007/s00371-022-02518-x SN - 0178-2789 SN - 1432-2315 VL - 38 IS - 12 SP - 4019 EP - 4033 PB - Springer CY - New York ER - TY - JOUR A1 - Shekhar, Sumit A1 - Reimann, Max A1 - Mayer, Maximilian A1 - Semmo, Amir A1 - Pasewaldt, Sebastian A1 - Döllner, Jürgen A1 - Trapp, Matthias T1 - Interactive photo editing on smartphones via intrinsic decomposition JF - Computer graphics forum : journal of the European Association for Computer Graphics N2 - Intrinsic decomposition refers to the problem of estimating scene characteristics, such as albedo and shading, when one view or multiple views of a scene are provided. The inverse problem setting, where multiple unknowns are solved given a single known pixel-value, is highly under-constrained. When provided with correlating image and depth data, intrinsic scene decomposition can be facilitated using depth-based priors, which nowadays is easy to acquire with high-end smartphones by utilizing their depth sensors. In this work, we present a system for intrinsic decomposition of RGB-D images on smartphones and the algorithmic as well as design choices therein. Unlike state-of-the-art methods that assume only diffuse reflectance, we consider both diffuse and specular pixels. For this purpose, we present a novel specularity extraction algorithm based on a multi-scale intensity decomposition and chroma inpainting. At this, the diffuse component is further decomposed into albedo and shading components. We use an inertial proximal algorithm for non-convex optimization (iPiano) to ensure albedo sparsity. Our GPU-based visual processing is implemented on iOS via the Metal API and enables interactive performance on an iPhone 11 Pro. Further, a qualitative evaluation shows that we are able to obtain high-quality outputs. Furthermore, our proposed approach for specularity removal outperforms state-of-the-art approaches for real-world images, while our albedo and shading layer decomposition is faster than the prior work at a comparable output quality. Manifold applications such as recoloring, retexturing, relighting, appearance editing, and stylization are shown, each using the intrinsic layers obtained with our method and/or the corresponding depth data. KW - CCS Concepts KW - center dot Computing KW - methodologie KW - Image-based rendering KW - Image KW - processing KW - Computational photography Y1 - 2021 U6 - https://doi.org/10.1111/cgf.142650 SN - 0167-7055 SN - 1467-8659 VL - 40 SP - 497 EP - 510 PB - Blackwell CY - Oxford ER - TY - JOUR A1 - Isailović, Dušan A1 - Stojanovic, Vladeta A1 - Trapp, Matthias A1 - Richter, Rico A1 - Hajdin, Rade A1 - Döllner, Jürgen Roland Friedrich T1 - Bridge damage BT - detection, IFC-based semantic enrichment and visualization JF - Automation in construction : an international research journal N2 - Building Information Modeling (BIM) representations of bridges enriched by inspection data will add tremendous value to future Bridge Management Systems (BMSs). This paper presents an approach for point cloud-based detection of spalling damage, as well as integrating damage components into a BIM via semantic enrichment of an as-built Industry Foundation Classes (IFC) model. An approach for generating the as-built BIM, geometric reconstruction of detected damage point clusters and semantic-enrichment of the corresponding IFC model is presented. Multiview-classification is used and evaluated for the detection of spalling damage features. The semantic enrichment of as-built IFC models is based on injecting classified and reconstructed damage clusters back into the as-built IFC, thus generating an accurate as-is IFC model compliant to the BMS inspection requirements. KW - damage detection KW - building information modeling KW - 3D point clouds KW - multiview classification KW - bridge management systems Y1 - 2020 U6 - https://doi.org/10.1016/j.autcon.2020.103088 SN - 0926-5805 SN - 1872-7891 VL - 112 PB - Elsevier CY - Amsterdam ER - TY - JOUR A1 - Scheibel, Willy A1 - Trapp, Matthias A1 - Limberger, Daniel A1 - Döllner, Jürgen Roland Friedrich T1 - A taxonomy of treemap visualization techniques JF - Science and Technology Publications N2 - A treemap is a visualization that has been specifically designed to facilitate the exploration of tree-structured data and, more general, hierarchically structured data. The family of visualization techniques that use a visual metaphor for parent-child relationships based “on the property of containment” (Johnson, 1993) is commonly referred to as treemaps. However, as the number of variations of treemaps grows, it becomes increasingly important to distinguish clearly between techniques and their specific characteristics. This paper proposes to discern between Space-filling Treemap TS, Containment Treemap TC, Implicit Edge Representation Tree TIE, and Mapped Tree TMT for classification of hierarchy visualization techniques and highlights their respective properties. This taxonomy is created as a hyponymy, i.e., its classes have an is-a relationship to one another: TS TC TIE TMT. With this proposal, we intend to stimulate a discussion on a more unambiguous classification of treemaps and, furthermore, broaden what is understood by the concept of treemap itself. KW - Treemaps KW - Taxonomy Y1 - 2020 PB - Springer CY - Berlin ER - TY - JOUR A1 - Söchting, Maximilian A1 - Trapp, Matthias T1 - Controlling image-stylization techniques using eye tracking JF - Science and Technology Publications N2 - With the spread of smart phones capable of taking high-resolution photos and the development of high-speed mobile data infrastructure, digital visual media is becoming one of the most important forms of modern communication. With this development, however, also comes a devaluation of images as a media form with the focus becoming the frequency at which visual content is generated instead of the quality of the content. In this work, an interactive system using image-abstraction techniques and an eye tracking sensor is presented, which allows users to experience diverting and dynamic artworks that react to their eye movement. The underlying modular architecture enables a variety of different interaction techniques that share common design principles, making the interface as intuitive as possible. The resulting experience allows users to experience a game-like interaction in which they aim for a reward, the artwork, while being held under constraints, e.g., not blinking. The co nscious eye movements that are required by some interaction techniques hint an interesting, possible future extension for this work into the field of relaxation exercises and concentration training. KW - Eye-tracking KW - Image Abstraction KW - Image Processing KW - Artistic Image Stylization KW - Interactive Media Y1 - 2020 SN - 2184-4321 PB - Springer CY - Berlin ER - TY - JOUR A1 - Vollmer, Jan Ole A1 - Trapp, Matthias A1 - Schumann, Heidrun A1 - Döllner, Jürgen Roland Friedrich T1 - Hierarchical spatial aggregation for level-of-detail visualization of 3D thematic data JF - ACM transactions on spatial algorithms and systems N2 - Thematic maps are a common tool to visualize semantic data with a spatial reference. Combining thematic data with a geometric representation of their natural reference frame aids the viewer’s ability in gaining an overview, as well as perceiving patterns with respect to location; however, as the amount of data for visualization continues to increase, problems such as information overload and visual clutter impede perception, requiring data aggregation and level-of-detail visualization techniques. While existing aggregation techniques for thematic data operate in a 2D reference frame (i.e., map), we present two aggregation techniques for 3D spatial and spatiotemporal data mapped onto virtual city models that hierarchically aggregate thematic data in real time during rendering to support on-the-fly and on-demand level-of-detail generation. An object-based technique performs aggregation based on scene-specific objects and their hierarchy to facilitate per-object analysis, while the scene-based technique aggregates data solely based on spatial locations, thus supporting visual analysis of data with arbitrary reference geometry. Both techniques can apply different aggregation functions (mean, minimum, and maximum) for ordinal, interval, and ratio-scaled data and can be easily extended with additional functions. Our implementation utilizes the programmable graphics pipeline and requires suitably encoded data, i.e., textures or vertex attributes. We demonstrate the application of both techniques using real-world datasets, including solar potential analyses and the propagation of pressure waves in a virtual city model. KW - Level-of-detail visualization KW - spatial aggregation KW - real-time rendering Y1 - 2018 U6 - https://doi.org/10.1145/3234506 SN - 2374-0353 SN - 2374-0361 VL - 4 IS - 3 PB - Association for Computing Machinery CY - New York ER - TY - JOUR A1 - Stojanovic, Vladeta A1 - Trapp, Matthias A1 - Richter, Rico A1 - Döllner, Jürgen Roland Friedrich T1 - Service-oriented semantic enrichment of indoor point clouds using octree-based multiview classification JF - Graphical Models N2 - The use of Building Information Modeling (BIM) for Facility Management (FM) in the Operation and Maintenance (O&M) stages of the building life-cycle is intended to bridge the gap between operations and digital data, but lacks the functionality of assessing the state of the built environment due to non-automated generation of associated semantics. 3D point clouds can be used to capture the physical state of the built environment, but also lack these associated semantics. A prototypical implementation of a service-oriented architecture for classification of indoor point cloud scenes of office environments is presented, using multiview classification. The multiview classification approach is tested using a retrained Convolutional Neural Network (CNN) model - Inception V3. The presented approach for classifying common office furniture objects (chairs, sofas and desks), contained in 3D point cloud scans, is tested and evaluated. The results show that the presented approach can classify common office furniture up to an acceptable degree of accuracy, and is suitable for quick and robust semantics approximation - based on RGB (red, green and blue color channel) cubemap images of the octree partitioned areas of the 3D point cloud scan. Additional methods for web-based 3D visualization, editing and annotation of point clouds are also discussed. Using the described approach, captured scans of indoor environments can be semantically enriched using object annotations derived from multiview classification results. Furthermore, the presented approach is suited for semantic enrichment of lower resolution indoor point clouds acquired using commodity mobile devices. KW - Semantic enrichment KW - 3D point clouds KW - Multiview classification KW - Service-oriented KW - Indoor environments Y1 - 2019 U6 - https://doi.org/10.1016/j.gmod.2019.101039 SN - 1524-0703 SN - 1524-0711 VL - 105 PB - Elsevier CY - San Diego ER - TY - JOUR A1 - Reimann, Max A1 - Klingbeil, Mandy A1 - Pasewaldt, Sebastian A1 - Semmo, Amir A1 - Trapp, Matthias A1 - Döllner, Jürgen Roland Friedrich T1 - Locally controllable neural style transfer on mobile devices JF - The Visual Computer N2 - Mobile expressive rendering gained increasing popularity among users seeking casual creativity by image stylization and supports the development of mobile artists as a new user group. In particular, neural style transfer has advanced as a core technology to emulate characteristics of manifold artistic styles. However, when it comes to creative expression, the technology still faces inherent limitations in providing low-level controls for localized image stylization. In this work, we first propose a problem characterization of interactive style transfer representing a trade-off between visual quality, run-time performance, and user control. We then present MaeSTrO, a mobile app for orchestration of neural style transfer techniques using iterative, multi-style generative and adaptive neural networks that can be locally controlled by on-screen painting metaphors. At this, we enhance state-of-the-art neural style transfer techniques by mask-based loss terms that can be interactively parameterized by a generalized user interface to facilitate a creative and localized editing process. We report on a usability study and an online survey that demonstrate the ability of our app to transfer styles at improved semantic plausibility. KW - Non-photorealistic rendering KW - Style transfer KW - Neural networks KW - Mobile devices KW - Interactive control KW - Expressive rendering Y1 - 2019 U6 - https://doi.org/10.1007/s00371-019-01654-1 SN - 0178-2789 SN - 1432-2315 VL - 35 IS - 11 SP - 1531 EP - 1547 PB - Springer CY - New York ER - TY - JOUR A1 - Buschmann, Stefan A1 - Trapp, Matthias A1 - Döllner, Jürgen Roland Friedrich T1 - Animated visualization of spatial-temporal trajectory data for air-traffic analysis JF - The Visual Computer N2 - With increasing numbers of flights worldwide and a continuing rise in airport traffic, air-traffic management is faced with a number of challenges. These include monitoring, reporting, planning, and problem analysis of past and current air traffic, e.g., to identify hotspots, minimize delays, or to optimize sector assignments to air-traffic controllers. To cope with these challenges, cyber worlds can be used for interactive visual analysis and analytical reasoning based on aircraft trajectory data. However, with growing data size and complexity, visualization requires high computational efficiency to process that data within real-time constraints. This paper presents a technique for real-time animated visualization of massive trajectory data. It enables (1) interactive spatio-temporal filtering, (2) generic mapping of trajectory attributes to geometric representations and appearance, and (3) real-time rendering within 3D virtual environments such as virtual 3D airport or 3D city models. Different visualization metaphors can be efficiently built upon this technique such as temporal focus+context, density maps, or overview+detail methods. As a general-purpose visualization technique, it can be applied to general 3D and 3+1D trajectory data, e.g., traffic movement data, geo-referenced networks, or spatio-temporal data, and it supports related visual analytics and data mining tasks within cyber worlds. KW - Spatio-temporal visualization KW - Trajectory visualization KW - 3D visualization KW - Visual analytics KW - Real-time rendering Y1 - 2016 U6 - https://doi.org/10.1007/s00371-015-1185-9 SN - 0178-2789 SN - 1432-2315 VL - 32 SP - 371 EP - 381 PB - Springer CY - New York ER - TY - JOUR A1 - Semmo, Amir A1 - Trapp, Matthias A1 - Jobst, Markus A1 - Döllner, Jürgen Roland Friedrich T1 - Cartography-Oriented Design of 3D Geospatial Information Visualization - Overview and Techniques JF - The cartographic journal N2 - In economy, society and personal life map-based interactive geospatial visualization becomes a natural element of a growing number of applications and systems. The visualization of 3D geospatial information, however, raises the question how to represent the information in an effective way. Considerable research has been done in technology-driven directions in the fields of cartography and computer graphics (e.g., design principles, visualization techniques). Here, non-photorealistic rendering (NPR) represents a promising visualization category - situated between both fields - that offers a large number of degrees for the cartography-oriented visual design of complex 2D and 3D geospatial information for a given application context. Still today, however, specifications and techniques for mapping cartographic design principles to the state-of-the-art rendering pipeline of 3D computer graphics remain to be explored. This paper revisits cartographic design principles for 3D geospatial visualization and introduces an extended 3D semiotic model that complies with the general, interactive visualization pipeline. Based on this model, we propose NPR techniques to interactively synthesize cartographic renditions of basic feature types, such as terrain, water, and buildings. In particular, it includes a novel iconification concept to seamlessly interpolate between photorealistic and cartographic representations of 3D landmarks. Our work concludes with a discussion of open challenges in this field of research, including topics, such as user interaction and evaluation. KW - 3D information visualization KW - 3D semiotic model KW - cartographic design KW - user interaction KW - real-time rendering Y1 - 2015 U6 - https://doi.org/10.1080/00087041.2015.1119462 SN - 0008-7041 SN - 1743-2774 VL - 52 IS - 2 SP - 95 EP - 106 PB - Routledge, Taylor & Francis Group CY - Leeds ER -