Unsupervised Continuous Object Representation Networks for Novel View Synthesis [preprint]

Three computers with locks in a circle illustration

Preprint date

July 30, 2020

Authors

Nicolai Häni (Ph.D. student), Selim Engin (Ph.D. student), Jun-Jee Chao (M.S. student), Volkan Isler (professor)

Abstract

Novel View Synthesis (NVS) is concerned with synthesizing views under camera viewpoint transformations from one or multiple input images. NVS requires explicit reasoning about 3D object structure and unseen parts of the scene to synthesize convincing results. As a result, current approaches typically rely on supervised training with either ground truth 3D models or multiple target images. We propose Continuous Object Representation Networks (CORN), a conditional architecture that encodes an input image's geometry and appearance that map to a 3D consistent scene representation. We can train CORN with only two source images per object by combining our model with a neural renderer. A key feature of CORN is that it requires no ground truth 3D models or target view supervision. Regardless, CORN performs well on challenging tasks such as novel view synthesis and single-view 3D reconstruction and achieves performance comparable to state-of-the-art approaches that use direct supervision.

Link to project website and full paper

Unsupervised Continuous Object Representation Networks for Novel View Synthesis

Keywords

computer vision