Эти чуваки такие крутые — они взяли 3д фильмы в Blu-ray качестве, выкачали оттуда стереопары, сделали из них карты глубины (используя PWC-Net: CNNs for optical flow using pyramid, warping,and cost volume. InCVPR, 2018) и натренировали на этом датасете сетку, которая предсказывает глубину с одной картинки. Модель доступна в гитхабе. Но идея с фильмами вообще топ.
https://github.com/intel-isl/MiDaS
Спасибо @ferluht 👽✌🏽🌚✨

#3D #depth

https://github.com/tencent-ailab/hifi3dface
#face_reconstruction #face #3d #morphable_model #depth

We present a fully automatic system that can produce high-fidelity, photo-realistic 3D digital human characters with a consumer RGB-D selfie camera. The system only needs the user to take a short selfie RGB-D video while rotating his/her head, and can produce a high quality reconstruction in less than 30 seconds. Our main contribution is a new facial geometry modeling and reflectance synthesis procedure that significantly improves the state-of-the-art. Specifically, given the input video a two-stage frame selection algorithm is first employed to select a few high-quality frames for reconstruction. A novel, differentiable renderer based 3D Morphable Model (3DMM) fitting method is then applied to recover facial geometries from multiview RGB-D data, which takes advantages of extensive data generation and perturbation. Our 3DMM has much larger expressive capacities than conventional 3DMM.
#depth #morphable_model

Learning to Recover 3D Scene Shape from a Single Image

https://yvanyin.github.io/LearnShape.github.io/

Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length. We investigate this problem in detail, and propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then use 3D point cloud encoders to predict the missing depth shift and focal length that allow us to recover a realistic 3D scene shape. In addition, we propose an image-level normalized regression loss and a normal-based geometry loss to enhance depth prediction models trained on mixed datasets. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot dataset generalization.

#single_image #depth #scene_reconstruction

https://github.com/angeladai/sgnn
#depth #scene_reconstruction #indoor

iMAP, first real-time SLAM system to use an implicit scene network as map representation
* Project page
* Paper: https://arxiv.org/abs/2103.12352
* twitter

iMAP is a new way to do SLAM: we learn an implicit neural representation *in real time* and track an RGB-D camera against it. The implicit map fills holes; completes the unseen backs of objects; and maps a whole room in only 1MB of weights. From the Dyson Robotics Lab, Imperial.
#implicit_geometry #depth #slam #indoor

Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
* project page
* abs
* pdf
We present ObSuRF, a method which turns a single image of a scene into a 3D model represented as a set of NeRFs, with each NeRF corresponding to a different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently to condition a NeRF decoder, defining the geometry and appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRFs on RGB-D inputs without explicit ray marching. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also to segment it into objects, despite receiving no supervision in that regard.
#nerf #segmentation #depth

NPMs: Neural Parametric Models
for 3D Deformable Shapes

* project page
* abs
* pdf
Parametric 3D models have enabled a wide variety of tasks in computer graphics and vision, such as modeling human bodies, faces, and hands. However, the construction of these parametric models is often tedious, as it requires heavy manual tweaking, and they struggle to represent additional complexity and details such as wrinkles or clothing.

To this end, we propose Neural Parametric Models (NPMs), a novel, learned alternative to traditional, parametric 3D models, which does not require hand-crafted, object-specific constraints. In particular, we learn to disentangle 4D dynamics into latent-space representations of shape and pose, leveraging the flexibility of recent developments in learned implicit functions. Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit new observations, similar to the fitting of a traditional parametric model, e.g., SMPL. This enables NPMs to achieve a significantly more accurate and detailed representation of observed deformable sequences.

We show that NPMs improve notably over both parametric and non-parametric state of the art in reconstruction and tracking of monocular depth sequences of clothed humans and hands. Latent-space interpolation as well as shape / pose transfer experiments further demonstrate the usefulness of NPMs.
#implicit_geometry #depth #non_rigid_reconstruction #avatars

#3d #novel_view #transformer #single_image #depth