QiYun

Note

Posted on Edited on In
2.7k 10 mins.
Title High-fidelity 3D Human Digitization from Single 2K Resolution Images
Author Sang-Hun Han1, Min-Gyu Park2, Ju Hong Yoon2,Ju-Mi Kang2, Young-Jae Park1, and Hae-Gon Jeon1
Conf/Jour CVPR 2023 Highlight
Year 2023
Project High-fidelity 3D Human Digitization from Single 2K Resolution Images Project Page (sanghunhan92.github.io)
Paper High-fidelity 3D Human Digitization from Single 2K Resolution Images (readpaper.com)

image.png

可以看成一种估计深度图的方法
缺点：需要好的数据集

  • 需要提供法线图、mask、深度图(低分辨率+高分辨率)
  • 需要人体模型的关节点信息
  • 无法预测自遮挡部位
  • 对低分辨率重建效果不好
Read more »

Posted on Edited on In
3k 11 mins.

NeRF-based重建方法之于前作监督的重建(新视图生成)方式，如MVS需要真实的深度图作监督，之前的包括生成式的方法需要三维模型的信息(PointCloud、Voxel、Mesh)作监督，NeRF-based方法构建了一种自监督的重建方式，从图像中重建物体只需要用图像作监督

NeRF将三维空间中所有点，通过MLP预测出对应的密度/SDF，是一种连续的方法(理论上，实际上由于计算机精度还是离散的)。至少在3D上不会由于离散方法(voxel)，而出现很大的锯齿/aliasing

NeRF-based self-supervised 3D Reconstruction

  1. image and pose(COLMAP)
  2. NeRF(NeuS) or 3DGS(SuGaR)
    1. 损失函数(对比像素颜色、深度、法向量、SDF梯度累积<Eikonal term>Eikonal Equation and SDF - Lin’s site)
  3. PointCloud后处理，根据不同用途如3D打印、有限元仿真分析、游戏assets，有许多格式mesh/FEMode/AMs
Read more »

Posted on Edited on In
5.8k 21 mins.
Title Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
Author Shen, Tianchang and Munkberg, Jacob and Hasselgren, Jon and Yin, Kangxue and Wang, Zian and Chen, Wenzheng and Gojcic, Zan and Fidler, Sanja and Sharp, Nicholas and Gao, Jun
Conf/Jour ACM Trans. on Graph. (SIGGRAPH 2023)
Year 2023
Project Flexible Isosurface Extraction for Gradient-Based Mesh Optimization (FlexiCubes) (nvidia.com)
Paper Flexible Isosurface Extraction for Gradient-Based Mesh Optimization (nv-tlabs.github.io)

一种新的Marching Cube的方法
image.png

Read more »

Posted on Edited on In
6.3k 23 mins.
Title BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis
Author Lior Yariv and Peter Hedman and Christian Reiser and Dor Verbin and Pratul P. Srinivasan and Richard Szeliski and Jonathan T. Barron and Ben Mildenhall
Conf/Jour SIGGRAPH
Year 2023
Project BakedSDF
Paper BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (readpaper.com)

image.png

  • 对前景物体采用类似VolSDF的方法训练Modeling density with an SDF
  • 使用Marching Cube 的方法来提取高分辨率网格Baking a high-resolution mesh
  • Modeling view-dependent appearance，对baked后的高分辨率网格上顶点：采用漫反射颜色和球形高斯叶（前景3个波瓣，远处背景使用1个波瓣）
    • $\mathbf{C}=\mathbf{c}_{d}+\sum_{i=1}^{N}\mathbf{c}_{i}\exp\left(\lambda_{i}\left(\mu_{i}\cdot\mathbf{d}-1\right)\right).$
Read more »

Posted on Edited on In
6.1k 22 mins.
Title Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection
Author Wenhang Ge1 Tao Hu 2 Haoyu Zhao 1 Shu Liu 3 Ying-Cong Chen1,∗
Conf/Jour ICCV Oral
Year 2023
Project Ref-NeuS (g3956.github.io)
Paper Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection (readpaper.com)
  • Anomaly Detection for Reflection Score + Visibility Identification for Reflection Score
  • Reflection Direction Dependent Radiance反射感知的光度损失
Read more »

Posted on Edited on In
5.3k 19 mins.
Title Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
Author Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, Yaron Lipman
Conf/Jour NeurIPS
Year 2020
Project Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (lioryariv.github.io)
Paper Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance (readpaper.com)

image.png

端到端的IDR：可以从masked的2D图像中学习3D几何、外观，允许粗略的相机估计

Read more »

Posted on Edited on In
5.5k 20 mins.
Title Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction
Author Fu, Qiancheng and Xu, Qingshan and Ong, Yew-Soon and Tao, Wenbi
Conf/Jour NeurIPS
Year 2022
Project GhiXu/Geo-Neus: Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction (NeurIPS 2022)
Paper Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction (readpaper.com) Geo-Neus: Geometry-Consistent Neural Implicit Surfaces Learning for Multi-view Reconstruction 基于神经隐式学习的多视图三维重建算法研究 - 中国知网

推到了SDF-based Volume Rendering 渲染的颜色监督会造成表面颜色和几何的偏差。(对渲染贡献权重最大的颜色值的位置并不是物体的表面) Bias in color rendering
几何先验：使用COLMAP产生的稀疏点来作为SDF的显示监督—>可以捕获强纹理的复杂几何细节
具有多视图立体约束的隐式曲面上的几何一致监督—>大面积的光滑区域

image.png

Read more »

Posted on Edited on In
3.8k 14 mins.
Title HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details
Author _Yiqun Wang, Ivan Skorokhodov, Peter Wonka_
Conf/Jour NeurIPS
Year 2022
Project yiqun-wang/HFS: HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details (NeurIPS 2022) (github.com)
Paper Improved surface reconstruction using high-frequency details (readpaper.com)

贡献：

  • 新的SDF与透明度$\alpha$关系函数，相较于NeuS更简单
  • 将SDF分解为两个独立隐函数的组合：基和位移。并利用自适应尺度约束对隐函数分布不理想的区域进行重点优化，可以重构出比以往工作更精细的曲面
Read more »

Posted on Edited on In
4.8k 17 mins.
Title Color-NeuS: Reconstructing Neural Implicit Surfaces with Color
Author Licheng Zhong1 , Lixin Yang1,2 , Kailin Li1, Haoyu Zhen1, Mei Han3, Cewu Lu1,2
Conf/Jour arXiv
Year 2023
Project Color-NeuS (colmar-zlicheng.github.io)
Paper Color-NeuS: Reconstructing Neural Implicit Surfaces with Color (readpaper.com)

集成了与视图无关的全局颜色变量和与视图相关的relight效果
image.png

Read more »

Posted on Edited on In
5.8k 21 mins.
Title NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images
Author Mingwu Zheng, Haiyu Zhang, Hongyu Yang, Di Huang
Conf/Jour CVPR
Year 2023
Project aejion/NeuFace: Official code for CVPR 2023 paper NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images. (github.com)
Paper NeuFace: Realistic 3D Neural Face Rendering from Multi-view Images (readpaper.com)

image.png

贡献：

  • BRDF+SDF+PBR新框架，端到端训练，重建出Face的外观和几何
  • 简单而新的低秩先验，镜面反射部分的Material Integral. 表示为线性组合的BRDF基
Read more »

Posted on Edited on In
4.4k 16 mins.
Title FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization
Author Jiawei Yang Marco Pavone Yue Wang
Conf/Jour CVPR
Year 2023
Project FreeNeRF: Frequency-regularized NeRF (jiawei-yang.github.io)
Paper FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization (readpaper.com)

Frequency regularized NeRF (FreeNeRF)
image.png
T为正则化持续时间，t为当前训练迭代，L为输入位置编码的长度

How：

  • High-frequency inputs cause the catastrophic failure in few-shot neural rendering.
    • 位置编码中高频信号可以使高频分量更快收敛，但是过快收敛将导致少样本神经渲染中灾难性的过拟合
    • 测试：将高频位置编码位设置为0，pos_enc[int(L * x%): ] = 0, ， L为位置编码的长度，x是可见比率
  • Frequency regularization enjoys the benefits of both high-frequency and low-frequency signals.
    • 频率正则化：根据训练时间steps，线性增加的频率mask，来正则化可见频谱。即刚开始使用低频，逐步增加高频信号的可见性
    • 频率正则化有助于降低在开始时导致灾难性故障的过度拟合风险，并避免在最终导致过度平滑的欠拟合
  • Occlusion regularization addresses the near-camera floaters.
    • 遮挡正则化，对相机附近密集场进行乘法
Read more »

Posted on Edited on In
6.6k 24 mins.
Title Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training
Author Julien Philip1, Valentin Deschaintre1
Conf/Jour The Eurographics Association
Year 2023
Project Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training (gradient-scaling.github.io)
Paper Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training (readpaper.com)

image.png

消除由近平面过度采样导致的摄像头附近漂浮物
可以通过几行代码简单的用于：

  • Mip-NeRF 360
  • InstantNGP
  • DVGO
  • TensoRF
Read more »

Posted on Edited on In
6.9k 25 mins.
Title NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction
Author Yiming Wang 1 Qin Han 1 Marc Habermann 2 Kostas Daniilidis 3 Christian Theobalt2 Lingjie Liu 2,3
Conf/Jour ICCV
Year 2023
Project NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (mpg.de)
Paper NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (readpaper.com)

image.png

在Neus基础上添加了：

  • 哈希编码加速
    • 定制的二阶导数反向传播计算
    • 渐进式学习策略(渐进添加高leve的哈希表)
  • 动态场景重建
    • 全局变换预测
    • 增量训练策略

主要代码通过cuda c++编写

Read more »