Yuliang commited on
Commit
0718695
1 Parent(s): de4d7c5

clean unnecessary libs

Browse files
README.md CHANGED
@@ -14,7 +14,7 @@
14
  ·
15
  <a href="https://ps.is.tuebingen.mpg.de/person/black"><strong>Michael J. Black</strong></a>
16
  </p>
17
- <h2 align="center">CVPR 2023</h2>
18
  <div align="center">
19
  <img src="./assets/teaser.gif" alt="Logo" width="100%">
20
  </div>
@@ -48,7 +48,7 @@ ECON is designed for "Human digitization from a color image", which combines the
48
 
49
  ## News :triangular_flag_on_post:
50
 
51
- - [2023/02/27] ECON got accepted by CVPR 2023!
52
  - [2023/01/12] [Carlos Barreto](https://twitter.com/carlosedubarret/status/1613252471035494403) creates a Blender Addon ([Download](https://carlosedubarreto.gumroad.com/l/CEB_ECON), [Tutorial](https://youtu.be/sbWZbTf6ZYk)).
53
  - [2023/01/08] [Teddy Huang](https://github.com/Teddy12155555) creates [install-with-docker](docs/installation-docker.md) for ECON .
54
  - [2023/01/06] [Justin John](https://github.com/justinjohn0306) and [Carlos Barreto](https://github.com/carlosedubarreto) creates [install-on-windows](docs/installation-windows.md) for ECON .
@@ -60,6 +60,28 @@ ECON is designed for "Human digitization from a color image", which combines the
60
  - [ ] Blender add-on for FBX export
61
  - [ ] Full RGB texture generation
62
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  <br>
64
 
65
  <!-- TABLE OF CONTENTS -->
 
14
  ·
15
  <a href="https://ps.is.tuebingen.mpg.de/person/black"><strong>Michael J. Black</strong></a>
16
  </p>
17
+ <h2 align="center">CVPR 2023 (Highlight)</h2>
18
  <div align="center">
19
  <img src="./assets/teaser.gif" alt="Logo" width="100%">
20
  </div>
 
48
 
49
  ## News :triangular_flag_on_post:
50
 
51
+ - [2023/02/27] ECON got accepted by CVPR 2023 as Highlight (top 10%)!
52
  - [2023/01/12] [Carlos Barreto](https://twitter.com/carlosedubarret/status/1613252471035494403) creates a Blender Addon ([Download](https://carlosedubarreto.gumroad.com/l/CEB_ECON), [Tutorial](https://youtu.be/sbWZbTf6ZYk)).
53
  - [2023/01/08] [Teddy Huang](https://github.com/Teddy12155555) creates [install-with-docker](docs/installation-docker.md) for ECON .
54
  - [2023/01/06] [Justin John](https://github.com/justinjohn0306) and [Carlos Barreto](https://github.com/carlosedubarreto) creates [install-on-windows](docs/installation-windows.md) for ECON .
 
60
  - [ ] Blender add-on for FBX export
61
  - [ ] Full RGB texture generation
62
 
63
+ ## Key idea: d-BiNI
64
+
65
+ d-BiNI jointly optimizes front-back 2.5D surfaces such that: (1) high-frequency surface details agree with normal maps, (2) low-frequency surface variations, including discontinuities, align with SMPL-X surfaces, and (3) front-back 2.5D surface silhouettes are coherent with each other.
66
+
67
+ |Front-view|Back-view|Side-view|
68
+ |:--:|:--:|:---:|
69
+ |![](assets/front-45.gif)|![](assets/back-45.gif)|![](assets/double-90.gif)||
70
+
71
+ <details><summary>Please consider cite <strong>BiNI</strong> if it also helps on your project</summary>
72
+
73
+ ```bibtex
74
+ @inproceedings{cao2022bilateral,
75
+ title={Bilateral normal integration},
76
+ author={Cao, Xu and Santo, Hiroaki and Shi, Boxin and Okura, Fumio and Matsushita, Yasuyuki},
77
+ booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part I},
78
+ pages={552--567},
79
+ year={2022},
80
+ organization={Springer}
81
+ }
82
+ ```
83
+ </details>
84
+
85
  <br>
86
 
87
  <!-- TABLE OF CONTENTS -->
configs/econ.yaml CHANGED
@@ -35,5 +35,5 @@ bni:
35
  face_thres: 6e-2
36
  thickness: 0.02
37
  hps_type: "pixie"
38
- texture_src: "SD"
39
  cut_intersection: True
 
35
  face_thres: 6e-2
36
  thickness: 0.02
37
  hps_type: "pixie"
38
+ texture_src: "image"
39
  cut_intersection: True
docs/installation-ubuntu.md CHANGED
@@ -10,11 +10,11 @@ cd ECON
10
  ## Environment
11
 
12
  - Ubuntu 20 / 18, (Windows as well, see [issue#7](https://github.com/YuliangXiu/ECON/issues/7))
13
- - **CUDA=11.4, GPU Memory > 12GB**
14
  - Python = 3.8
15
  - PyTorch >= 1.13.0 (official [Get Started](https://pytorch.org/get-started/locally/))
16
  - Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
17
- - PyTorch3D (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
18
 
19
  ```bash
20
 
@@ -26,6 +26,9 @@ conda env create -f environment.yaml
26
  conda activate econ
27
  pip install -r requirements.txt
28
 
 
 
 
29
  # install libmesh & libvoxelize
30
  cd lib/common/libmesh
31
  python setup.py build_ext --inplace
 
10
  ## Environment
11
 
12
  - Ubuntu 20 / 18, (Windows as well, see [issue#7](https://github.com/YuliangXiu/ECON/issues/7))
13
+ - **CUDA=11.6, GPU Memory > 12GB**
14
  - Python = 3.8
15
  - PyTorch >= 1.13.0 (official [Get Started](https://pytorch.org/get-started/locally/))
16
  - Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
17
+ - PyTorch3D = 0.7.1 (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
18
 
19
  ```bash
20
 
 
26
  conda activate econ
27
  pip install -r requirements.txt
28
 
29
+ # the installation(incl. compilation) of PyTorch3D will take ~20min
30
+ pip install git+https://github.com/facebookresearch/[email protected]
31
+
32
  # install libmesh & libvoxelize
33
  cd lib/common/libmesh
34
  python setup.py build_ext --inplace
docs/installation-windows.md CHANGED
@@ -31,11 +31,11 @@ cd ECON
31
  ## Environment
32
 
33
  - Windows 10 / 11
34
- - **CUDA=11.4**
35
  - Python = 3.8
36
  - PyTorch >= 1.12.1 (official [Get Started](https://pytorch.org/get-started/locally/))
37
  - Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
38
- - PyTorch3D (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
39
 
40
  ```bash
41
  # install required packages
@@ -44,18 +44,10 @@ conda env create -f environment-windows.yaml
44
  conda activate econ
45
 
46
  # install pytorch and cupy
47
- pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113
48
- pip install -r requirements-win.txt
49
  pip install cupy-cuda11x
50
-
51
- ## If you have a RTX 30 series GPU then run this cmd below for installing neural_voxelization_layer
52
- pip install git+https://github.com/YuliangXiu/neural_voxelization_layer.git
53
- ## If you have GPU below RTX 30 series then you gotta build neural_voxelization_layer (steps below)
54
- git clone https://github.com/justinjohn0306/neural_voxelization_layer.git
55
-
56
- cd neural_voxelization_layer
57
- python setup install
58
- cd..
59
 
60
  # install libmesh & libvoxelize
61
  cd lib/common/libmesh
 
31
  ## Environment
32
 
33
  - Windows 10 / 11
34
+ - **CUDA=11.3**
35
  - Python = 3.8
36
  - PyTorch >= 1.12.1 (official [Get Started](https://pytorch.org/get-started/locally/))
37
  - Cupy >= 11.3.0 (offcial [Installation](https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi))
38
+ - PyTorch3D = 0.7.1 (official [INSTALL.md](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), recommend [install-from-local-clone](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md#2-install-from-a-local-clone))
39
 
40
  ```bash
41
  # install required packages
 
44
  conda activate econ
45
 
46
  # install pytorch and cupy
47
+ pip install torch==1.12.1+cu113 torchvision==0.13.1+cu113 --extra-index-url https://download.pytorch.org/whl/cu113
48
+ pip install -r requirements.txt
49
  pip install cupy-cuda11x
50
+ pip install git+https://github.com/facebookresearch/[email protected]
 
 
 
 
 
 
 
 
51
 
52
  # install libmesh & libvoxelize
53
  cd lib/common/libmesh
environment-windows.yaml CHANGED
@@ -1,6 +1,7 @@
1
  name: econ
2
  channels:
3
  - nvidia
 
4
  - conda-forge
5
  - fvcore
6
  - iopath
@@ -8,6 +9,7 @@ channels:
8
  - defaults
9
  dependencies:
10
  - python=3.8
 
11
  - fvcore
12
  - iopath
13
  - cupy
 
1
  name: econ
2
  channels:
3
  - nvidia
4
+ - pytorch
5
  - conda-forge
6
  - fvcore
7
  - iopath
 
9
  - defaults
10
  dependencies:
11
  - python=3.8
12
+ - pytorch-cuda=11.3
13
  - fvcore
14
  - iopath
15
  - cupy
environment.yaml CHANGED
@@ -9,11 +9,12 @@ channels:
9
  - defaults
10
  dependencies:
11
  - python=3.8
12
- - pytorch
 
 
13
  - torchvision
14
  - fvcore
15
  - iopath
16
- - nvidiacub
17
  - pyembree
18
  - cupy
19
  - cython
 
9
  - defaults
10
  dependencies:
11
  - python=3.8
12
+ - pytorch-cuda=11.7
13
+ - pytorch=1.13.0
14
+ - nvidiacub
15
  - torchvision
16
  - fvcore
17
  - iopath
 
18
  - pyembree
19
  - cupy
20
  - cython
lib/dataset/EvalDataset.py CHANGED
@@ -239,13 +239,10 @@ class EvalDataset:
239
  mesh_path = data_dict["mesh_path"]
240
  scale = data_dict["scale"]
241
 
242
- # tinyobjloader has wired bug
243
  scan_mesh = trimesh.load(mesh_path)
244
  verts = scan_mesh.vertices
245
  faces = scan_mesh.faces
246
 
247
- # verts, faces = obj_loader(mesh_path, with_uv=False)
248
-
249
  mesh = HoppeMesh(verts * scale, faces)
250
 
251
  return {
 
239
  mesh_path = data_dict["mesh_path"]
240
  scale = data_dict["scale"]
241
 
 
242
  scan_mesh = trimesh.load(mesh_path)
243
  verts = scan_mesh.vertices
244
  faces = scan_mesh.faces
245
 
 
 
246
  mesh = HoppeMesh(verts * scale, faces)
247
 
248
  return {
lib/dataset/TestDataset.py CHANGED
@@ -144,47 +144,6 @@ class TestDataset:
144
  ]).unsqueeze(0).to(self.device),
145
  }
146
 
147
- def compute_voxel_verts(self, body_pose, global_orient, betas, trans, scale):
148
-
149
- smpl_path = osp.join(self.smpl_data.model_dir, "smpl/SMPL_NEUTRAL.pkl")
150
- tetra_path = osp.join(self.smpl_data.tedra_dir, "tetra_neutral_adult_smpl.npz")
151
- smpl_model = TetraSMPLModel(smpl_path, tetra_path, "adult")
152
-
153
- pose = torch.cat([global_orient[0], body_pose[0]], dim=0)
154
- smpl_model.set_params(rotation_matrix_to_angle_axis(rot6d_to_rotmat(pose)), beta=betas[0])
155
-
156
- verts = (
157
- np.concatenate([smpl_model.verts, smpl_model.verts_added], axis=0) * scale.item() +
158
- trans.detach().cpu().numpy()
159
- )
160
- faces = (
161
- np.loadtxt(
162
- osp.join(self.smpl_data.tedra_dir, "tetrahedrons_neutral_adult.txt"),
163
- dtype=np.int32,
164
- ) - 1
165
- )
166
-
167
- pad_v_num = int(8000 - verts.shape[0])
168
- pad_f_num = int(25100 - faces.shape[0])
169
-
170
- verts = (
171
- np.pad(verts, ((0, pad_v_num),
172
- (0, 0)), mode="constant", constant_values=0.0).astype(np.float32) * 0.5
173
- )
174
- faces = np.pad(faces, ((0, pad_f_num), (0, 0)), mode="constant",
175
- constant_values=0.0).astype(np.int32)
176
-
177
- verts[:, 2] *= -1.0
178
-
179
- voxel_dict = {
180
- "voxel_verts": torch.from_numpy(verts).to(self.device).unsqueeze(0).float(),
181
- "voxel_faces": torch.from_numpy(faces).to(self.device).unsqueeze(0).long(),
182
- "pad_v_num": torch.tensor(pad_v_num).to(self.device).unsqueeze(0).long(),
183
- "pad_f_num": torch.tensor(pad_f_num).to(self.device).unsqueeze(0).long(),
184
- }
185
-
186
- return voxel_dict
187
-
188
  def __getitem__(self, index):
189
 
190
  img_path = self.subject_list[index]
 
144
  ]).unsqueeze(0).to(self.device),
145
  }
146
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
147
  def __getitem__(self, index):
148
 
149
  img_path = self.subject_list[index]
lib/dataset/mesh_util.py CHANGED
@@ -21,7 +21,6 @@ import torchvision
21
  import trimesh
22
  import json
23
  import open3d as o3d
24
- import tinyobjloader
25
  import os.path as osp
26
  import _pickle as cPickle
27
  from termcolor import colored
@@ -110,7 +109,6 @@ class SMPLX:
110
  self.smplx_to_smpl = cPickle.load(open(self.smplx_to_smplx_path, "rb"))
111
 
112
  self.model_dir = osp.join(self.current_dir, "models")
113
- self.tedra_dir = osp.join(self.current_dir, "../tedra_data")
114
 
115
  self.ghum_smpl_pairs = torch.tensor(
116
  [
@@ -288,33 +286,6 @@ def part_removal(full_mesh, part_mesh, thres, device, smpl_obj, region, clean=Tr
288
  return full_mesh
289
 
290
 
291
- def obj_loader(path, with_uv=True):
292
- # Create reader.
293
- reader = tinyobjloader.ObjReader()
294
-
295
- # Load .obj(and .mtl) using default configuration
296
- ret = reader.ParseFromFile(path)
297
-
298
- # note here for wavefront obj, #v might not equal to #vt, same as #vn.
299
- attrib = reader.GetAttrib()
300
- v = np.array(attrib.vertices).reshape(-1, 3)
301
- vt = np.array(attrib.texcoords).reshape(-1, 2)
302
-
303
- shapes = reader.GetShapes()
304
- tri = shapes[0].mesh.numpy_indices().reshape(-1, 9)
305
- f_v = tri[:, [0, 3, 6]]
306
- f_vt = tri[:, [2, 5, 8]]
307
-
308
- if with_uv:
309
- face_uvs = vt[f_vt].mean(axis=1) #[m, 2]
310
- vert_uvs = np.zeros((v.shape[0], 2), dtype=np.float32) #[n, 2]
311
- vert_uvs[f_v.reshape(-1)] = vt[f_vt.reshape(-1)]
312
-
313
- return v, f_v, vert_uvs, face_uvs
314
- else:
315
- return v, f_v
316
-
317
-
318
  class HoppeMesh:
319
  def __init__(self, verts, faces, uvs=None, texture=None):
320
  """
 
21
  import trimesh
22
  import json
23
  import open3d as o3d
 
24
  import os.path as osp
25
  import _pickle as cPickle
26
  from termcolor import colored
 
109
  self.smplx_to_smpl = cPickle.load(open(self.smplx_to_smplx_path, "rb"))
110
 
111
  self.model_dir = osp.join(self.current_dir, "models")
 
112
 
113
  self.ghum_smpl_pairs = torch.tensor(
114
  [
 
286
  return full_mesh
287
 
288
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
289
  class HoppeMesh:
290
  def __init__(self, verts, faces, uvs=None, texture=None):
291
  """
lib/net/IFGeoNet.py CHANGED
@@ -2,9 +2,7 @@ from pickle import TRUE
2
  import torch
3
  import torch.nn as nn
4
  import torch.nn.functional as F
5
- from lib.net.voxelize import Voxelization
6
  from lib.net.geometry import orthogonal
7
- from lib.dataset.mesh_util import read_smpl_constants, SMPLX
8
 
9
 
10
  class SelfAttention(torch.nn.Module):
@@ -88,31 +86,11 @@ class IFGeoNet(nn.Module):
88
  self.conv3_1_bn = nn.InstanceNorm3d(128)
89
  self.conv4_1_bn = nn.InstanceNorm3d(128)
90
 
91
- self.smplx = SMPLX()
92
- voxel_param = read_smpl_constants(self.smplx.tedra_dir)
93
-
94
- self.voxelization = Voxelization(
95
- torch.ones_like(voxel_param["smpl_vertex_code"]),
96
- torch.ones_like(voxel_param["smpl_face_code"]),
97
- voxel_param["smpl_faces"],
98
- voxel_param["smpl_tetras"],
99
- volume_res=cfg.dataset.voxel_res,
100
- sigma=0.05,
101
- smooth_kernel_size=7,
102
- batch_size=cfg.batch_size,
103
- )
104
-
105
  self.l1_loss = nn.SmoothL1Loss()
106
 
107
  def forward(self, batch):
108
 
109
- if "body_voxels" in batch.keys():
110
- x_smpl = batch["body_voxels"]
111
- else:
112
- with torch.no_grad():
113
- self.voxelization.update_param(batch["voxel_faces"])
114
- x_smpl = self.voxelization(batch["voxel_verts"])[:, 0] #[B, 128, 128, 128]
115
-
116
  p = orthogonal(batch["samples_geo"].permute(0, 2, 1),
117
  batch["calib"]).permute(0, 2, 1) #[2, 60000, 3]
118
  x = batch["depth_voxels"] #[B, 128, 128, 128]
 
2
  import torch
3
  import torch.nn as nn
4
  import torch.nn.functional as F
 
5
  from lib.net.geometry import orthogonal
 
6
 
7
 
8
  class SelfAttention(torch.nn.Module):
 
86
  self.conv3_1_bn = nn.InstanceNorm3d(128)
87
  self.conv4_1_bn = nn.InstanceNorm3d(128)
88
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
  self.l1_loss = nn.SmoothL1Loss()
90
 
91
  def forward(self, batch):
92
 
93
+ x_smpl = batch["body_voxels"]
 
 
 
 
 
 
94
  p = orthogonal(batch["samples_geo"].permute(0, 2, 1),
95
  batch["calib"]).permute(0, 2, 1) #[2, 60000, 3]
96
  x = batch["depth_voxels"] #[B, 128, 128, 128]
lib/net/IFGeoNet_nobody.py CHANGED
@@ -2,9 +2,7 @@ from pickle import TRUE
2
  import torch
3
  import torch.nn as nn
4
  import torch.nn.functional as F
5
- from lib.net.voxelize import Voxelization
6
  from lib.net.geometry import orthogonal
7
- from lib.dataset.mesh_util import read_smpl_constants, SMPLX
8
 
9
 
10
  class SelfAttention(torch.nn.Module):
@@ -82,20 +80,6 @@ class IFGeoNet(nn.Module):
82
  self.conv3_1_bn = nn.InstanceNorm3d(128)
83
  self.conv4_1_bn = nn.InstanceNorm3d(128)
84
 
85
- self.smplx = SMPLX()
86
- voxel_param = read_smpl_constants(self.smplx.tedra_dir)
87
-
88
- self.voxelization = Voxelization(
89
- torch.ones_like(voxel_param["smpl_vertex_code"]),
90
- torch.ones_like(voxel_param["smpl_face_code"]),
91
- voxel_param["smpl_faces"],
92
- voxel_param["smpl_tetras"],
93
- volume_res=cfg.dataset.voxel_res,
94
- sigma=0.05,
95
- smooth_kernel_size=7,
96
- batch_size=cfg.batch_size,
97
- )
98
-
99
  self.l1_loss = nn.SmoothL1Loss()
100
 
101
  def forward(self, batch):
 
2
  import torch
3
  import torch.nn as nn
4
  import torch.nn.functional as F
 
5
  from lib.net.geometry import orthogonal
 
6
 
7
 
8
  class SelfAttention(torch.nn.Module):
 
80
  self.conv3_1_bn = nn.InstanceNorm3d(128)
81
  self.conv4_1_bn = nn.InstanceNorm3d(128)
82
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83
  self.l1_loss = nn.SmoothL1Loss()
84
 
85
  def forward(self, batch):
requirements-win.txt DELETED
@@ -1,19 +0,0 @@
1
- matplotlib
2
- scikit-image
3
- trimesh
4
- rtree
5
- pytorch_lightning
6
- kornia>0.4.0
7
- chumpy
8
- opencv-python
9
- opencv_contrib_python
10
- scikit-learn
11
- protobuf
12
- dataclasses
13
- mediapipe
14
- einops
15
- boto3
16
- open3d
17
- tinyobjloader==2.0.0rc7
18
- git+https://github.com/facebookresearch/pytorch3d.git
19
- git+https://github.com/YuliangXiu/rembg.git
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
requirements.txt CHANGED
@@ -14,7 +14,4 @@ mediapipe
14
  einops
15
  boto3
16
  open3d
17
- tinyobjloader==2.0.0rc7
18
- git+https://github.com/facebookresearch/pytorch3d.git
19
- git+https://github.com/YuliangXiu/neural_voxelization_layer.git
20
  git+https://github.com/YuliangXiu/rembg.git
 
14
  einops
15
  boto3
16
  open3d
 
 
 
17
  git+https://github.com/YuliangXiu/rembg.git