<jupyter_start><jupyter_text>mymodelnet40- Prerocessed dataset of [modelnet40_normal_resampled](https://shapenet.cs.stanford.edu/media/modelnet40_normal_resampled.zip) for time-saving training.  Import packages.<jupyter_code>import os
import h5py
import numpy as np
import k3d

import torch

# my packages
from libs import tpcpath
from torchpcp.utils import pytorch_tools
from torchpcp.datasets.PointNet2.ModelNet import ModelNet40
from torchpcp.datasets.PointNet2.ModelNet import MyModelNet40<jupyter_output><empty_output><jupyter_text>Set a seed.<jupyter_code>pytorch_tools.set_seed(0)
device = pytorch_tools.select_device("cuda")<jupyter_output><empty_output><jupyter_text>Visualizetion function<jupyter_code>def visualize_pc(pc, colors=None):
    if colors is None:
        colors = np.full(len(pc), 0x000000, dtype=np.float64) # white: 0xFFFFFF
    plot = k3d.plot()
    points = k3d.points(pc, colors.astype(np.float32), point_size=0.04, shader='flat')
    plot += points
    plot.display()<jupyter_output><empty_output><jupyter_text>Load Dataset of modelnet40_normal_resampled.<jupyter_code>path = "/mnt/databox/datasets/ModelNet/modelnet40_normal_resampled"
# path = "/path/to/modelnet40_normal_resampled"
datasets = ModelNet40(path)<jupyter_output><empty_output><jupyter_text>Define output path.<jupyter_code># output_path = "/path/to/outputs/"
output_path = "/mnt/databox/datasets/ModelNet/"
preprocessed_dataset = "mymodelnet40"
preprocessed_dataset_path = os.path.join(output_path, preprocessed_dataset)<jupyter_output><empty_output><jupyter_text>Preprocess the dataset.<jupyter_code>def create_modelnet_dataset(file_name, dataset):
    loader = range(len(dataset))
    # loader = tqdm(loader, desc=file_name)
    with h5py.File(file_name, "w") as f:
        point_clouds = []
        labels = []
        for i in loader:
            points, label = dataset[i]
            point_clouds.append(points)
            labels.append(label)

        f.create_dataset("point_clouds", data=point_clouds, compression='gzip', compression_opts=1, dtype="float32")
        f.create_dataset("labels", data=labels, compression='gzip', compression_opts=1, dtype="uint8")

os.mkdir(preprocessed_dataset_path)
train_dataset_path = os.path.join(preprocessed_dataset_path, "train_modelnet40.h5")
create_modelnet_dataset(train_dataset_path, datasets["train"])
test_dataset_path = os.path.join(preprocessed_dataset_path, "test_modelnet40.h5")
create_modelnet_dataset(test_dataset_path, datasets["test"])<jupyter_output><empty_output><jupyter_text>Use MyModelNet40 (preprocessed modelnet40_normal_resampled).<jupyter_code>my_dataset = MyModelNet40(preprocessed_dataset_path)<jupyter_output><empty_output><jupyter_text>Visualize a point cloud.<jupyter_code>points, label = my_dataset["train"][3001]
visualize_pc(points[:, :3])
visualize_pc(points[:, 3:6])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11381830838844413, 'fraction_numerical': 0.023320570831883047, 'mean_word_length': 7.027932960893855, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '524288', 'license': 'MIT', 'n_tokens_mistral': 1059, 'n_tokens_neox': 1011, 'n_words': 144, 'path': 'examples/tools/mymodelnet40/mymodelnet40.ipynb', 'repo_name': 'Obarads/torch_point_cloud', 'size': '2873'}