<jupyter_start><jupyter_text>Prerequisites* Download dataset from https://shapenet.cs.stanford.edu/media/modelnet40_normal_resampled.zip and unzip.* Install [PCL](https://pointclouds.org/downloads/), [NumPy](https://numpy.org/install/), [Open3D](http://www.open3d.org/), and [pyquaternion](http://kieranwynn.github.io/pyquaternion/). 1. Prepare random uniform downsampled indices of pointsChange `root` in `gen_downsample_index.py` according to where the dataset is located. Run it for both splits (train and test).Note that this step is not a prerequisite for the following data preparation steps.<jupyter_code>%cd my_dataloader
!python gen_downsample_index.py
%cd ..<jupyter_output><empty_output><jupyter_text>2. Optional: convert txt file into PCD file for LRF computationtxt file could also parsed and used for the LRF computation.<jupyter_code>import open3d as o3d
import numpy as np
import glob
all_files = glob.glob('modelnet40_normal_resampled/*/*.txt')

for file in all_files:
    f = open(file)
    pc = []
    
    for line in f:
        l = line.split(',')
        pc.append([float(l[0]), float(l[1]), float(l[2])])

    pc = np.array(pc)
    # Create PointCloud object
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(pc)
    
    o3d.io.write_point_cloud(file[:-3] + "pcd", pcd)<jupyter_output><empty_output><jupyter_text>3. Compute LRFPCL implementation of FLARE can be used for the computation of LRFs ([Example usage of FLARE in PCL](https://github.com/PointCloudLibrary/pcl/blob/master/test/features/test_flare_estimation.cpp)).Save the computed LRFs in a file such that each row is `x_0 x_1 x_2 y_0 y_1 y_2 z_0 z_1 z_2` (split by whitespace). The file name should end with `_rot.txt`. For instance, if input file is lamp/lamp_0007.pcd, LRFs should be saved in lamp/lamp_0007_rot.txt 4. Convert LRF orientation to quaternion and find the indices of invalid points<jupyter_code>from pyquaternion import Quaternion

all_lrf_files = [f[:-4] + "_rot.txt" for f in all_files]
for f in all_lrf_files:
    file = open(f).readlines()
    quat = np.zeros((len(file), 4))
    wrong_ids = []
    for i, line in enumerate(file[1:]):  # assuming that there is a header row in the file
        lrf = line.split(" ")
        
        vecs = np.zeros((3, 3))
        vecs[0] = [float(lrf[0]), float(lrf[1]), float(lrf[2])]  # x
        vecs[1] = [float(lrf[3]), float(lrf[4]), float(lrf[5])]  # y
        vecs[2] = [float(lrf[6]), float(lrf[7]), float(lrf[8][:-1])] # z - don't take last char since it is \n
        
        rotation = vecs.transpose()
        try:
            potential = Quaternion._from_matrix(rotation, rtol=1e-03, atol=1e-03).q
            if all(p < 1e+20 for p in potential):
                quat[i] = potential
            else:
                quat[i] = np.array([0,0,0,0])
                wrong_ids.append(i)
        except:
            quat[i] = np.array([0,0,0,0])
            wrong_ids.append(i)
    np.savetxt(f[:-8] + '.qua', quat, encoding='ascii')
    np.savetxt(f[:-8] + '.idx', np.array(wrong_ids), encoding='ascii')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13451043338683788, 'fraction_numerical': 0.02375601926163724, 'mean_word_length': 3.4450784593437946, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 13, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '97166', 'license': 'MIT', 'n_tokens_mistral': 1150, 'n_tokens_neox': 1105, 'n_words': 286, 'path': 'dataset_preparation.ipynb', 'repo_name': 'tolgabirdal/qecnetworks', 'size': '3115'}