<jupyter_start><jupyter_text>This notebook contains the entire environment setup needed to run https://github.com/rosiezou/ssl_3d_recon in Google collab and/or AWS<jupyter_code>ls /usr/local/cuda-8.0/
# Check pyton requirements
! python --version
# Sanity check GPU and cuda version
!nvidia-smi

!nvcc --version
# Remove the version of CUDA installed on the machine
!apt-get --purge remove cuda* nvidia* libnvidia-*
# Steps below in this cell, may fail on AWS. That should be okay.
!dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge
!s
!apt autoremove
!apt-get update
# Install CUDA 8
!wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
!dpkg -i --force-overwrite cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
!apt-get update
!apt-get install cuda-8-0
# install will fail, need to force dpkg to overwrite the configuration file

# Note: Skip this cell for AWS. Not needed.

!wget http://archive.ubuntu.com/ubuntu/pool/main/m/mesa/libglx-mesa0_18.0.5-0ubuntu0~18.04.1_amd64.deb
!dpkg -i --force-overwrite libglx-mesa0_18.0.5-0ubuntu0~18.04.1_amd64.deb

!wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/nvidia-410_410.48-0ubuntu1_amd64.deb
!dpkg -i --force-overwrite nvidia-410_410.48-0ubuntu1_amd64.deb

!apt --fix-broken install
!apt-get install cuda-8-0
# Sanity check if the correct version of cuda is installed now.
!nvcc --version
# Experimental install of tensorflow 1.13
! pip install tensorflow==1.13.2

# Make sure that tensorflow is working
import tensorflow as tf
print(tf.sysconfig.get_lib())
# Missing module! :|
! pip install tflearn
# Downgrade GNU to a CUDA compatible version

# Note: Skip this cell for AWS. Not needed.

! apt install g++-4.8

! update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7
! update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50 --slave /usr/bin/g++ g++ /usr/bin/g++-4.8

# List available compilers
! update-alternatives --list gcc

# Set gcc 4.x as default
!update-alternatives --set gcc /usr/bin/gcc-4.8
# git repo clone
# Original author repository
! git clone https://github.com/klnavaneet/ssl_3d_recon.git

# OR Forked version of the repository with changes made to actually make the code work on collab.
! git clone https://github.com/rosiezou/ssl_3d_recon.git

# In order to run this on AWS, make sure to add the flag for -D_GLIBCXX_USE_CXX11_ABI=0 back in makefile as on https://github.com/klnavaneet/ssl_3d_recon/blob/master/makefile
# AWS runs with a GNU version  > 4.9. This flag will be needed in that case. Still relies on the extra tensorflow and cuda library paths provided in the custom makefile from 
# Makefile
# Note: Makefile should be inside src. Place it over there if it's not there.
# Also: Makefile and the shell scripts within chamfer_utils have been changed. Use the ones uploaded
# on the drive (For collab only).
# Makefile used on AWS will be different.
% cd /content/ssl_3d_recon/src
! make clean

! make
# Mount google drive to connect to the training data
# from google.colab import drive
# drive.mount('/content/drive')<jupyter_output><empty_output><jupyter_text>The steps below should be used for training the model on collab. This works as long as the training data is uploaded to my google drive. :)<jupyter_code># Following the steps in  https://github.com/klnavaneet/ssl_3d_recon/blob/master/README.md after running "make"

# % cd /content/ssl_3d_recon/src/utils/
# !pwd
# Note: The path to test data may point to my google drive folder for prototyping. Make sure the python script contains the right path.
# Create tfrecords file for OURS-CC model
# ! python create_tf_records.py
# Create tfrecords file for OURS-NN model
# ! python create_tf_records_knn.py
# Try training if possible
# % cd /content/ssl_3d_recon/
# ! bash run.sh<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10339545570589737, 'fraction_numerical': 0.034209854480469745, 'mean_word_length': 4.094928478543563, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 10, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '507168', 'license': 'MIT', 'n_tokens_mistral': 1458, 'n_tokens_neox': 1340, 'n_words': 445, 'path': 'SSL_python2_proto.ipynb', 'repo_name': 'rosiezou/ssl_3d_recon', 'size': '3917'}