<jupyter_start><jupyter_text>**Final Notebook of the AML Final Project***Customized UltraFastLaneDetection model through the backbone RedNetInvolution*[Link to UltraFastLaneDetection repository](https://github.com/cfzd/Ultra-Fast-Lane-Detection.git)[Link to RedNetInvolution repository](https://github.com/d-li14/involution.git)[Link to the leaderboard for the Tusimple dataset](https://paperswithcode.com/sota/lane-detection-on-tusimple)  Steps to train and test the model Mount the Google Drive<jupyter_code>from google.colab import drive
drive._mount('/content/drive')<jupyter_output><empty_output><jupyter_text>Before running the following cell:* change the path to the shared project folder<jupyter_code>import os

path_to_final_project = '/content/drive/MyDrive/University/Second_year/AML/AML-Final_Project'
os.environ['FINAL_PROJECT'] = path_to_final_project<jupyter_output><empty_output><jupyter_text>Get the customized code from our repository<jupyter_code>!git clone https://github.com/stdrr/AML-Final_Project.git<jupyter_output><empty_output><jupyter_text>Setup the environment: change working directory and install the required packages ("pip -q" makes pip quiet and no output is printed)<jupyter_code>%cd AML-Final_Project/code
!pip install -qr requirements.txt<jupyter_output><empty_output><jupyter_text>**Don't open the "Train the model" section if you have to perform only the test**  Train the model Link the data and the checkpoints' directories<jupyter_code>!ln -s $FINAL_PROJECT/data/tusimple_old ../tusimple
!ln -s $FINAL_PROJECT/checkpoints ../checkpoints<jupyter_output><empty_output><jupyter_text>Load Tensorboard to monitor the loss and the metrics<jupyter_code>%load_ext tensorboard
%tensorboard --logdir ../checkpoints/tusimple --bind_all 
!python UltraFastLaneDetection/train.py UltraFastLaneDetection/configs/tusimple.py<jupyter_output><empty_output><jupyter_text>Test the baseline **Run this section only if you are testing a baseline (ResNet) model. Go directly to the "Test the model" section otherwise** Import the original code<jupyter_code>!git clone https://github.com/cfzd/Ultra-Fast-Lane-Detection.git /content/UltraFastLaneDetection<jupyter_output><empty_output><jupyter_text>Create a symbolic link to the checkpoints folder, to the test set and to the results folder<jupyter_code>!ln -s $FINAL_PROJECT/checkpoints ../checkpoints
!ln -s $FINAL_PROJECT/data/tusimple_test ../tusimple_test
!ln -s $FINAL_PROJECT/results ../results<jupyter_output><empty_output><jupyter_text>Before running the test:* add the path to the checkpoint to test in `test_model`, e.g. `test_model='../checkpoints/{YOUR FOLDER}/{YOUR CHECKPOINT}'`* change the variable `model_name` with the name of the model you are testing* change the variable `epoch` with the number of the epoch of the checkpoint* change the variable `backbone` with the depth of the backbone<jupyter_code>test_model = '../checkpoints/tusimple/20211209_130934_lr_4e-04_b_32_baseline18/ep099.pth' # path to checkpoint
model_name = 'baseline18' # name of the model to test
epoch = '99' # epoch of the checkpoint
backbone = '18'

os.environ['TEST_MODEL'] = test_model
os.environ['MODEL_NAME'] = model_name
os.environ['EPOCH'] = epoch
os.environ['BACKBONE'] = backbone
!mkdir -p ../results/$MODEL_NAME/$EPOCH<jupyter_output><empty_output><jupyter_text>Run the test script<jupyter_code>!python ../../UltraFastLaneDetection/test.py UltraFastLaneDetection/configs/tusimple.py \
--data_root ../tusimple_test \
--backbone $BACKBONE \
--test_model $TEST_MODEL \
--test_work_dir ../results/$MODEL_NAME/$EPOCH<jupyter_output><empty_output><jupyter_text>Test the model's speed.Before running the following command:* open /content/UltraFastLaneDetection/speed_simple.py* change the backbone number at line 9 with the depth of the backbone of your model \[either 18 or 34\]<jupyter_code>!python ../../UltraFastLaneDetection/speed_simple.py<jupyter_output><empty_output><jupyter_text>**If you have to test another model that is not a baseline, please restart the kernel and re-run the initial section; then go to "Test the model"**  Test the model Create a symbolic link to the checkpoints folder, to the test set and to the results folder<jupyter_code>!ln -s $FINAL_PROJECT/checkpoints ../checkpoints
!ln -s $FINAL_PROJECT/data/tusimple_test ../tusimple_test
!ln -s $FINAL_PROJECT/results ../results<jupyter_output><empty_output><jupyter_text>Before running the test:* add the path to the checkpoint to test in `test_model`, e.g. `test_model='../checkpoints/{YOUR FOLDER}/{YOUR CHECKPOINT}'`* change the variable `model_name` with the name of the model you are testing* change the variable `epoch` with the number of the epoch of the checkpoint* change the variable `backbone` with the depth of the backbone<jupyter_code>test_model = '../checkpoints/tusimple/20211212_161502_lr_4e-04_b_32_rednet38_fixed/ep099.pth' # path to checkpoint
model_name = 'rednet38_fixed' # name of the model to test
epoch = '99' # epoch of the checkpoint
backbone = '38'

os.environ['TEST_MODEL'] = test_model
os.environ['MODEL_NAME'] = model_name
os.environ['EPOCH'] = epoch
os.environ['BACKBONE'] = backbone
!mkdir -p ../results/$MODEL_NAME/$EPOCH<jupyter_output><empty_output><jupyter_text>Before running the following command:* open the file /content/AML-Final_Project/code/cfg_file.json* make sure that the pretrained checkpoint of the backbone matches with the backbone of your modelRun the test script<jupyter_code>!python UltraFastLaneDetection/test.py UltraFastLaneDetection/configs/tusimple.py \
--data_root ../tusimple_test \
--backbone $BACKBONE \
--test_model $TEST_MODEL \
--test_work_dir ../results/$MODEL_NAME/$EPOCH<jupyter_output><empty_output><jupyter_text>Test the model's speed.Before running the following command:* open /content/AML-Final_Project/code/UltraFastLaneDetection/speed_simple.py* change the backbone number at line 22 with the depth of the backbone of your model<jupyter_code>!python UltraFastLaneDetection/speed_simple.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10193933366484337, 'fraction_numerical': 0.011437095972153158, 'mean_word_length': 6.3585365853658535, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '59060', 'license': 'MIT', 'n_tokens_mistral': 2055, 'n_tokens_neox': 1930, 'n_words': 526, 'path': 'code/Final_notebook.ipynb', 'repo_name': 'Jeremy98-alt/AML-Final_Project', 'size': '6033'}