<jupyter_start><jupyter_text>Tutorial for Geoseg > __version__ ==   0.1.0>  __author__ ==  Go-Hiroaki Overview: 1. Evaluating with pretrained models> Test model performance by providing pretrained models 2. Re-training with provided dataset> Trained new models with provide training datastet 3. Training with personal dataset> Train and test models with your own dataset<jupyter_code>ls<jupyter_output>How-to.ipynb  __init__.py  [0m[01;34mdataset[0m/  [01;34mlogs[0m/    [01;34msrc[0m/
LICENSE       [01;34mcheckpoint[0m/  eva.sh    [01;34mresult[0m/  visSingle.py
README.md     [01;34mdata[0m/        [01;34mexample[0m/  run.sh   visSingleComparison.py<jupyter_text>1. Evaluating with pretrained models 1.1 Prepared and loaded dataset > Prepared  dataset```YOUR_DATASET/|-- img|   |-- train_1.png|   |-- train_2.png|   `-- |-- msk|   |-- train_1.png|   |-- train_2.png|   `-- |-- ref.csv|-- statistic.csv|-- train.csv|-- val.csv``` > Modified src/datasets.py to make sure YOUR_DATASET ```if __name__ == "__main__":     ====================== parameter initialization =======================     parser = argparse.ArgumentParser(description='ArgumentParser')    parser.add_argument('-idx', type=int, default=0,                        help='index of sample image')    args = parser.parse_args()    idx = args.idx    for root in ['YOUR_DATASET']:        for mode in ["IM", "IMS", "IME"]:            print("Load {}/{}.".format(root, mode))            trainset, valset = load_dataset(root, mode)                         print("Load train set = {} examples, val set = {} examples".format(                 len(trainset), len(valset)))            sample = trainset[idx]            trainset.show(idx)            sample = valset[idx]            valset.show(idx)            print("\tsrc:", sample["src"].shape,                  "tar:", sample["tar"].shape,)``` > Run src/datasets.py> python src/datasets.pyif success, sample image will show up in example/  1.2 Download pretrained models> 1. FCN8s_iter_5000.pth [LINK](https://drive.google.com/open?id=1KHs7coyXAipz8t5cN_lbTC4MOYi8FddI)> 2. FCN16s_iter_5000.pth [LINK](https://drive.google.com/open?id=1wlORkMx_ykmHysShUKY4UcCYs-fVaen6)> 3. FCN32s_iter_5000.pth [LINK](https://drive.google.com/open?id=1OR_Sk66RAGtKrp0quvqazRkL0xtAH8RY)> 4. SegNet_iter_5000.pth [LINK](https://drive.google.com/open?id=1J0aRjFG-zOSSXnynm02VaYxjw1tjx-qC)> 5. UNet_iter_5000.pth [LINK](https://drive.google.com/open?id=17X0aCgRx3XXgH1fcfLoLwgcbWIzxZe5K)> 6. FPN_iter_5000.pth [LINK](https://drive.google.com/open?id=1fWrCnGQJBZTw7m5OZlQvH5-R_JJlBA-r)> 7. ResUNet_iter_5000.pth [LINK](https://drive.google.com/open?id=1jGs_PxEMXCshOzXdg9LuFJxe8kO39oxT)> 8. MC-FCN_iter_5000.pth [LINK](https://drive.google.com/open?id=1Kt_JmR0ZGXvK9kuTmDOek5l1SsHX4xhz)> 9. BR-Net_iter_5000.pth [LINK](https://drive.google.com/open?id=1rytD9tzAq2mne5yf3XEh-jTSHlvQvedT)> * Upcoming ...    After downloading corresponding pretrained models, save them at checkpoints/ .<jupyter_code>ls ./checkpoint/<jupyter_output>BRNet-3*1*24-NZ32km2_iter_5000.pth      FPN-3*1*24-NZ32km2_iter_5000.pth
BRNet-3*6*24-Vaihingen_iter_5000.pth    FPN-3*6*24-Vaihingen_iter_5000.pth
FCN16s-3*1*24-NZ32km2_iter_5000.pth     MCFCN-3*1*24-NZ32km2_iter_5000.pth
FCN16s-3*6*24-Vaihingen_iter_5000.pth   MCFCN-3*6*24-Vaihingen_iter_5000.pth
FCN32s-3*1*24-NZ32km2_iter_5000.pth     SegNet-3*1*24-NZ32km2_iter_5000.pth
FCN32s-3*6*24-PotsdamRGB_iter_5000.pth  SegNet-3*6*24-Vaihingen_iter_5000.pth
FCN32s-3*6*24-Vaihingen_iter_5000.pth   UNet-3*1*24-NZ32km2_iter_5000.pth
FCN8s-3*1*24-NZ32km2_iter_5000.pth      UNet-3*6*24-Vaihingen_iter_5000.pth
FCN8s-3*6*24-Vaihingen_iter_5000.pth    checkpoint.csv<jupyter_text>1.3 Run evaluation scripts* sinle model```visSingle.py -h   optional arguments:  -h, --help            show this help message and exit  -checkpoints CHECKPOINTS [CHECKPOINTS ...]                        checkpoints used for making prediction  -spaces SPACES [SPACES ...]                        barrier space for merging  -direction {horizontal,vertical}                        merge image direction  -disp_cols DISP_COLS  cols for displaying image  -edge_fn {shift,canny}                        method used for edge extraction  -gen_nb GEN_NB        number of generated image  -color COLOR          background color for generated rgb result  -partition PARTITION  partition of dataset for loading  -disk DISK            dilation level  -cuda CUDA            using cuda for optimization```    The generate result will show up at result/single   - BR-Net ![time](./result/single/BR-Net_canny_segmap_edge_0.png)   * multi models```visSingleComparison.py -h    optional arguments:    -h, --help            show this help message and exit    -checkpoints CHECKPOINTS [CHECKPOINTS ...]                        checkpoints used for making prediction    -spaces SPACES [SPACES ...]                        barrier spaces for merging    -direction {horizontal,vertical}                        merge image direction    -disp_cols DISP_COLS  cols for displaying image    -target {segmap,edge}                        target for model prediction [segmap, edge]    -edge_fn {shift,canny}                        method used for edge extraction    -gen_nb GEN_NB        number of generated image    -eval_fn {ov,precision,recall,f1_score,jaccard,kappa}                        method used for evaluate performance    -significance SIGNIFICANCE                        significant different level between methods    -color COLOR          background color for generated rgb result    -partition PARTITION  partition of dataset for loading    -disk DISK            dilation level    -batch_size BATCH_SIZE                        batch size for model prediction    -cuda CUDA            using cuda for optimization```  The generate result will show up at result/single-comparison   - Segmap FCN32s_FCN16s_FCN8s ![time](./result/single-comparison/segmap_FCN32s_FCN16s_FCN8s_1.png)   - Edge FCN32s_FCN16s_FCN8s ![time](./result/single-comparison/edge_FCN32s_FCN16s_FCN8s_1.png)  2. Re-train with provided dataset 2.1 Download training dataset> Training dataset [LINK](https://drive.google.com/file/d/1boGcJz9TyK9XB4GUhjCHVu8XGtbgjjbi/view?usp=sharing).Unzip and place to datasets/ 2.2 Run training scripts```python src/train.py -husage: train.py [-h] [-root ROOT] [-net NET] [-base_kernel BASE_KERNEL] [-trigger {epoch,iter}] [-interval INTERVAL]                [-terminal TERMINAL] [-batch_size BATCH_SIZE] [-lr LR] [-cuda CUDA]ArgumentParseroptional arguments:  -h, --help            show this help message and exit  -root ROOT            root dir of dataset for training models  -net NET              network type for training  -base_kernel BASE_KERNEL                        base number of kernels  -trigger {epoch,iter}                        trigger type for logging  -interval INTERVAL    interval for logging  -terminal TERMINAL    terminal for training  -batch_size BATCH_SIZE                        batch_size for training  -lr LR                learning rate for optimization  -cuda CUDA            using cuda for optimization```    3. Training with personal dataset 3.1 Prepare your own dataset 3.2 Run training scripts<jupyter_code>Step<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10997935306262904, 'fraction_numerical': 0.04969029593943565, 'mean_word_length': 3.0501672240802677, 'pattern_counts': {'":': 1, '<': 11, '<?xml version=': 0, '>': 31, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '4946', 'license': 'MIT', 'n_tokens_mistral': 2687, 'n_tokens_neox': 2434, 'n_words': 541, 'path': 'How-to.ipynb', 'repo_name': 'nthuy190991/geoseg', 'size': '7265'}