<jupyter_start><jupyter_text>Preprocessing Images for Built-in Algorithms (Part 3/4)Download | Structure | **Preprocessing (Built-in)** | Train Model (Built-in)**Notes**: * This notebook should be used with the conda_amazonei_mxnet_p36 kernel* This notebook is part of a series of notebooks beginning with `01_download_data` and `02_structuring_data`. From here on it will focus on SageMaker's built-in algorithms. The next notebook in this series is `04a_builtin_training`.* You can also explore preprocessing with TensorFlow and PyTorch by running `03b_tensorflow_preprocessing` and `03c_pytorch_preprocessing`, respectively.  In this notebook we will explore the different ways to format your image dataset for SageMaker's built-in algorithms. The first involves creating a manifest file for the train and validations sets and the other has you creating .REC files (RecordIO format) which are single binary files made up of all the images for the train and validation sets. Since the RecordIO format is preferred, we will upload the .REC files to S3 for training in the nedxt notebook.   Overview*  [Dependencies](ipg3a.1)*  [Application/x-image format](ipg3a.2)*  [Application/x-recordio format](ipg3a.3) (preferred format)*  [Upload the data to S3](ipg3a.4)   Dependencies___<jupyter_code>import uuid
import boto3
import shutil
import urllib
import pickle
import pathlib
import sagemaker
import subprocess<jupyter_output><empty_output><jupyter_text>Load Category LabelsThe `category_labels` file was generated from the first notebook in this series `01_download_data.ipynb`. You will need to run that notebook before running the code here.<jupyter_code>with open('pickled_data/category_labels.pickle', 'rb') as f:
    category_labels = pickle.load(f)<jupyter_output><empty_output><jupyter_text>Application/x-image format___This format is also referred to as "Image Format" or "LST" format. The benefit of using this format is that it doesn't require any modification or restructuring of your dataset. Instead, you create a manifest of the images for your training set and validation set. These two manifests are separate `.lst` files which list all the images giving each of them a unique index, the class they belong to and the relative path to the image file from the main training folder. The data in the `.lst` file is in tab separated values.While its the easiest format to use, it requires SageMaker to do more work behind the scenes. For datasets with many images, this will cause training to take longer. For datasets with fewer images, the performance difference isn't as pronounced.Below are two examples of how to create your .LST manifest files. One uses your own code and the other uses a script from MXNet. If you want to create .REC files of your images, you should skip to Option 2.  Option 1: Manually generate the .LST files<jupyter_code>category_ids = {name: idx for idx, name in enumerate(sorted(category_labels.values()))}
print(category_ids)
image_paths = pathlib.Path('./data_structured').rglob('*.jpg')

for idx, p in enumerate(image_paths):
    image_id = f'{idx:010}'
    category = category_ids[p.parts[-2]]
    path = p.as_posix()
    split = p.parts[-3]
    with open(f'{split}.lst', 'a') as f:
        line = f"{image_id}\t{category}\t{path}\n"
        f.write(line)<jupyter_output><empty_output><jupyter_text>View the contents of the `train.lst` file<jupyter_code>!head train.lst<jupyter_output><empty_output><jupyter_text>Option 2: Use im2rec.py script to generate the .LST files<jupyter_code>script_url = 'https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/im2rec.py'
urllib.request.urlretrieve(script_url, "im2rec.py");<jupyter_output><empty_output><jupyter_text>`python im2rec.py --list --recursive LST_FILE_PREFIX DATA_DIR`* --list - generate an LST file* --recursive - looks inside subfolders for image data* LST_FILE_PREFIX - choose the name you want for the `.lst` file* DATA_DIR - relative path to directory with the data<jupyter_code>!python im2rec.py --list --recursive train data_structured/train
!python im2rec.py --list --recursive val data_structured/val<jupyter_output><empty_output><jupyter_text>View the contents of the `train.lst` file<jupyter_code>!head train.lst<jupyter_output><empty_output><jupyter_text>Application/x-recordio (preferred format)___This format is commonly referred to as RecordIO. It creates a new file for your each of your training and validation datasets with the `.rec` suffix. The `.rec` file is a single file that contains all of the images in the dataset so it can be streamed directly to the SageMaker training algorithm without the overhead involved with transfering thousands of individual files. For datasets with many images this provides a huge reduction in training time because SageMaker doesn't need to download all the image files before it can run the training algorithm. If you use the `im2rec.py` script, it will also resize the images for you as well. The benefits of resizing the files before saving them in the RecordIO format is that it'll reduce the amount of data you need to transfer to s3 and will also speed up trainging by doing the resizing ahead of time instead of at training.  1. Run Option 2 from application/x-image above and copy LST filesOnce you've run Option 2 from above then proceed below.<jupyter_code>recordio_dir = pathlib.Path('./data_recordio')
recordio_dir.mkdir(exist_ok=True)
shutil.copy('train.lst', 'data_recordio/');
shutil.copy('val.lst', 'data_recordio/');<jupyter_output><empty_output><jupyter_text>2. Generate .rec files in the RecordIO FormatOnce the `.lst` file is generated, the same `im2rec.py` script will also generate the `.rec` file.`python im2rec.py --resize 224 --quality 90 --num-thread 16 LST_FILE_PREFIX DATA_DIR/`* **--resize**: Have the script resize the files before saving them all to a `.rec` file. For the image classification algorithm the default dimensions are 224x224. Resizing now will also reduce the size of your `.rec` file.* **--quality**: Default settings will save the image data uncompressed. Adding some compression will keep the filesize of your `.rec` down especially if you're not resizing them.* **--num_thread**: Set how many threads to parallelize the work* **--LST_FILE_PREFIX**: Name of the `.lst` you're referencing for creating the `.rec` file* **--DATA_DIR**: Relative path directory which holds the data listed in the `.lst` file  Training dataset<jupyter_code>!python im2rec.py --resize 224 --quality 90 --num-thread 16 data_recordio/train data_structured/train<jupyter_output><empty_output><jupyter_text>Validation dataset<jupyter_code>!python im2rec.py --resize 224 --quality 90 --num-thread 16 data_recordio/val data_structured/val<jupyter_output><empty_output><jupyter_text>Upload the data to S3___In order for SageMaker's built-in algrorithms to train on the data, it must be stored in an S3 bucket. Here, we will create a bucket, but you can use an existing bucket if you like by replacing the `bucket_name` variable in the first line of the `else` statement below.  Create a bucket for your project<jupyter_code>if pathlib.Path('pickled_data/builtin_bucket_name.pickle').exists():
    with open('pickled_data/builtin_bucket_name.pickle', 'rb') as f:
        bucket_name = pickle.load(f)
        print('Bucket Name:', bucket_name)
else:
    bucket_name = f'sagemaker-builtin-ic-{str(uuid.uuid4())}'
    s3 = boto3.resource('s3')
    region = sagemaker.Session().boto_region_name
    bucket_config = {'LocationConstraint': region}
    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration=bucket_config)
    
    with open('pickled_data/builtin_bucket_name.pickle', 'wb') as f:
        pickle.dump(bucket_name, f)
    print('Bucket Name:', bucket_name)<jupyter_output><empty_output><jupyter_text>Upload .rec files to S3<jupyter_code>s3_uploader = sagemaker.s3.S3Uploader()

data_path = recordio_dir / 'train.rec'

data_s3_uri = s3_uploader.upload(
    local_path=data_path.as_posix(), 
    desired_s3_uri=f's3://{bucket_name}/data/train')
data_path = recordio_dir / 'val.rec'

data_s3_uri = s3_uploader.upload(
    local_path=data_path.as_posix(), 
    desired_s3_uri=f's3://{bucket_name}/data/val')<jupyter_output><empty_output><jupyter_text>Rollback to default version of SDK and TensorFlowOnly do this if you're done with this guide and want to use the same kernel for other notebooks with an incompatible version of the SageMaker SDK or TensorFlow.<jupyter_code># print(f'Original version: {original_sagemaker_version[0]}')
# print(f'Current version:  {sagemaker.__version__}')
# print('')
# print(f'Rolling back to {original_sagemaker_version[0]}')
# print('Restart notebook kernel to use changes.')
# print('')
# s = f'sagemaker=={original_sagemaker_version[0]}'
# !{sys.executable} -m pip install -q {s}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08903954802259886, 'fraction_numerical': 0.011299435028248588, 'mean_word_length': 5.3221428571428575, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '321182', 'license': 'Apache-2.0', 'n_tokens_mistral': 2709, 'n_tokens_neox': 2619, 'n_words': 989, 'path': 'prep_data/image_data_guide/03a_builtin_preprocessing.ipynb', 'repo_name': 'P15241328/amazon-sagemaker-examples', 'size': '8850'}