<jupyter_start><jupyter_text>Creating your own dataset from Google Images*by: <NAME> and <NAME>. Inspired by [<NAME>](https://www.pyimagesearch.com/2017/12/04/how-to-create-a-deep-learning-dataset-using-google-images/)* In this tutorial we will see how to easily create an image dataset through Google Images. **Note**: You will have to repeat these steps for any new category you want to Google (e.g once for dogs and once for cats).<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
# You need to mount your google drive  to the /content/gdrive folder of your virtual computer
# located in the colab server

from google.colab import drive
drive.mount("/content/gdrive")
#drive.mount("/content/gdrive", force_remount=True)

from fastai.vision import *<jupyter_output><empty_output><jupyter_text>Get a list of URLs  Search and scroll Question 1: (1.1) Please download 3 categories of animal images from google. Download about 100 images for each category.  Go to [Google Images](http://images.google.com) and search for the images you are interested in. The more specific you are in your Google Search, the better the results and the less manual pruning you will have to do.Scroll down until you've seen all the images you want to download, or until you see a button that says 'Show more results'. All the images you scrolled past are now available to download. To get more, click on the button, and continue scrolling. The maximum number of images Google Images shows is 700.It is a good idea to put things you want to exclude into the search query, for instance if you are searching for the Eurasian wolf, "canis lupus lupus", it might be a good idea to exclude other variants:    "canis lupus lupus" -dog -arctos -familiaris -baileyi -occidentalisYou can also limit your results to show only photos by clicking on Tools and selecting Photos from the Type dropdown.   Download into file Question 1 (1.2) Move the downloaded files to your google dirve and make the names of the files in the form of *.csv. Now you must run some Javascript code in your browser which will save the URLs of all the images you want for you dataset.In Google Chrome press Ctrl+Shift+j on Windows/Linux and CmdOptj on macOS, and a small window the javascript 'Console' will appear. In Firefox press CtrlShiftk on Windows/Linux or CmdOptk on macOS. That is where you will paste the JavaScript commands.You will need to get the urls of each of the images. Before running the following commands, you may want to disable ad blocking extensions (uBlock, AdBlockPlus etc.) in Chrome. Otherwise the window.open() command doesn't work. Then you can run the following commands:```javascripturls=Array.from(document.querySelectorAll('.rg_i')).map(el=> el.hasAttribute('data-src')?el.getAttribute('data-src'):el.getAttribute('data-iurl'));window.open('data:text/csv;charset=utf-8,' + escape(urls.join('\n')));```  upload urls file into /content folder You will need to run this cell once per each category. The following is an illustration.<jupyter_code>path = Path('gdrive/My Drive/fastai-v3/data/bears')<jupyter_output><empty_output><jupyter_text>Download images Now you will need to download your images from their respective urls.fast.ai has a function that allows you to do just that. You just have to specify the urls filename as well as the destination folder and this function will download and save all images that can be opened. If they have some problem in being opened, they will not be saved.Let's download our images! Notice you can choose a maximum number of images to be downloaded. In this case we will not download all the urls.You will need to run this line once for every category.  The following is an illustration.<jupyter_code>classes = ['teddys','grizzly','black']
# For example, Do this when download "urls_black.csv' file:
folder = 'teddys'
dest = path/folder
file = 'urls_teddy.csv'
download_images(dest/file, dest, max_pics=100)
# Question 2: Explain what happens when you execute download_images() statement.

for c in classes:
    print(c)
    verify_images(path/c, delete=True, max_size=500)<jupyter_output>teddys<jupyter_text>View data<jupyter_code>np.random.seed(42)
data = ImageDataBunch.from_folder(path, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224, num_workers=4).normalize(imagenet_stats
                                                                    
# Question 3: Explain how the categories of the images are extracted when you execute the above statement.<jupyter_output><empty_output><jupyter_text>Good! Let's take a look at some of our pictures then.  Train model<jupyter_code>learn = cnn_learner(data, models.resnet34, metrics=error_rate)
# Question 4: 4.1) cnn_learner() has input paramters other than the shown above.
# One of them is pretrained, which is True by default when you do not specify it. 
# What happens when you specify pretrained=True as in 
# learn = cnn_learner(data, models.resnet34, metrics=error_rate, pretrained=False) 

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
# Question 5: What does your confusion matrix tell you about the prediction capability of your neural network?
# Explain in a conscise manner but do not omit important points.
#Question 6: use interp.plot_top_losses() to find out the prediction capability of your neural network?
# Explain in a conscise manner but do not omit important points.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07281375777533845, 'fraction_numerical': 0.008964507866813026, 'mean_word_length': 4.7668776371308015, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '1621', 'license': 'Apache-2.0', 'n_tokens_mistral': 1540, 'n_tokens_neox': 1472, 'n_words': 702, 'path': 'MidTermPart2.ipynb', 'repo_name': 'moonryul/course-v3', 'size': '5493'}