<jupyter_start><jupyter_text>Download datasets  Note the datasets could be very large! Feel free to check the [dataset website](https://sites.google.com/eng.ucsd.edu/ucsdbookgraph/home) and download files you really need. **The datasets are hosted on the google drive. Thus downloading without GUI requires installing the [gdown](https://github.com/wkentaro/gdown) package**:  Download in bash commands**We can install the gdown package in shell:**`$ pip install gdown`**and download the file by specifying its gdrive ID in the following command**`$ gdown 'https://drive.google.com/uc?id={fileID}'`  Download in python**We can also install the gdown package here and download files in python**<jupyter_code>import sys
!{sys.executable} -m pip install gdown
import pandas as pd
import gdown
import os<jupyter_output><empty_output><jupyter_text>**Specify your directory here:**<jupyter_code>DIR = './'
OUT_DIR = '/home/mengting/'<jupyter_output><empty_output><jupyter_text>**Load the google drive ids of the datasets**<jupyter_code>file_ids = pd.read_csv(os.path.join(DIR, 'gdrive_id.csv'))
display(file_ids)<jupyter_output><empty_output><jupyter_text>**Now we can create a fileName-gdriveID map and a function to download files by name**<jupyter_code>file_id_map = dict(zip(file_ids['name'].values, file_ids['id'].values))

def download_by_name(fname, output=None, quiet=False):
    if fname in file_id_map:
        url = 'https://drive.google.com/uc?id='+file_id_map[fname]
        gdown.download(url, output=output, quiet=quiet)
    else:
        print('The file', fname, 'can not be found!')<jupyter_output><empty_output><jupyter_text>**Switch to the output directory specified before:**<jupyter_code>os.chdir(OUT_DIR)<jupyter_output><empty_output><jupyter_text>**Here we go!**<jupyter_code>download_by_name('goodreads_books_poetry.json.gz')<jupyter_output>Downloading...
From: https://drive.google.com/uc?id=1H6xUV48D5sa2uSF_BusW-IBJ7PCQZTS1
To: /home/mengting/goodreads_books_poetry.json.gz
27.9MB [00:00, 73.3MB/s]
{'fraction_non_alphanumeric': 0.1263001485884101, 'fraction_numerical': 0.008915304606240713, 'mean_word_length': 6.593984962406015, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '209854', 'license': 'Apache-2.0', 'n_tokens_mistral': 710, 'n_tokens_neox': 689, 'n_words': 163, 'path': 'download.ipynb', 'repo_name': 'xiw483/cse158-assignment2', 'size': '2019'}