<jupyter_start><jupyter_text>testing the reading from a zip file and extracting just one file<jupyter_code>z = ZipFile('/home/nfarrugi/bigdisk2/new/dropbox/SilentCitiesData/0003/Sebastien Puechmaille - 003_01.zip')
filelist = z.namelist()
filelist[:10]<jupyter_output><empty_output><jupyter_text>Pipeline--We will do this in the following steps : 1. For each folder or dropbox, search for an existing folder on bigdisk1 or bigdisk2. 2. If the folder is not founddd, this is a new site. 3. Parse the list of new sites and put them on bigdisk2, extract them flat. Save the list of new sites as sites for which we will need to restart "metadata_file"4. Parse all the other archives by checking their file list, compare with the csv of the corresponding site, copy if needed. If ANY copy is done, add the corresponding site to the list of sites to update. 5. Relaunch metadata_file.py for all the sites that have been updated6. Relaunch metadata_site.py 7. Relaunch audio_processing.py for all the sites that have been updated 1. For each folder or dropbox, search for an existing folder on bigdisk1 or bigdisk2. 2. If the folder is not found, this is a new site.<jupyter_code>bigdisk1 = '/home/nfarrugi/bigdisk1'
bigdisk2 = '/home/nfarrugi/bigdisk2'

dropboxfolder = '/home/nfarrugi/bigdisk2/new/dropbox/SilentCitiesData/'
newsites = []
bigdisk1sites = []
bigdisk2sites = []

for curdir in os.listdir(dropboxfolder):
    if os.path.isdir(os.path.join(bigdisk1,'silentcities',curdir)):
        print(f"{curdir} found on bigdisk1")
    elif os.path.isdir(os.path.join(bigdisk2,'silentcities',curdir)):
        print(f"{curdir} found on bigdisk2")
    else:
        newsites.append(curdir)
        print(f"{curdir} is a new site")<jupyter_output>0308 is a new site
0334 is a new site
0145 found on bigdisk1
0149 found on bigdisk1
0137 found on bigdisk1
0098 found on bigdisk1
0238 found on bigdisk1
0167 found on bigdisk1
0342 found on bigdisk2
0142 found on bigdisk1
0233 found on bigdisk2
0096 found on bigdisk1
0108 found on bigdisk1
0131 found on bigdisk1
0196 found on bigdisk1
0270 found on bigdisk1
0315 found on bigdisk2
0344 found on bigdisk1
0141 found on bigdisk1
0361 found on bigdisk1
0286 found on bigdisk2
0015 is a new site
0247 found on bigdisk1
0057 found on bigdisk1
0082 found on bigdisk1
0050 found on bigdisk2
0243 found on bigdisk2
0201 found on bigdisk2
0324 found on bigdisk1
0014 found on bigdisk1
0181 found on bigdisk1
0115 found on bigdisk1
0090 found on bigdisk1
0176 found on bigdisk1
0039 is a new site
0213 found on bigdisk1
0236 found on bigdisk1
0148 found on bigdisk1
0268 found on bigdisk1
0296 found on bigdisk1
0256 found on bigdisk1
0322 is a new site
0151 found on bigdisk1
0085 found on bigdisk1
0349 fou[...]<jupyter_text>3. Parse the list of new sites and put them on bigdisk2, extract them flat. Save the list of new sites as sites for which we will need to restart "metadata_file"<jupyter_code>def extract_list_from_zip(zipfile,destdir,filelist=None):
    ### filelist is a subset (potentially all) from the list of files in zipfile
    ### If not, take the whole list of files from the zip

    os.makedirs(destdir,exist_ok=True)

    z = ZipFile(zipfile)

    if filelist is None:
        filelist = z.namelist()
        print("Extracting all files...")
    
    print(f"Extracting {len(filelist)}files")

    for curfile in filelist:
        
        # In case in the filelist there are paths
        filenameonly = os.path.split(curfile)[1]

        # extract a specific file from the zip container
        f = z.open(curfile)

        # save the extracted file 
        content = f.read()
        f = open(os.path.join(destdir,filenameonly), 'wb')
        f.write(content)
        f.close()
def check_folder_zip_copy(zipfile,destfolder)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0663867751246392, 'fraction_numerical': 0.06796116504854369, 'mean_word_length': 3.9506493506493507, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '43844', 'license': 'MIT', 'n_tokens_mistral': 1358, 'n_tokens_neox': 1231, 'n_words': 490, 'path': 'AddDropboxFolders.ipynb', 'repo_name': 'brain-bzh/SilentCities', 'size': '3811'}