<jupyter_start><jupyter_text>Converting .tar.gz Files to AVROThe files in the Google Open Image dataset are compressed into large .tar.gz archive files. Unfortunately, this file format does not fit into the Hadoop ecosystem well. I even tried to just extract and upload the individual files but I ran into a maximum number of files in a directory limit of HDFS (above 2 million something files).This document converts the .tar.gz files into an AVRO file containing the file name and the image data.  AVRO SchemaRead more about AVRO [AVRO Docs](https://avro.apache.org/docs/current/spec.htmlschemas).<jupyter_code>record_schema = """
{
     "type": "record",
     "namespace": "google.open.image",
     "name": "image",
     "fields": [
       { "name": "id", "type": "string" },
       { "name": "subset", "type": {
               "type": "enum",
               "name": "subsets",
               "symbols": ["TEST", "TRAIN", "VALIDATION"]
           }
       },
       { "name": "format", "type": {
         "type": "enum",
               "name": "image_formats",
               "symbols": ["JPG", "PNG"]
       }
       },
       { "name": "data",
         "type": "bytes"}
     ]
}
"""<jupyter_output><empty_output><jupyter_text>Write a Dummy Document to Test AVRO Lib<jupyter_code># note, for python 3, pip install avro-python3
#
import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter
import os
schema = avro.schema.parse(record_schema)

writer = DataFileWriter(open("test.avro", "wb"), DatumWriter(), schema)

# Create some dummy records. os.urandom just creates some random bytes
writer.append({"id": "first",  "subset": "TEST",       "format": "JPG", "data": os.urandom(10)})
writer.append({"id": "second", "subset": "VALIDATION", "format": "PNG", "data": os.urandom(15)})
writer.append({"id": "third",  "subset": "TRAIN",      "format": "JPG", "data": os.urandom(3)})

writer.close()<jupyter_output><empty_output><jupyter_text>if you dump the raw bytes of that file in the terminal, you'll see that the enum fields are replaced with integers, so we're not re-writing "TEST" or "JPG" for each record.  Now convert entry from .tar.gz to AVRORead .tar.gz files from HDFS<jupyter_code>import pyarrow as pa
hdfs = pa.hdfs.HadoopFileSystem('10.3.0.2', 8020)
hdfs.connect()<jupyter_output><empty_output><jupyter_text>Find data in HDFSThe data we're interested in is in /data/google_open_image/<jupyter_code># get list of files
image_dir_list = hdfs.ls('/data/google_open_image/images')
# filter list of anything not a .tar.gz file
image_tar_gz_files = [file for file in image_dir_list if ".tar.gz" in file]
image_tar_gz_files<jupyter_output><empty_output><jupyter_text>Reading files from a tarfile<jupyter_code>import tarfile

# grab a random image from the first tar archive.
tarball = image_tar_gz_files[0]

print(f"opening tarball in HDFS: {tarball}")
tf_handle = hdfs.open(tarball)

# pass hdfs handle to tarfile library
tar = tarfile.open(fileobj = tf_handle)
member = tar.next() # read first object / file from archive

file_counter = 0
while member is not None and file_counter < 814:
    # tar archives generally contain paths / directories, so be sure to only grab things that are actually files
    if not member.isfile():
        member = tar.next()
        continue
    f = tar.extractfile(member)
    content = f.read()
    f.close()
    member = tar.next()
    file_counter = file_counter + 1

tar.close() # be a good citizen and close files.

member.path
from PIL import Image
import io

jpg = Image.open(io.BytesIO(content))
jpg<jupyter_output><empty_output><jupyter_text>Create an AVRO record with the real data  Extract Subset (Test, Validation, Train) From Path<jupyter_code>member.path
directory = os.path.dirname(member.path)
image_subset = directory.upper()
image_subset<jupyter_output><empty_output><jupyter_text>Extract Image ID from Filename<jupyter_code>filename = os.path.basename(member.path)
filename
filename_parts = os.path.splitext(filename)
filename_parts
image_id = filename_parts[0]
image_id<jupyter_output><empty_output><jupyter_text>Get File Type from File name<jupyter_code>image_format = filename_parts[1].replace(".", "").upper()
image_format<jupyter_output><empty_output><jupyter_text>Create Dict With AVRO Schema Naming<jupyter_code>record = {"id": image_id,  "subset": image_subset, "format": image_format, "data": content}<jupyter_output><empty_output><jupyter_text>Write the record to a test file in HDFS<jupyter_code>avro_outfile = hdfs.open(path = f'/user/{os.environ["USER"]}/test2.avro', 
          mode = 'wb')

writer = DataFileWriter(avro_outfile, DatumWriter(), schema)

writer.append(record)
writer.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12384356602186712, 'fraction_numerical': 0.0054667788057190915, 'mean_word_length': 4.499421965317919, 'pattern_counts': {'":': 34, '<': 42, '<?xml version=': 0, '>': 41, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '58934', 'license': 'CNRI-Python', 'n_tokens_mistral': 1586, 'n_tokens_neox': 1549, 'n_words': 448, 'path': 'data603/ImageTarballtoAVRO.ipynb', 'repo_name': 'sanaamironov/Data-603---Platforms-for-Big-Data-Processing', 'size': '4756'}