I have a bunch of compressed text files each line of which containing a JSON object. Simplified my workflow looks like this:string_json = sc.textFile('/folder/with/gzip/textfiles/')json_objects = string_json.map(make_a_json)DataRDD = json_objects.map(extract_data_from_json)DataDF = sqlContext.createDataFrame(DataRDD,schema).collect()'''followed by some transformations to the dataframe'''Now, the code works fine. The problem arises as soon as the number files can not be evenly divided between executors. That is as far as I understand it, because spark is not extracting the files and then distributing the rows to the executors, but rather each executioner gets one file to work with. e.g If i have 5 files and 4 executors, the first 4 files are processed in parallel and then the 5th file. Because the 5th is not being processed in parallel with the other 4 and cannot be divided between the 4 executors, it takes the same amount of time as the first 4 together. This happens at every stages of the program.Is there a way to transform this kind compartmentalized RDD either into a RDD or Dataframe that is not ?
RDD of gziped files to uncompressed Dataframe
apache spark;pyspark