I need to parse logs and have got following code. I can see two problems: map().filter() may induce some performance penalties and copy-paste blockparser.py:class Info(object):    a =     j =     z =     infoline =     def __init__(self, a, j, z, infoline):        self.a = a        self.j = j        self.z = z        self.infoline = infoline# function check if the line parameter is produced by provider 'prov'# yes - returns certain substring of line# no - returns Nonedef get_infoline(line, prov):   ...def process(line, prov):    retA = None    retJ = None    retZ = None    infoline = get_infoline(line, prov)line param from     if infoline:        # filling some of retA, retJ, retZ        ...        return Info(retA, retJ, retZ, infoline)job.py:from pyspark import SparkContextimport parser...prov = ...log = sc.textFile(pathTofile)parsed = log.map(lambda ln: parser.proccess(ln, prov)).filter(lambda i: i)summaryA = parsed.map(lambda info: (info.a, 1)).reduceByKey(add) \    .map(lambda (a,b): (b,a)).sortByKey(False) \    .map(lambda (count, name): (%s\t%i % (name, count))) \    .saveAsTextFile('/output/path/a.tsv')summaryJ = parsed.map(lambda info: (info.j, 1)).reduceByKey(add) \    .map(lambda (a,b): (b,a)).sortByKey(False) \    .map(lambda (count, name): (%s\t%i % (name, count))) \    .saveAsTextFile('/output/path/j.tsv')summaryZ = parsed.map(lambda info: (info.z, 1)).reduceByKey(add) \    .map(lambda (a,b): (b,a)).sortByKey(False) \    .map(lambda (count, name): (%s\t%i % (name, count))) \    .saveAsTextFile('/output/path/z.tsv')
Python + spark to parse and save logs
python;performance;logging;apache spark