I am working with very large datasets in a distributed environment. In particular, I am writing some code in Python for analyzing web logs in Spark. Because of the nature of Map-Reduce computing and the size of the logs I am working with, I would like to restrict the amount of data being passed around as much as possible.For any given analysis, I am likely to only be interested in a few fields out of the several dozen available, some of which are very large. I don't want to store the whole object in memory; I only want a pared-down version containing the fields I'm interested in.A simple data structure like a list would be possible, but it would be convenient to be able to refer to fields by name rather than keep track of what position they've been placed in the list. A dict would allow this, but with the heavy overhead of storing key names for each log line. This led me to namedtuple.So here's a simplified example of what I have in mind. I have four main concerns (but any other tips are welcome, as I am quite new to Python):Is using namedtuple here the best choice? (As opposed to, possibly, another object with only a few fields initialized.)Is this going to create a new namedtuple type for each line, thus creating a new list of field names each time and defeating the purpose of avoiding dict?The parsing method makes me long for an auto-increment syntax, so perhaps I'm missing a more pythonic way of doing itThis method does not seem to offer a clean way of including individual values from the properties dictionaryThe code:import astfrom collections import namedtuplefrom IPy import IPclass LogRecord:    def __init__(self, line):        self.parse(line)    def parse(self, line):        fields = line.split('\t')        i = 0        self.version = fields[i]        i += 1        self.date = fields[i]        i += 1        self.time = long(fields[i])        i += 1        self.ipAddress = IP(fields[i])        i += 1        self.url = fields[i]        i += 1        self.userAgentString = fields[i]        i += 1        self.properties = ast.literal_eval(fields[i])        i += 1        self.customerId = fields[i]    def select(self, *fields):        d = { key: getattr(self, key) for key in fields }        Tuple = namedtuple('Tuple', fields)        return Tuple(**d)l = LogRecord(8.53\t2014-12-13\t1420007099253\t127.0.0.1\thttp://www.google.com/abc/123\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36\t{'x': 11, 'y': 22}\tx97ag89x77d)print l.select('version', 'url', 'customerId')When run, it produces the following output:Tuple(version='8.53', url='http://www.google.com/abc/123', customerId='x97ag89x77d')
Using namedtuple for slice of class
python;parsing
Don't call long's constructor; int has pretty much no disadvantages and is more idiomatic. It's also Python 3 compatible and could use less space.Josay mentioned iterable unpacking; that's a great idea, although I would suggest wrapping the line:[self.version, self.date, time, ipAddress, self.url,    self.userAgentString, self.properties, self.customerId] = fieldsor even[    self.version,    self.date,    time,    ipAddress,    self.url,    self.userAgentString,    self.properties,    self.customerId] = fieldsI would actually avoid doing this at all by making LogRecord itself a namedtuple:_LogRecordFields = namedtuple(LogRecordFields, [    version, date, time, ipAddress, url,    userAgentString, properties, customerId])class LogRecord(_LogRecordFields):    def __new__(cls, line):        fields = line.split('\t')        fields[2] = int(fields[2])        fields[3] = IP(fields[3])        return super().__new__(cls, *fields)In select you are making a new class with its own source code each time. You can use a backport of functools.lru_cache to cache the namedtuples:_get_namedtuple_cached = lru_cache(maxsize=None)(namedtuple)Alternatively, roll your own.It also has a somewhat poor name (just Tuple, eh). I would just put the fields in there:name = '_'.join(field.title() for field in fields).replace(_, )Tuple = _get_namedtuple_cached(name, fields)Although I would understand any reasonable compromise.Finally, I would move to snake_case for the fields in true Python convention. This all gives:from collections import namedtuplefrom functools import lru_cache_get_namedtuple_cached = lru_cache(maxsize=None)(namedtuple)_LogRecordFields = namedtuple(LogRecordFields, [    version, date, time, ip_address, url,    user_agent_string, properties, customer_id])class LogRecord(_LogRecordFields):    def __new__(cls, line):        fields = line.split('\t')        fields[2] = int(fields[2])        fields[3] = IP(fields[3])        return super().__new__(cls, *fields)    def select(self, *fields):        selected = (getattr(self, field) for field in fields)        name = '_'.join(fields).title().replace(_, )        Tuple = _get_namedtuple_cached(name, fields)        return Tuple(*selected)l = LogRecord(8.53\t2014-12-13\t1420007099253\t127.0.0.1\thttp://www.google.com/abc/123\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36\t{'x': 11, 'y': 22}\tx97ag89x77d)print(l.select('version', 'url', 'customer_id'))#>>> VersionUrlCustomerId(version='8.53', url='http://www.google.com/abc/123', customer_id='x97ag89x77d')One last thing; if youfrom operator import itemgetterand remove the auto-generated __dict__ in LogRecord with__slots__ = ()you can doselected = itemgetter(*fields)(self._asdict())which avoids calling getattr.