4
\$\begingroup\$

I am working with very large datasets in a distributed environment. In particular, I am writing some code in Python for analyzing web logs in Spark. Because of the nature of Map-Reduce computing and the size of the logs I am working with, I would like to restrict the amount of data being passed around as much as possible.

For any given analysis, I am likely to only be interested in a few fields out of the several dozen available, some of which are very large. I don't want to store the whole object in memory; I only want a pared-down version containing the fields I'm interested in.

A simple data structure like a list would be possible, but it would be convenient to be able to refer to fields by name rather than keep track of what position they've been placed in the list. A dict would allow this, but with the heavy overhead of storing key names for each log line. This led me to namedtuple.

So here's a simplified example of what I have in mind. I have four main concerns (but any other tips are welcome, as I am quite new to Python):

  • Is using namedtuple here the best choice? (As opposed to, possibly, another object with only a few fields initialized.)
  • Is this going to create a new namedtuple type for each line, thus creating a new list of field names each time and defeating the purpose of avoiding dict?
  • The parsing method makes me long for an auto-increment syntax, so perhaps I'm missing a more pythonic way of doing it
  • This method does not seem to offer a clean way of including individual values from the properties dictionary

The code:

import ast
from collections import namedtuple
from IPy import IP

class LogRecord:
    def __init__(self, line):
        self.parse(line)

    def parse(self, line):
        fields = line.split('\t')
        i = 0
        self.version = fields[i]

        i += 1
        self.date = fields[i]

        i += 1
        self.time = long(fields[i])

        i += 1
        self.ipAddress = IP(fields[i])

        i += 1
        self.url = fields[i]

        i += 1
        self.userAgentString = fields[i]

        i += 1
        self.properties = ast.literal_eval(fields[i])

        i += 1
        self.customerId = fields[i]

    def select(self, *fields):
        d = { key: getattr(self, key) for key in fields }
        Tuple = namedtuple('Tuple', fields)
        return Tuple(**d)

l = LogRecord("8.53\t2014-12-13\t1420007099253\t127.0.0.1\thttp://www.google.com/abc/123\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36\t{'x': 11, 'y': 22}\tx97ag89x77d")

print l.select('version', 'url', 'customerId')

When run, it produces the following output:

Tuple(version='8.53', url='http://www.google.com/abc/123', customerId='x97ag89x77d')
\$\endgroup\$
3
\$\begingroup\$

Don't call long's constructor; int has pretty much no disadvantages and is more idiomatic. It's also Python 3 compatible and could use less space.

Josay mentioned iterable unpacking; that's a great idea, although I would suggest wrapping the line:

[self.version, self.date, time, ipAddress, self.url,
    self.userAgentString, self.properties, self.customerId] = fields

or even

[
    self.version,
    self.date,
    time,
    ipAddress,
    self.url,
    self.userAgentString,
    self.properties,
    self.customerId
] = fields

I would actually avoid doing this at all by making LogRecord itself a namedtuple:

_LogRecordFields = namedtuple("LogRecordFields", [
    "version", "date", "time", "ipAddress", "url",
    "userAgentString", "properties", "customerId"
])

class LogRecord(_LogRecordFields):
    def __new__(cls, line):
        fields = line.split('\t')
        fields[2] = int(fields[2])
        fields[3] = IP(fields[3])
        return super().__new__(cls, *fields)

In select you are making a new class with its own source code each time. You can use a backport of functools.lru_cache to cache the namedtuples:

_get_namedtuple_cached = lru_cache(maxsize=None)(namedtuple)

Alternatively, roll your own.

It also has a somewhat poor name (just Tuple, eh). I would just put the fields in there:

name = '_'.join(field.title() for field in fields).replace("_", "")
Tuple = _get_namedtuple_cached(name, fields)

Although I would understand any reasonable compromise.

Finally, I would move to snake_case for the fields in true Python convention. This all gives:

from collections import namedtuple
from functools import lru_cache

_get_namedtuple_cached = lru_cache(maxsize=None)(namedtuple)

_LogRecordFields = namedtuple("LogRecordFields", [
    "version", "date", "time", "ip_address", "url",
    "user_agent_string", "properties", "customer_id"
])

class LogRecord(_LogRecordFields):
    def __new__(cls, line):
        fields = line.split('\t')
        fields[2] = int(fields[2])
        fields[3] = IP(fields[3])
        return super().__new__(cls, *fields)

    def select(self, *fields):
        selected = (getattr(self, field) for field in fields)

        name = '_'.join(fields).title().replace("_", "")
        Tuple = _get_namedtuple_cached(name, fields)
        return Tuple(*selected)

l = LogRecord("8.53\t2014-12-13\t1420007099253\t127.0.0.1\thttp://www.google.com/abc/123\tMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36\t{'x': 11, 'y': 22}\tx97ag89x77d")

print(l.select('version', 'url', 'customer_id'))
#>>> VersionUrlCustomerId(version='8.53', url='http://www.google.com/abc/123', customer_id='x97ag89x77d')

One last thing; if you

from operator import itemgetter

and remove the auto-generated __dict__ in LogRecord with

__slots__ = ()

you can do

selected = itemgetter(*fields)(self._asdict())

which avoids calling getattr.

\$\endgroup\$
  • \$\begingroup\$ Thanks for the ideas. I'm using long because the values exceed 2^31-1. I was using namedtuple as a parent class originally (though I didn't notice the great benefit in parsing) but I was having difficulties with inheritance because I also want to have a hierarchy of log record types: LogRecord and AugmentedLogRecord, which has all the same fields with the same meaning, plus some additional ones. The multiple inheritance was giving me fits with new and init. But maybe it's worth another shot and I'll post another question if I hack something together. \$\endgroup\$ – reo katoa Dec 31 '14 at 17:54
  • \$\begingroup\$ And thanks to your comment, I learned that in Python, ints can actually be up to 64 bits (i.e., the size of long in C on my system). Great! \$\endgroup\$ – reo katoa Dec 31 '14 at 18:30
  • 1
    \$\begingroup\$ @reokatoa It doesn't matter how big int can be since it'll automatically promote to long. \$\endgroup\$ – Veedrac Dec 31 '14 at 18:33
  • \$\begingroup\$ Thanks. One more question -- why do you want to avoid calls to getattr? (Would you also want to avoid calls to setattr?) \$\endgroup\$ – reo katoa Dec 31 '14 at 19:04
  • \$\begingroup\$ getattr and setattr invoke dynamism that is best avoided in static scenarios. One of the disadvantages is that getattr will succeed for bogus input like _asdict/__dict__/_make/_source/etc. Going through a well-definied interface (a dictionary) significantly restricts the leakiness of the abstraction. Like eval, there are uses but if you find those uses crop up more than a couple of times a year you should rethink how dynamic your code is. \$\endgroup\$ – Veedrac Dec 31 '14 at 19:30
4
\$\begingroup\$

Just a few ideas

You could use tuple unpacking to make parse more concise.

class LogRecord:
    def __init__(self, line):
        self.parse(line)

    def parse(self, line):
        fields = line.split('\t')
        self.version, self.date, time, self.ipAddress, self.url, self.userAgentString, properties, self.customerId = fields
        self.time = long(time)
        self.properties = ast.literal_eval(properties)

    def select(self, *fields):
        d = { key: getattr(self, key) for key in fields }
        Tuple = namedtuple('Tuple', fields)
        return Tuple(**d)

Alternatively, you could define a list of (name, function to apply) and iterate through both lists in the same with zip to fill a dictionary.

class LogRecord:
    def __init__(self, line):
        self.parse(line)

    def parse(self, line):
        identity = lambda x: x
        desc = [('version', identity), ('date', identity), ('time', long), ('ipAddress', identity), ('url', identity), ('userAgentString', identity), ('properties', ast.literal_eval), ('customerId', identity)]

        fields = line.split('\t')
        assert len(desc) == len(fields)
        self.values = {name: func(field) for (name, func), field in zip(desc, fields)}

    def select(self, *fields):
        d = { key: self.values[key] for key in fields }
        Tuple = namedtuple('Tuple', fields)
        return Tuple(**d)
\$\endgroup\$
2
\$\begingroup\$

Is this going to create a new namedtuple type for each line, thus creating a new list of field names each time and defeating the purpose of avoiding dict?

Yes. Assuming you want to extract the same fields from many records, arrange your code so that you create the named tuple type just once.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.