Creed of Python Developers

Pythonistas are eager to extol the lovely virtues of our language. Most beginning Python programmers are invited to run import this from the interpreter right after the canonical hello world. One of the favorite quips from running that command is:

There should be one-- and preferably only one --obvious way to do it.

But the path to Python enlightenment is often covered in rocky terrain, or thorns hidden under leaves.

A Dilemma

On that note, I recently had to use some code that parsed a file. A problem arose when the API had been optimized around the assumption that what I wanted to parse would be found in the filesystem of a POSIX compliant system. The implementation was a staticmethod on a class that was called from_filepath. Well in 2013, we tend to ignore files and shove those lightweight chisels of the 70s behind in favor of a shiny new super-powered jack-hammers called NoSQL.

It so happened that I found myself with a string (pulled out of a NoSQL database) containing the contents of a file I wanted to parse. There was no file, no filename, only the data. But my API only supported access through the filename.

Perhaps the pragmatic solution would be to simply throw the contents into a temporary file and be done with it:

	import tempfile
	
	data = get_string_data()  # fancy call out to NoSQL
	with tempfile.NamedTemporaryFile() as fp:
	    fp.write(data)
	    fp.seek(0)
	    obj = Foo.from_filepath(fp.name)
	

But I spent a bit of time thinking about the root of the problem and wanted to see how others solved it. Having a parsing interface that just supports parsing a string is probably a premature optimization on the other end of the spectrum.

A Little Light Reading

My first thought was to look to the source of all truth—The Python Standard Library. Surely it would enlighten me by illuminating all 19 tenets of “The Zen of Python”. I asked myself what modules I used to parse with the standard library and came up with the following list:

• json
• pickle
• xml.etree.ElementTree
• xml.dom.minidom
• ConfigParser
• csv

(Note: all of the above are module names. The nested namespace of xml.* violates Zen tenet #5 “Flat is better than nested”, and ConfigParser violates PEP 8 naming conventions. This is not news to long time Python programmers, but to newbies here it is a dose of reality. The standard library is not perfect and has its quirks. Even in Python 3. And this is only the tip of the iceberg.)

A Clear Picture

I went through the documentation and source code for these modules to determine the single best, most Pythonic, beautiful, explicit, simple, readable, practical, non-ambiguous, and easy to explain solution to parsing. Specifically, should I parse a filename, a file-like object, or a string? Here is the resulting table I came up with:

Module	String Data	File	Filename
json	loads	load
pickle	loads	load
xml.etree.ElementTree	fromstring	parse	parse
xml.dom.minidom	parseString	parse	parse
ConfigParser		cp.readfp	cp.read(filenames)
csv		reader DictReader
pyyaml	load, safe_load	load, safe_load

(Note: pyyaml is a 3rd party library, but there has been much hubbub going around recently on the naming of load, which is unsafe (but probably the method most will use unless they really pour through the docs), and safe_load, which is safe (and hidden away in the docs).)

The trick to this table is to spin around three times, really squint your eyes, and pick something from the File column.

The Cover Your Bases Answer

The simplest solution to parsing is to parse a file-like object. Again for the newbies, I say file-like because Python has duck typing. According to Wikipedia this term, duck typing, comes from the saying of an old poet—James Whitcomb Riley:

When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.

But I prefer the scene in Monty Python and the Holy Grail, when wise Bedemir uses his powers of logic to determine whether witches are made of wood and therefore flammable—fit to be burned:

	BEDEMIR:  So, how do we tell whether she is made of wood?
	VILLAGER #1:  Build a bridge out of her.
	...
	BEDEMIR:  Aah, but can you not also build bridges out of stone?
	VILLAGER #2:  Oh, yeah.
	BEDEMIR:  Does wood sink in water?
	VILLAGER #1:  No, no.
	VILLAGER #2:  It floats!  It floats!
	BEDEMIR:  What also floats in water?
	VILLAGER #1:  Bread!
	VILLAGER #2:  Apples!
	VILLAGER #3:  Very small rocks!
	VILLAGER #1:  Cider!
	VILLAGER #2:  Great gravy!
	VILLAGER #1:  Cherries!
	VILLAGER #2:  Mud!
	VILLAGER #3:  Churches -- churches!
	VILLAGER #2:  Lead -- lead!
	ARTHUR:  A duck.
	CROWD:  Oooh.
	BEDEMIR:  Exactly!  So, logically...,
	VILLAGER #1:  If... she.. weighs the same as a duck, she's made of wood.
	BEDEMIR:  And therefore--?
	VILLAGER #1:  A witch!
	

(Courtesy of sacred-texts.com/neu/mphg/mphg.htm)

We can apply the same principle—duck typing—to file objects. If an object behaves like a file—has a read method—then it is a file. By having this interface we can easily deal with file-like objects, but also string data, and filenames as will be illustrated.

Implementing the Interface

Another interesting note from the table above is that most of the parsing mechanisms are not tied to a class. Most are functions that create an object, populate it from data within the file and return the object (ConfigParser.readfp being the exception as it has no function, though the ElementTree module has both a parse function and a parse method on the ElementTree class). In addition isolating parsing from the data adheres to the Single Responsibility Principle.

The author’s opinion is that an overloaded function or method (such as ElementTree.parse which accepts both files and filenames, or pyyaml.load which accepts both strings or files) is a violation of “Explicit is better than implicit” and “In the face of ambiguity, refuse the temptation to guess”. These methods are not taking advantage of duck typing, but magic typing, going beyond Postel’s Law. The Python language makes a hard distinction about the difference between an Iterable and an Iterator. A similar comparison could be made between a filename, and a file object. One is a duck and the other is a duck factory.

A Simple Example

The example below parses the contents of the Unix /etc/passwd file and stores the result in a list within the Passwd object:

	from collections import namedtuple
	
	User = namedtuple('User', 'name, pw, id, gid, groups, home, shell')
	
	class Passwd(object):
	    def __init__(self):
	        self.users = []
	        
	        
	def parse(fp):
		pw = Passwd()
	    for line in fp:
	        data = line.strip().split(':')
	        pw.users.append(User(*data))
	    return pw
	

A couple of notes regarding the implementation:

• A parse function populates a Passwd object
• Passwd object has one responsibility, knowing about the users. It is not responsible for parsing

If a user wanted to create a Passwd object from a filename—/etc/passwd—this is easy:

	# from filename
	with open('/etc/passwd') as fin:
	    pw = parse(fin)
	

Neither the parse function nor the Passwd object needs to worry about opening or closing the file.

If the data from the file was already in memory, utilizing the built-in module StringIO enables parsing from that end:

	# from string
	from StringIO import StringIO as sio
	fin = sio('''pulse:x:113:121:PulseAudio daemon,,,:/var/run/pulse:/bin/false\nsaned:x:114:123::/home/saned:/bin/false''')
	pw = parse(fin)
	

Advantages of File-interfaces

Hopefully this long-winded post has provided you some insight into parsing files in Python. Prior art is found in the standard library and 3rd party modules. If you want to be fancy and all accommodating, you can, but in reality you are probably just adding unnecessary complexity.

To review a parsing method that relies only on a file interface is:

• Scalable - accepts generators, iterators that are potentially very long, in addition to files, sockets, and StringIO
• Easier to manage - you don’t want to be in the business of handling and closing people’s files
• Usable for users with a filename
• Usable for users with data from a file

Go forth and parse!