I'm writing a script that deals reads UTF-8-encoded XML files and writes parts of those files into a tempfile for further processing.
Sometimes, the input files will have a few malformed characters. If it's just "a few" occurrences, I want those glossed over, but if the file is majorly corrupted, I want to see an error message. I have a solution for this working right now, but I have this nagging feeling that this isn't a clean way to do this.
This creates an example file with malformed data:
f = open("test.xml","w+b") f.write("\x3c\x61\x3e\x31\x3c\x2f\x61\x3e\x0a\x3c\x62\x3e\x32\xc3\x0a\xbc\x3c\x2f\x62\x3e\x0a") f.close()
Yes, I know it's probably a stray linebreak that's causing my "malformed utf-8" problem, but let's not make that the issue here.
Now, the following is a simplified version I have in place to deal with malformed utf-8 data up to a certain limit:
import codecs sourcefile = open("test.xml", "r") # First line always XML declaration, no special treatment line = sourcefile.readline() i = 1 unicodeerror = 0 while line: try: pos = sourcefile.tell() line = sourcefile.readline().decode("utf-8") except UnicodeDecodeError as e: print "WARNING: Encoding error in line " + str(i+1) sourcefile.seek(pos, 0) line = sourcefile.readline().decode("utf-8","ignore") unicodeerror += 1 i += 1 if unicodeerror > 5: raise StandardError('ERROR: Too many Unicode errors')
As you can see, I read the input file as a byte string, and then read each line individually, doing the UTF-8-Decode within the try-except-clause. That way, I can catch the
UnicodeDecodeError and count it. Only after counting it, reread the line, this time ignoring possible mistakes.
However, it "feels wrong" to simply read the file like this instead of doing a
codecs.open(FILE, MODE, "utf-8"). With the codecs method, however, I see no way to reread the line with a different error handling directive after I hit the
Am I being too cautious? Is this maybe the best / only reliable way to do this? OR is there a much easier solution (which can be achieved within the standard library!)?