2
\$\begingroup\$

Here is how I'm detecting a field separator and line break in a CSV file for which I do not already know the format. Does this look sufficient, or are there other things I should be adding here?

SEPARATORS=['\x00', '\x01',  '^', ':', ',', '\t', ':', ';', '|', '~', ' ']
LINE_TERMINATORS_IN_ORDER = ['\x02\n', '\r\n', '\n', '\r']

def get_csv_info(f):
    with open(f, 'r') as csvfile:
        line = next(csvfile)
        dialect = csv.Sniffer().sniff(line, SEPARATORS)
        for _terminator in LINE_TERMINATORS_IN_ORDER:
            if line.endswith(_terminator):
                terminator = _terminator
                break
        return (dialect.delimiter, terminator)

get_csv_info('/Users/david/Desktop/validate_headers/artist')
('\x01', '\x02\n')

Note: using the normal csv.Sniffer without params fails most of the time I use it.

\$\endgroup\$
  • \$\begingroup\$ SEPARATORS=['\x00', ..., ':', ..., ':',...] - why two :? \$\endgroup\$ – Patrick Artner Dec 18 '18 at 6:48
  • \$\begingroup\$ There is only one case where you need a single byte, which is the line ending with \r. If it doesn't end with it then you need two bytes, no need to go through a list. \$\endgroup\$ – ChatterOne Dec 18 '18 at 8:43
  • \$\begingroup\$ @ChatterOne could you please clarify what you mean with that? For example, what should the LINE_TERMINATORS be instead? \$\endgroup\$ – David542 Dec 18 '18 at 16:18

Browse other questions tagged or ask your own question.