6
\$\begingroup\$

Below there is a function eating all strings and convert it to Unicode. It works fine. But is it the best way to do so? Is there maybe already an existing function for it?

The reason was a German Windows version which prints German ä ö ü - error-messages which are retrieved from popen/communicate etc. There must have been someone before me with the same issue.

# python2.7
def all_eating_unicode_converter(input_string):
    test1 = 'äöü'
    test2 = b'\xe4\xf6\xfc'
    test3 = u'äöü'
    """
    Converts every input to unicode. tested with testcases abouve!
    :param input_string: may be a string or unicode or bytes.
    :return: returns unicode
    """

    l.debug("type: %s\n message:%s" % (type(input_string), input_string))
    if type(input_string) is not unicode:
        # i don't need to decode unicode. because it already is!
        output_string = input_string.decode("utf-8")  # converts bytes to unicode
    else:
        output_string = input_string
    return output_string
\$\endgroup\$
  • 2
    \$\begingroup\$ While you say it works for the three test strings, it actually raises an exception for test2: b'\xe4\xf6\xfc'.decode("utf-8") -> UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 0: invalid continuation byte. \$\endgroup\$ – Graipher Sep 30 '19 at 13:00
  • 1
    \$\begingroup\$ Although that is just because that is not a valid UTF-8 string. Maybe use b'\xc3\xa4\xc3\xb6\xc3\xbc' instead. \$\endgroup\$ – Graipher Sep 30 '19 at 13:14
7
\$\begingroup\$

First, you should not be using Python 2 anymore, if at all possible. It will reach end of support at the end of the year.

In Python 3 your code would just be:

def all_eating_unicode_converter(input_string):
    """
    Converts every input to unicode.
    :param input_string: may be a string or unicode or bytes.
    :return: returns unicode

    Tested with: 'äöü', b'\xc3\xa4\xc3\xb6\xc3\xbc', u'äöü'
    """
    if isinstance(input_string, bytes):
        return input_string.decode("utf-8")
    return input_string

Note that I used isinstance instead of type, to also allow derived types of bytes to work properly.

I also shortened the logic by not insisting on having a single return from the function. Here it is clear enough what is happening, and I am in general a fan of early returns.

Docstrings only work when they are actually right after the function header, so I moved the testcases inside the docstring.

In addition, one reason the logging module is so powerful is that it allows you to write log messages which are costly to print and never print them. In other words the replacement of the placeholders is only performed if the message is actually being printed, i.e. if the log level is high enough. So you should always do this:

l.debug("type: %s\n message:%s",  type(input_string), input_string)
\$\endgroup\$
  • 1
    \$\begingroup\$ thank you for your effort and detailed explanation and comments to the performance. I've learnde a lot and one time I'll be a great python programmer =). \$\endgroup\$ – Cutton Eye Sep 30 '19 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.