4
\$\begingroup\$

1. Summary

I can't find, how I can to refactor multiple with open for one file.


2. Expected behavior of program

Program detect encoding for each file in the directory. If encoding ≠ UTF-8, file convert to UTF-8.


3. Minimal example of working code

(I'm sorry, Repl.it and another online Python interpreters incorrect works with non-UTF-8 files. But just in case, I created an online demonstration.)

  • kira_encoding.py
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
# @Last Modified time: 2018-09-01 14:39:30
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')


def kira_encoding_function():
    """Check encoding and convert to UTF-8, if encoding no UTF-8."""
    for filename in ALL_FILES:

        # Not 100% accuracy:
        # https://stackoverflow.com/a/436299/5951529
        # Check:
        # https://chardet.readthedocs.io/en/latest/usage.html#example-using-the-detect-function
        # https://stackoverflow.com/a/37531241/5951529
        with open(filename, 'rb') as opened_file:
            bytes_file = opened_file.read()
            chardet_data = chardet.detect(bytes_file)
            fileencoding = (chardet_data['encoding'])
            print('fileencoding', fileencoding)

            if fileencoding in ['utf-8', 'ascii']:
                print(filename + ' in UTF-8 encoding')
            else:
                # Convert file to UTF-8:
                # https://stackoverflow.com/a/191403/5951529
                with codecs.open(filename, 'r') as file_for_conversion:
                    read_file_for_conversion = file_for_conversion.read()
                with codecs.open(filename, 'w', 'utf-8') as converted_file:
                    converted_file.write(read_file_for_conversion)
                print(filename +
                      ' in ' +
                      fileencoding +
                      ' encoding automatically converted to UTF-8 ')


kira_encoding_function()
  • Kira1.md in UTF-8:
Kira Goddess!
  • Kira2.md in Cyrillic-1251:
Кира Богиня!
  • Kira3.md in Central European Cyrillic 1250:
Kiara Istennő!

4. Problem

I use with 3 times for opening same file. I don't think, that is a good practice.

Radon Cyclomatic Complexity not A:

D:\SashaDebugging\KiraEncoding>radon cc kira_encoding.py
kira_encoding.py
    F 13:0 kira_encoding_function - B

I can not use with, but it anti-pattern.


5. Not helped

5.1. Modes

  • That get encoding via chardet I need rb — bytes mode;
  • That convert file via codecs I need non-bytes modes.

I can't find, what can I do, that to have same mode for these actions.

5.2. decode

I can remove 1 with, if I know file encoding.

cyrillic_file = bytes_file.decode('cp1251')
with codecs.open(filename, 'w', 'utf-8') as converted_file:
    converted_file.write(cyrillic_file)
  • Full file:
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
# @Last Modified time: 2018-09-01 16:26:57
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')

def kira_encoding_function():
    """Check encoding and convert to UTF-8, if encoding no UTF-8."""
    for filename in ALL_FILES:

        # Not 100% accuracy:
        # https://stackoverflow.com/a/436299/5951529
        # Check:
        # https://chardet.readthedocs.io/en/latest/usage.html#example-using-the-detect-function
        # https://stackoverflow.com/a/37531241/5951529
        with open(filename, 'rb') as opened_file:
            bytes_file = opened_file.read()
            chardet_data = chardet.detect(bytes_file)
            fileencoding = (chardet_data['encoding'])
            print('fileencoding', fileencoding)

            if fileencoding in ['utf-8', 'ascii']:
                print(filename + ' in UTF-8 encoding')
            else:
                # Convert file to UTF-8:
                # https://stackoverflow.com/q/19932116/5951529
                cyrillic_file = bytes_file.decode('cp1251')
                with codecs.open(filename, 'w', 'utf-8') as converted_file:
                    converted_file.write(cyrillic_file)
                print(filename +
                      ' in ' +
                      fileencoding +
                      ' encoding automatically converted to UTF-8')


kira_encoding_function()

But the files may not necessarily be in Cyrillic-1251, they can be in any encoding. I can't find, How can I decode from any encoding. For example, this can't work:

>>> kiragoddess = b'\xca\xe8\xf0\xe0 \xc1\xee\xe3\xe8\xed\xff!'
>>> kiragoddess.decode('cp1251')
'Кира Богиня!'
>>> kiragoddess.decode()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte

5.3. Nested syntax

Nested syntax doesn't work for me. If:

with codecs.open(filename, 'r') as file_for_conversion, codecs.open(filename, 'w', 'utf-8') as converted_file:
    read_file_for_conversion = file_for_conversion.read()
    converted_file.write(read_file_for_conversion)
  • Full file:
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
# @Last Modified time: 2018-09-01 16:01:29
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')


def kira_encoding_function():
    """Check encoding and convert to UTF-8, if encoding no UTF-8."""
    for filename in ALL_FILES:

        # Not 100% accuracy:
        # https://stackoverflow.com/a/436299/5951529
        # Check:
        # https://chardet.readthedocs.io/en/latest/usage.html#example-using-the-detect-function
        # https://stackoverflow.com/a/37531241/5951529
        with open(filename, 'rb') as opened_file:
            bytes_file = opened_file.read()
            chardet_data = chardet.detect(bytes_file)
            fileencoding = (chardet_data['encoding'])
            print('fileencoding', fileencoding)

            if fileencoding in ['utf-8', 'ascii']:
                print(filename + ' in UTF-8 encoding')
            else:
                # Convert file to UTF-8:
                # https://stackoverflow.com/a/191403/5951529
                with codecs.open(filename, 'r') as file_for_conversion, codecs.open(filename, 'w', 'utf-8') as converted_file:
                    read_file_for_conversion = file_for_conversion.read()
                    converted_file.write(read_file_for_conversion)
                print(filename +
                      ' in ' +
                      fileencoding +
                      ' encoding automatically converted to UTF-8')


kira_encoding_function()

Content of non-UTF-8 files will be removed in this case.

6. Do not offer

  1. Yes, I know that I need to use logging, not print in real programs. Please, do not offer it; my question not about it.
  2. I need in-place conversion; my program must convert to UTF-8 to the same file, not to another.
\$\endgroup\$
  • \$\begingroup\$ If you know not to use print, why didn't you replace it with logging already or drop it? \$\endgroup\$ – 409_Conflict Sep 1 '18 at 16:01
  • \$\begingroup\$ @MathiasEttinger, because I endeavored to create a minimal example and remove code, which does not belong to my problem from item 4 of this question. Thanks. \$\endgroup\$ – Саша Черных Sep 1 '18 at 16:06
  • 1
    \$\begingroup\$ And in your decode approach, why did you hardcode cp1251 instead of reusing what is stored in the fileencoding variable? \$\endgroup\$ – 409_Conflict Sep 1 '18 at 16:40
  • \$\begingroup\$ @MathiasEttinger, this is my stupidity and lack of experience. Rewritten example. Thanks. \$\endgroup\$ – Саша Черных Sep 1 '18 at 18:37
4
\$\begingroup\$
  1. The code in the post uses the chardet library to determine the encoding of the file, but then the only use it makes of that information is to decide whether or not to try transcoding the file. The detected encoding should also be used to decode the content, using bytes.decode:

    with open(filename, 'rb') as f:
        content_bytes = f.read()
    detected = chardet.detect(content_bytes)
    encoding = detected['encoding']
    print(f"{filename}: detected as {encoding}.")
    content_text = content_bytes.decode(encoding)
    with open(filename, 'w', 'utf-8') as f:
        f.write(content_text)
    

    Notice that in this version the file only needs to be read once.

  2. There ought to be some check against the confidence value returned by chardet, to reduce the risk of making nonsense out of files where the encoding is detected wrongly. For example, you could have a global constant:

    CONFIDENCE_THRESHOLD = 0.8 # minimum confidence to transcode the file
    

    and then skip files where the detected confidence is below the threshold:

    confidence = detected['confidence']
    print(f"{filename}: detected as {encoding} with confidence {confidence}.")
    if confidence < CONFIDENCE_THRESHOLD:
        print(f"{filename}: skipped.")
        continue
    
  3. Editing a file by reading it and then writing back to the same file is an anti-pattern.

    with codecs.open(filename, 'r') as file_for_conversion:
        read_file_for_conversion = file_for_conversion.read()
    with codecs.open(filename, 'w', 'utf-8') as converted_file:
        # what if there is an exception here?
        converted_file.write(read_file_for_conversion)
    

    The problem with this approach is that there might be an exception after the file is opened for writing and before it is closed. For example, the user might type control-C. If this happens, then the content of the file might be lost.

    To make the code robust against this risk, you need to write the output to a temporary file, and then rename the temporary file using os.replace. The Python documentation guarantees that

    the renaming will be an atomic operation

    where "atomic" means that it either succeeds completely (meaning that the file is converted) or fails completely (leaving the original file unchanged), with no other possibilities. So what you need is something like this, using tempfile.NamedTemporaryFile to create the temporary file:

    with open(filename, 'rb') as f:
        content_bytes = f.read()
    detected = chardet.detect(content_bytes)
    encoding = detected['encoding']
    confidence = detected['confidence']
    print(f"{filename}: detected as {encoding} with confidence {confidence}.")
    if confidence < CONFIDENCE_THRESHOLD:
        print(f"{filename} skipped.")
        continue
    content_text = content_bytes.decode(encoding)
    with tempfile.NamedTemporaryFile(mode='w', dir=os.path.dirname(filename),
                                     encoding='utf-8', delete=False) as f:
        f.write(content_text)
    os.replace(f.name, filename)
    
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.