# Convert to UTF-8 all files in a directory

## 1. Summary

I can't find, how I can to refactor multiple with open for one file.

## 2. Expected behavior of program

Program detect encoding for each file in the directory. If encoding ≠ UTF-8, file convert to UTF-8.

## 3. Minimal example of working code

(I'm sorry, Repl.it and another online Python interpreters incorrect works with non-UTF-8 files. But just in case, I created an online demonstration.)

• kira_encoding.py
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')

def kira_encoding_function():
"""Check encoding and convert to UTF-8, if encoding no UTF-8."""
for filename in ALL_FILES:

# Not 100% accuracy:
# https://stackoverflow.com/a/436299/5951529
# Check:
# https://stackoverflow.com/a/37531241/5951529
with open(filename, 'rb') as opened_file:
chardet_data = chardet.detect(bytes_file)
fileencoding = (chardet_data['encoding'])
print('fileencoding', fileencoding)

if fileencoding in ['utf-8', 'ascii']:
print(filename + ' in UTF-8 encoding')
else:
# Convert file to UTF-8:
# https://stackoverflow.com/a/191403/5951529
with codecs.open(filename, 'r') as file_for_conversion:
with codecs.open(filename, 'w', 'utf-8') as converted_file:
print(filename +
' in ' +
fileencoding +
' encoding automatically converted to UTF-8 ')

kira_encoding_function()

• Kira1.md in UTF-8:
Kira Goddess!

• Kira2.md in Cyrillic-1251:
Кира Богиня!

• Kira3.md in Central European Cyrillic 1250:
Kiara Istennő!


## 4. Problem

I use with 3 times for opening same file. I don't think, that is a good practice.

Radon Cyclomatic Complexity not A:

D:\SashaDebugging\KiraEncoding>radon cc kira_encoding.py
kira_encoding.py
F 13:0 kira_encoding_function - B


I can not use with, but it anti-pattern.

## 5. Not helped

### 5.1. Modes

• That get encoding via chardet I need rb — bytes mode;
• That convert file via codecs I need non-bytes modes.

I can't find, what can I do, that to have same mode for these actions.

### 5.2. decode

I can remove 1 with, if I know file encoding.

cyrillic_file = bytes_file.decode('cp1251')
with codecs.open(filename, 'w', 'utf-8') as converted_file:
converted_file.write(cyrillic_file)

• Full file:
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')

def kira_encoding_function():
"""Check encoding and convert to UTF-8, if encoding no UTF-8."""
for filename in ALL_FILES:

# Not 100% accuracy:
# https://stackoverflow.com/a/436299/5951529
# Check:
# https://stackoverflow.com/a/37531241/5951529
with open(filename, 'rb') as opened_file:
chardet_data = chardet.detect(bytes_file)
fileencoding = (chardet_data['encoding'])
print('fileencoding', fileencoding)

if fileencoding in ['utf-8', 'ascii']:
print(filename + ' in UTF-8 encoding')
else:
# Convert file to UTF-8:
# https://stackoverflow.com/q/19932116/5951529
cyrillic_file = bytes_file.decode('cp1251')
with codecs.open(filename, 'w', 'utf-8') as converted_file:
converted_file.write(cyrillic_file)
print(filename +
' in ' +
fileencoding +
' encoding automatically converted to UTF-8')

kira_encoding_function()


But the files may not necessarily be in Cyrillic-1251, they can be in any encoding. I can't find, How can I decode from any encoding. For example, this can't work:

>>> kiragoddess = b'\xca\xe8\xf0\xe0 \xc1\xee\xe3\xe8\xed\xff!'
>>> kiragoddess.decode('cp1251')
'Кира Богиня!'
>>> kiragoddess.decode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte


### 5.3. Nested syntax

Nested syntax doesn't work for me. If:

with codecs.open(filename, 'r') as file_for_conversion, codecs.open(filename, 'w', 'utf-8') as converted_file:

• Full file:
# @Author: SashaChernykh
# @Date: 2018-09-01 13:31:06
"""kira_encoding module."""
import codecs
import glob

import chardet

ALL_FILES = glob.glob('*.md')

def kira_encoding_function():
"""Check encoding and convert to UTF-8, if encoding no UTF-8."""
for filename in ALL_FILES:

# Not 100% accuracy:
# https://stackoverflow.com/a/436299/5951529
# Check:
# https://stackoverflow.com/a/37531241/5951529
with open(filename, 'rb') as opened_file:
chardet_data = chardet.detect(bytes_file)
fileencoding = (chardet_data['encoding'])
print('fileencoding', fileencoding)

if fileencoding in ['utf-8', 'ascii']:
print(filename + ' in UTF-8 encoding')
else:
# Convert file to UTF-8:
# https://stackoverflow.com/a/191403/5951529
with codecs.open(filename, 'r') as file_for_conversion, codecs.open(filename, 'w', 'utf-8') as converted_file:
print(filename +
' in ' +
fileencoding +
' encoding automatically converted to UTF-8')

kira_encoding_function()


Content of non-UTF-8 files will be removed in this case.

### 6. Do not offer

1. Yes, I know that I need to use logging, not print in real programs. Please, do not offer it; my question not about it.
2. I need in-place conversion; my program must convert to UTF-8 to the same file, not to another.
• If you know not to use print, why didn't you replace it with logging already or drop it? – Mathias Ettinger Sep 1 '18 at 16:01
• @MathiasEttinger, because I endeavored to create a minimal example and remove code, which does not belong to my problem from item 4 of this question. Thanks. – Саша Черных Sep 1 '18 at 16:06
• And in your decode approach, why did you hardcode cp1251 instead of reusing what is stored in the fileencoding variable? – Mathias Ettinger Sep 1 '18 at 16:40
• @MathiasEttinger, this is my stupidity and lack of experience. Rewritten example. Thanks. – Саша Черных Sep 1 '18 at 18:37

1. The code in the post uses the chardet library to determine the encoding of the file, but then the only use it makes of that information is to decide whether or not to try transcoding the file. The detected encoding should also be used to decode the content, using bytes.decode:

with open(filename, 'rb') as f:
detected = chardet.detect(content_bytes)
encoding = detected['encoding']
print(f"{filename}: detected as {encoding}.")
content_text = content_bytes.decode(encoding)
with open(filename, 'w', 'utf-8') as f:
f.write(content_text)


Notice that in this version the file only needs to be read once.

2. There ought to be some check against the confidence value returned by chardet, to reduce the risk of making nonsense out of files where the encoding is detected wrongly. For example, you could have a global constant:

CONFIDENCE_THRESHOLD = 0.8 # minimum confidence to transcode the file


and then skip files where the detected confidence is below the threshold:

confidence = detected['confidence']
print(f"{filename}: detected as {encoding} with confidence {confidence}.")
if confidence < CONFIDENCE_THRESHOLD:
print(f"{filename}: skipped.")
continue

3. Editing a file by reading it and then writing back to the same file is an anti-pattern.

with codecs.open(filename, 'r') as file_for_conversion:
with codecs.open(filename, 'w', 'utf-8') as converted_file:
# what if there is an exception here?


The problem with this approach is that there might be an exception after the file is opened for writing and before it is closed. For example, the user might type control-C. If this happens, then the content of the file might be lost.

To make the code robust against this risk, you need to write the output to a temporary file, and then rename the temporary file using os.replace. The Python documentation guarantees that

the renaming will be an atomic operation

where "atomic" means that it either succeeds completely (meaning that the file is converted) or fails completely (leaving the original file unchanged), with no other possibilities. So what you need is something like this, using tempfile.NamedTemporaryFile to create the temporary file:

with open(filename, 'rb') as f: