Replacing non-ASCII characters

Question

I wrote a C# program to remove non-ASCII characters in a text file, and then output the result to a .NonAsciiChars file.

The input file is in XML format. In fact, the data may all be on two lines, which is why I am not doing the replacement line by line. Instead, I'm using StreamReader.ReadToEnd().

The problem is, the input file can be as big as 4 GB. When this happens, I'm getting the following OutOfMemoryException:

DateTime:2014-08-04 12:55:26,035 Thread ID:[1] Log Level:ERROR Logger Property:OS_fileParser.Program property:[(null)] - Message:System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Text.StringBuilder.ExpandByABlock(Int32 minBlockCharCount)
   at System.Text.StringBuilder.Append(Char* value, Int32 valueCount)
   at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
   at System.IO.StreamReader.ReadToEnd()
   at OS_fileParser.MyProgram.FormatXmlFile(String inFile) in D:\Test\myProgram.cs:line 530
   at OS_fileParser.MyProgram.Run() in D:\Test\myProgram.cs:line 336

Line 530 contains content = Regex.Replace(content, pattern, "");, while line 336 calls a method with the following body:

const string pattern = @"[^\x20-\x7E]";

string content;
using (var reader = new StreamReader(inFile))
{
    content = reader.ReadToEnd();
    reader.Close();
}

content = Regex.Replace(content, pattern, "");

using (var writer = new StreamWriter(inFile + ".NonAsciiChars"))
{
    writer.Write(content);
    writer.Close();
}

using (var myXmlReader = XmlReader.Create(inFile + ".NonAsciiChars", myXmlReaderSettings))
{
    try
    {
        while (myXmlReader.Read())
        {
        }
    }
    catch (XmlException ex)
    {
        Logger.Error("Validation error: " + ex);
    }
}

How can I improve the memory footprint of my code?

Answer 1

@svick's approach is the right one, given these considerations

• the input file can be as big as 4 GB
• the data may all be on two lines

However I would suggest that regular expressions are the wrong tool for the job, and you will find it faster to use a StreamReader with a specified encoding.

There is a method Encoding.GetEncoding that does the following:

Returns the encoding associated with the specified code page name. Parameters specify an error handler for characters that cannot be encoded and byte sequences that cannot be decoded.

There is also a DecoderReplacementFallback class:

Provides a failure-handling mechanism, called a fallback, for an encoded input byte sequence that cannot be converted to an output character. The fallback emits a user-specified replacement string instead of a decoded input byte sequence.

Putting that all together would look like this:

var encoding = Encoding.GetEncoding("us-ascii", new EncoderExceptionFallback(), new DecoderReplacementFallback(string.Empty));
const int BufferSize = 4096;
using (var reader = new StreamReader("in.txt", encoding))
using (var writer = new StreamWriter("out.txt", false, new UTF8Encoding(false), BufferSize))
{
    var buffer = new char[BufferSize];
    int read;
    while ((read = reader.Read(buffer, 0, BufferSize)) != 0)
    {
        writer.Write(buffer, 0, read);
    }
}

Now that use of UTF8Encoding might look like a mistake, but according to MSDN:

If you are planning to use an ASCII encoding (ASCIIEncoding), choose UTF8Encoding instead. The two encodings are identical for the ASCII character set, but UTF8Encoding has the following advantages:

• It has been tuned to be as fast as possible and should be faster than any other encoding. Even for content that is entirely ASCII, operations performed with UTF8Encoding are faster than operations performed with ASCIIEncoding.

We use the constructor so that it doesn't emit a byte order mark, unlike Encoding.UTF8.

---

I wrote the code above based on your description "remove non-ASCII characters in a text file", but now realise your requirements are more specific: to remove characters outside the range 32-126. It can be easily modified to support that requirement:

var encoding = Encoding.GetEncoding("us-ascii", new EncoderExceptionFallback(), new DecoderReplacementFallback(string.Empty));
const int BufferSize = 4096;
using (var reader = new StreamReader("in.txt", encoding))
using (var writer = new StreamWriter("out.txt", false, new UTF8Encoding(false), BufferSize))
{
    var buffer = new char[BufferSize];
    var chars = new char[BufferSize];
    int read;
    while ((read = reader.Read(buffer, 0, BufferSize)) != 0)
    {
        var j = 0;
        for (var i = 0; i < read; i++)
        {
            char c = buffer[i];
            if (c >= 32 && c <= 126)
            {
                chars[j++] = c;
            }
        }

        writer.Write(chars, 0, j);
    }
}

Timing

I ran tests on a 4GB file with random content, generated by Dummy File Creator.

$ time tr -cd '\40-\176' < in.txt > out.txt

real    0m52.034s
user    0m24.897s
sys     0m22.791s

$ time ./AsciiCopy.exe in.txt out.txt

real    1m9.140s
user    0m0.015s
sys     0m0.031s

$ time ./AsciiCopyRegex.exe in.txt out.txt

real    17m59.618s
user    0m0.031s
sys     0m0.016s

Answer 2

You need to use the two Streams as, well, streams: read a manageable part of the input, transform it, write it to the output and repeat.

int bufferSize = 4096; // or whatever

char[] characters = new char[bufferSize];

using (var reader = new StreamReader(inFile))
using (var writer = new StreamWriter(inFile + ".NonAsciiChars"))
{
    while (true)
    {
        int read = reader.Read(characters, 0, characters.Length);

        if (read == 0)
            break;

        var replaced = Regex.Replace(new string(characters), pattern, string.Empty);

        writer.Write(replaced);
    }
}

Some notes about this code:

1. Notice the missing Close() calls: the whole point of using is safe closing of streams, and similar resources, so you don't need to close them twice.
2. This code (just like the original) creates a lot of garbage to be collected by the GC. Since your regex is actually very simple, it might be better to manually work directly with char[]s.
3. I used string.Empty instead of "". This makes it very clear that empty string was actually intended and it's not just “I started writing string and then forgot about it” bug.

Answer 3

okay first of all, you need to use buffered stream to read data from the file, there is no need to write data once in file as it might hung altogether.

so I would suggest that read the file line by line , parse it and then write to file.

using (FileStream fileStream = File.Open(infile, FileMode.Open))
{
    using (var bufferedStream = new BufferedStream(fileStream))
    {
        using (var writer = File.AppendText(infile + ".NonAsciiChars"))
        {
            using (var stremReader = new StreamReader(bufferedStream))
            {
                string inputContent;
                while ((inputContent = stremReader.ReadLine()) != null)
                {
                    string content = Regex.Replace(inputContent, pattern, "");
                    writer.WriteLine(content); //you can replace with write too if you want
                }
            }
            writer.Close();
        }
    }
}