Skip to main content
edited tags
Link
200_success
  • 144.2k
  • 22
  • 188
  • 473

Remove control and Nonnon-ASCII characters from Large Fileslarge files

Tweeted twitter.com/StackCodeReview/status/960382665382879232
Fixed spelling error
Link
Missy
  • 157
  • 1
  • 6

Remove control and nonNon-ASCII characters rrom large filesfrom Large Files

edited title and tags; edited tags
Source Link
t3chb0t
  • 44.3k
  • 9
  • 84
  • 190

Remove Controlcontrol and Nonnon-ASCII Characters From Large Files C#characters rrom large files

I get huge incoming files ( upup to 6GB) and they are littered with Control and non-ASCII characters. I need to strip them and I made this routine (below). The problem is that it is insanely slow. I would love any thoughts or advice on how I can speed it up.

    public void StripHighBitCharacters(string fn)
    {
        string writeFile = fn + "B";
        using (var reader = new StreamReader(fn))
        using (var writer = new StreamWriter(writeFile))
        {
            while (!reader.EndOfStream)
            {
                string line = reader.ReadLine();
                if (line.Length > 0)
                {
                    writer.WriteLine(BuildClearString(line));
                }
                else
                {
                    writer.WriteLine(line);
                }
            }
        }
        File.Copy(writeFile, fn, true);
        File.Delete(writeFile);
    }
    public string BuildClearString(string line)
    {

        StringBuilder sb = new StringBuilder();

        foreach (char c in line)
        {
            if (c >= 32 && c <= 175)
            {
                sb.Append(c);
            }
        }

        return (sb.ToString());
    }

Remove Control and Non-ASCII Characters From Large Files C#

I get huge incoming files ( up to 6GB) and they are littered with Control and non-ASCII characters. I need to strip them and I made this routine (below). The problem is that it is insanely slow. I would love any thoughts or advice on how I can speed it up.

    public void StripHighBitCharacters(string fn)
    {
        string writeFile = fn + "B";
        using (var reader = new StreamReader(fn))
        using (var writer = new StreamWriter(writeFile))
        {
            while (!reader.EndOfStream)
            {
                string line = reader.ReadLine();
                if (line.Length > 0)
                {
                    writer.WriteLine(BuildClearString(line));
                }
                else
                {
                    writer.WriteLine(line);
                }
            }
        }
        File.Copy(writeFile, fn, true);
        File.Delete(writeFile);
    }
    public string BuildClearString(string line)
    {

        StringBuilder sb = new StringBuilder();

        foreach (char c in line)
        {
            if (c >= 32 && c <= 175)
            {
                sb.Append(c);
            }
        }

        return (sb.ToString());
    }

Remove control and non-ASCII characters rrom large files

I get huge incoming files (up to 6GB) and they are littered with Control and non-ASCII characters. I need to strip them and I made this routine (below). The problem is that it is insanely slow. I would love any thoughts or advice on how I can speed it up.

public void StripHighBitCharacters(string fn)
{
    string writeFile = fn + "B";
    using (var reader = new StreamReader(fn))
    using (var writer = new StreamWriter(writeFile))
    {
        while (!reader.EndOfStream)
        {
            string line = reader.ReadLine();
            if (line.Length > 0)
            {
                writer.WriteLine(BuildClearString(line));
            }
            else
            {
                writer.WriteLine(line);
            }
        }
    }
    File.Copy(writeFile, fn, true);
    File.Delete(writeFile);
}
public string BuildClearString(string line)
{

    StringBuilder sb = new StringBuilder();

    foreach (char c in line)
    {
        if (c >= 32 && c <= 175)
        {
            sb.Append(c);
        }
    }

    return (sb.ToString());
}
Source Link
Missy
  • 157
  • 1
  • 6
Loading