3
\$\begingroup\$

I would like to make the below code more efficient by writing all text(CSV) to memory before writing to disk. The CSV file could have upwards of 10,000 or rows. With the current code, it opens and closes the file for R/W each time. There must be a better way. I am calling the WriteToCSV2 on each OnBarUpdate. The file on disk after it runs is approx. 10MB so there is no issue storing the entire contents of in memory before writing to disk.

This is part of a NinjaScript for Ninjatrader.

private void WriteToCSV2()
{

    if(CurrentBar <= BarsRequiredToPlot + LookBack) return;
        
    StringBuilder header = new StringBuilder();
    StringBuilder logEntry = new StringBuilder();
    StringBuilder signalCode = new StringBuilder();

    string fullPath = System.IO.Path.Combine(filePath,fileName);
    
    header.Append("Index Key,");
    header.Append("CurrentBar,");
    header.Append("SignalCode,");
    header.Append("Open,");
    header.Append("High,");
    header.Append("Low,");
    header.Append("Close,");
    header.Append("BarCloseTime");

    //INPUT1
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input1_1_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }

    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input1_2_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }
    
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input1_3_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }
    
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input1_4_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }
    
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input2_1_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }

    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input2_2_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }
    
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input2_3_trendchar[i]);
        if(i == LookBack) signalCode.Append("-");
    }
    
    for(int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(input2_4_trendchar[i]);
    }

    logEntry.Append(indexKey);
    logEntry.Append(",");
    logEntry.Append(CurrentBar-1);  
    logEntry.Append(",");
    logEntry.Append(signalCode.ToString());
    logEntry.Append(",");
    logEntry.Append(Open[1]);
    logEntry.Append(",");
    logEntry.Append(High[1]);
    logEntry.Append(",");
    logEntry.Append(Low[1]);
    logEntry.Append(",");
    logEntry.Append(Close[1]);
    logEntry.Append(",");
    logEntry.Append(Time[1]);
    indexKey++;
    
     try
        {
            if (File.Exists(fullPath) == false)
            {
                using (StreamWriter sw = new StreamWriter(fullPath, true))
                {
                    sw.WriteLine(header); // If file doesnt exist, create it and add the Header
                    sw.WriteLine(logEntry); // Append a new line to the file
                    sw.Close(); // Close the file to allow future calls to access the file again.
                }
            }
            else //File Does Exisit
            {
                if (IsFileLocked(fullPath) == false) //If file is not locked for editing
                {
                    using (StreamWriter sw = new StreamWriter(fullPath, true))
                    {
                        sw.WriteLine(logEntry); // Append a new line to the file
                        sw.Close(); // Close the file to allow future calls to access the file again.
                    }
                }
            }
        }
        catch (Exception e)
        {
            // Outputs the error to the log
            //Log(uniqueStrategy + ": ExecutionLog - cannot write and read at the same time.", NinjaTrader.Cbi.LogLevel.Error);
            Print("ERROR WRITTING SignalLog");
        }
\$\endgroup\$
8
  • \$\begingroup\$ too many loops for a single operation! This might be a design issue. You must share all related code including the code where you handle the input to store them in arrays ! as it seems you're using an array for each column in the csv. Also, explain why you need to store it in memory ? what's your goal by storing it in memory rather than buffering it to disk ? \$\endgroup\$
    – iSR5
    Oct 10, 2020 at 17:16
  • 1
    \$\begingroup\$ This is why we need to see the related code to this part. As you can simply use models instead of multiple arrays, which would give you more managed structure to your data for more maintainability. \$\endgroup\$
    – iSR5
    Oct 10, 2020 at 20:25
  • 2
    \$\begingroup\$ we need more code around this. Why can't you just keep the filestream open? Why do you need to close it? What is all the loops and, I assume, arrays in the loops? Hard to give better options when just a small subsection of code. \$\endgroup\$ Oct 11, 2020 at 0:23
  • 1
    \$\begingroup\$ Example of fast CSV builder here. Btw, you may use some existing CSV tool e.g. CSVHelper. \$\endgroup\$
    – aepot
    Oct 11, 2020 at 14:56
  • 1
    \$\begingroup\$ Instead of strings with a length of one character, it is better to use char. In other words, replace double quotes with single ones. Append(",") => Append(',') \$\endgroup\$ Oct 12, 2020 at 21:01

2 Answers 2

2
\$\begingroup\$

My try to prettify the code. Some comments inside.

private void WriteToCSV2()
{
    if (CurrentBar <= BarsRequiredToPlot + LookBack) return;

    StringBuilder logEntry = new StringBuilder();

    string fullPath = Path.Combine(filePath, fileName);

    // don't need a Stringbuilder to build a constant string
    const string header = "Index Key,CurrentBar,SignalCode,Open,High,Low,Close,BarCloseTime";

    // StringBuilder has Fluent API, it means you may write like .Append.Append.Append...
    logEntry.Append(indexKey).Append(",");
    logEntry.Append(CurrentBar - 1).Append(",");

    var trendchars = new[] // let's make an array of it ...
    {
        input1_1_trendchar,
        input1_2_trendchar,
        input1_3_trendchar,
        input1_4_trendchar,
        input2_1_trendchar,
        input2_2_trendchar,
        input2_3_trendchar,
        input2_4_trendchar
    };

    foreach (var trendchar in trendchars) // ... and process it in a loop
    {
        for (int i = 1; i <= LookBack; i++)
        {
            logEntry.Append(trendchar[i]);
        }
        logEntry.Append("-");
    }
    logEntry.Remove(logEntry.Length - 1, 1); // cut the last redundant "-"
           
    logEntry.Append(",");
    logEntry.Append(Open[1]).Append(",");
    logEntry.Append(High[1]).Append(",");
    logEntry.Append(Low[1]).Append(",");
    logEntry.Append(Close[1]).Append(",");
    logEntry.Append(Time[1]);
    indexKey++;

    try
    {
        // "something == false" may be written as "!something"
        if (!File.Exists(fullPath))
        {
            File.WriteAllLines(fullPath, new[] { header, logEntry.ToString() });
        }
        else
        {
            // Warning! If the file is locked, here you may loose all the data to write
            if (!IsFileLocked(fullPath))
            {
                File.AppendAllLines(fullPath, new[] { logEntry.ToString() });
            }
        }
    }
    catch (Exception e)
    {
        // e.Message will tell what's exactly went wrong
        Print($"ERROR WRITTING SignalLog: {e.Message}");
    }
}

Possibly I've processed not all the things to optimize but only visible part of code.


Btw if you want write to disk once, you may try single StringBuilder, something like this:

StringBuilder sb = new StringBuilder();
const string header = ...
if (!File.Exists(filename))
{
    sb.AppendLine(header);
}

for (...)
{
    WriteToCSV2(sb);
    sb.AppendLine();
}

File.AppendAllText(filename, sb.ToString());

And change signature of the method

private void WriteToCSV2(StringBuilder logEntry)
{
    //...
}

MemoryStream would be a kind of redundancy here.

\$\endgroup\$
0
\$\begingroup\$

Thank you all for your input. I was able to dramatically make it more efficient by creating models and using CSVhelper. https://joshclose.github.io/CsvHelper/ to export all the historical records to disk, only once during transitioning to realtime. Much faster.

var trendchars = new[] // array of arrays
{
    input1_1_trendchar,
    input1_2_trendchar,
    input1_3_trendchar,
    input1_4_trendchar,
    input2_1_trendchar,
    input2_2_trendchar,
    input2_3_trendchar,
    input2_4_trendchar
};
    
foreach (var trendchar in trendchars) // ... and process it in a loop
{   
    for (int i = 1; i <= LookBack; i++)
    {
        signalCode.Append(trendchar[i]);
    }
    signalCode.Append("-");
}
    
signalCode.Remove(signalCode.Length - 1, 1); // cut the last redundant "-"
\$\endgroup\$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.