Reducing memory footprint when manipulating big csv file

Question

I have CSV file which has structure like this (separated by tabulators):

VariableName    DateAdded   ValueNumeric
VariableName    DateAdded   ValueNumeric
VariableName    DateAdded   ValueNumeric

Examples of lines in input data: VariableName DateAdded ValueNumeric

St.podatkovnih_blokov   1.12.2015 0:00:21   0,2000
St.podatkovnih_blokov   1.12.2015 0:01:15   0,2000
St.podatkovnih_blokov   1.12.2015 0:02:14   0,2000
.
.
.
St.podatkovnih_blokov   31.12.2015 10:08:02 0,2000
St.podatkovnih_blokov   31.12.2015 22:31:04 0,2000
NAD_krmilnika   1.12.2015 0:00:21   1310,2000
NAD_krmilnika   1.12.2015 0:01:15   1310,2000

... and I am changing it to form where there are all variables with same date in same line:

DateAdded,VariableName,VariableName,VariableName,VariableName,VariableName //this is header and below are values for variables in the header
DateAdded,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric
DateAdded,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric

So lines in output data look like this:

datum,St.podatkovnih_blokov,NAD_krmilnika,K5_en_hlajenja_MWh_last_month,K5_en_hlajenja_MWh_this_year,K5_en_hlajenja_MWh_last_year,Rezerva_1,Run_Counter,negative_active_energy_today,negative_active_energy_yesterday,negative_active_energy_this_week,negative_active_energy_last_week,negative_active_energy_this_mont,negative_active_energy_last_mont,negative_active_energy_this_year,negative_active_energy_last_year,strosek_danes_EUR,strosek_vceraj_EUR,K1_en_gretja_kWh_today,K1_en_gretja_kWh_yesterday,K1_en_gretja_MWh_this_month,K1_en_gretja_MWh_last_month,K1_en_gretja_MWh_this_year,K1_en_gretja_MWh_last_year,K1_en_hlajenja_MWh_this_year,K2_en_gretja_kWh_today,K2_en_gretja_kWh_yesterday,K2_en_gretja_MWh_this_month,K2_en_gretja_MWh_last_month,K2_en_gretja_MWh_this_year,K2_en_gretja_MWh_last_year,K2_en_hlajenja_kWh_today,K2_en_hlajenja_kWh_yesterday,K2_en_hlajenja_MWh_this_month,K2_en_hlajenja_MWh_last_month,K2_en_hlajenja_MWh_this_year,K2_en_hlajenja_MWh_last_year,K3_en_gretja_kWh_today,K3_en_gretja_kWh_yesterday,K3_en_gretja_MWh_this_month,K3_en_gretja_MWh_last_month,K3_en_gretja_MWh_this_year,K3_en_gretja_MWh_last_year,K3_en_hlajenja_kWh_today,K3_en_hlajenja_kWh_yesterday,K3_en_hlajenja_MWh_this_month,K3_en_hlajenja_MWh_last_month,K3_en_hlajenja_MWh_this_year,K3_en_hlajenja_MWh_last_year,K5_en_gretja_kWh_today,K5_en_gretja_kWh_yesterday,K5_en_gretja_MWh_this_month,K5_en_gretja_MWh_last_month,K5_en_gretja_MWh_this_year,K5_en_gretja_MWh_last_year,K5_en_hlajenja_kWh_today,K5_en_hlajenja_kWh_yesterday,K5_en_hlajenja_MWh_this_month,K235_en_gretja_kWh_today,K235_en_gretja_kWh_yesterday,K235_en_gretja_MWh_this_month,K235_en_gretja_MWh_last_month,K235_en_gretja_MWh_this_year,K235_en_gretja_MWh_last_year,K235_en_hlajenja_kWh_today,K235_en_hlajenja_kWh_yesterday,K235_en_hlajenja_MWh_this_month,K235_en_hlajenja_MWh_last_month,K235_en_hlajenja_MWh_this_year,K235_en_hlajenja_MWh_last_year,K235_trenutna_moc_kW,tekoce_24h_povprecje_Tzun,tekoce_48h_povprecje_Tzun,sprememba_1h_Tzun,sprememba_3h_Tzun,sprememba_6h_Tzun,sprememba_12h_Tzun,sprememba_24h_Tzun,T23,T51,T52,T
"1.12.2015 0:00:21","0,2000","1310,2000","-0,0200","-32,5900","0,0000","8,5000","2263,6000","0,7000","53,0000","53,0000","459,0000","0,0000","1185,0000","-31491,0000","18347,0000","0,0900","7,1500","0,0000","124,4000","0,0000","9,8100","184,6000","0,0000","-2,5500","0,0000","7,3000","0,0000","0,3600","2,7800","0,0000","0,0000","0,0000","0,0000","0,0000","-11,2800","0,0000","12,1000","307,0000","0,0100","8,8600","64,9400","0,0000","0,0000","-1,7000","0,0000","-0,0200","-40,8700","0,0000","3,5000","116,2000","0,0000","1,6300","6,4800","0,0000","0,0000","-0,4000","0,0000","15,6000","430,7000","0,0000","10,8000","74,2000","0,0000","0,0000","-2,2000","0,0000","0,0000","-84,7000","-84,7000","14,2000","8,4000","8,4000","-0,1000","0,0000","0,1000","999,0000","999,0000","22,2000","16,4000","16,5000"
"1.12.2015 0:01:15","0,2000","1310,2000","-0,0200","-32,5900","0,0000","8,5000","2682,6000","0,8000","53,0000","53,0000","459,0000","0,0000","1185,0000","-31491,0000","18347,0000","0,1000","7,1500","0,0000","124,4000","0,0000","9,8100","184,6000","0,0000","-2,5500","0,0000","7,3000","0,0000","0,3600","2,7800","0,0000","0,0000","0,0000","0,0000","0,0000","-11,2800","0,0000","12,3000","307,0000","0,0100","8,8600","64,9400","0,0000","0,0000","-1,7000","0,0000","-0,0200","-40,8700","0,0000","3,5000","116,2000","0,0000","1,6300","6,4800","0,0000","0,0000","-0,4000","0,0000","15,8000","430,7000","0,0000","10,8000","74,2000","0,0000","0,0000","-2,2000","0,0000","0,0000","-84,7000","-84,7000","13,7000","8,4000","8,4000","-0,1000","0,0000","0,1000","999,0000","999,0000","22,2000","16,4000","16,5000"

... Size of files that I use is more then 160mb. After manipulation is done, file size is a little bit more then 30mb.

When application is finished it also consumes around 180mb of memory (TaskManager). Could I reduce memory consumption any further? Here is the code of application:

static ArrayList<String> lines = new ArrayList();
static String outputFile = "MIC1deci_2015.csv";
static File importFile = new File("export.csv");
static Scanner scanner;
static StringBuilder lineBuilder = new StringBuilder();
static StringBuilder variableBuilder = new StringBuilder();
static int countStolpec = 0;
static int lineCount = 1;
static String variable = "";
static String oldVariable = "";
static int index = 0;
static int multi = 1;
static String value = "";
static String time = "";
static String date = "";
static boolean firstRead = true;  

public static void main(String[] args) throws FileNotFoundException 
        {

            readSettingsFile();
            try
            {
                scanner = new Scanner(importFile); //Scnaer which reads out file
                scanner.next();
                scanner.next();
                scanner.next();

                oldVariable = scanner.next(); //in old variable we save, current variable that we are reading
                variableBuilder.append("datum,").append(oldVariable);
                lines.add(0,variableBuilder.toString());
                try
                {
                    while(scanner.hasNext())
                    {
                            if(firstRead)
                            {
                                buildFirstColumn(); //builds first column which contains date, until it reaches end of first variable
                            }
                            else
                            {
                                buildRestOfValues(); //adds variable values to appropriate date
                            }        
                    }
                }
                catch(Exception ex)
                {
                    System.out.println("ex in main loop: " + ex);
                }
                writeToOutputfile(); //when we have our data ready, we write it to export file, where values are seperated by ','
            }
            catch(IOException ex)
            {
                Logger.getLogger(ScadaParse.class.getName()).log(Level.SEVERE, null, ex);
            }
        }

        static public void printLines()
        {
            int i = 0;
            System.out.println("Printing lines");
            for(String line : lines)
            {
                i++;
                System.out.println("line " + i + ": " + line);
            }
        }

        static public void writeToOutputfile() throws IOException
        {
            File file = new File(outputFile);
            file.createNewFile();
            try 
            {
                FileWriter writer = new FileWriter(file);
                System.out.print("Writing raw... ");
                write(lines, writer);
            } finally {
            }
        }

        private static void write(List<String> records, Writer writer) throws IOException {
        long start = System.currentTimeMillis();
        for (String record: records) {
            writer.write(record + "\n");
        }
        writer.flush();
        writer.close();
        long end = System.currentTimeMillis();
        System.out.println((end - start) / 1000f + " seconds");
        }

        static void buildFirstColumn()
        {
            lineBuilder.append("\"").append(scanner.next()).append(" ").append(scanner.next()).append("\",\"").append(scanner.next()).append("\"");
            lines.add(lineCount,lineBuilder.toString());
            lineCount++;
            lineBuilder.setLength(0);
            variable = scanner.next();
            if(!oldVariable.equals(variable))
            {
                System.out.println("old: " + oldVariable + " new:" + variable);
                oldVariable = variable;
                variableBuilder.append(",").append(variable);
                lineCount = 1;
                firstRead = false;
            }
        }

        static void buildRestOfValues()
        {
            scanner.next();
            scanner.next();
            value = scanner.next();
            lineBuilder.setLength(0);

            try
            {
                String content = lines.get(lineCount);
                lines.set(lineCount,lineBuilder.append(content).append(",\"").append(value).append("\"").toString());
            }
            catch(Exception ex)
            {
                System.out.println("index for lines to big: " + ex);
            }
            lineCount++;
            variable = scanner.next();
            if(!oldVariable.equals(variable))
            {
                System.out.println("old: " + oldVariable + " new:" + variable);
                oldVariable = variable;
                variableBuilder.append(",").append(variable);
                lines.set(0,variableBuilder.toString());
                lineCount = 1;
            }
        }   

Answer 1

Never catch exception

        catch(Exception ex)
        {
            System.out.println("index for lines to big: " + ex);
        }

Catching the just the Exception class is a anti-pattern, you don't know what exceptions are going to be catch;t.

In this situation, you should check if the index is valid, and if it isn't valid, then print a message.

Don't do string concatenation when working with a writer

private static void write(List<String> records, Writer writer) throws IOException {
    long start = System.currentTimeMillis();
    for (String record: records) {
        writer.write(record + "\n");
    }
    writer.flush();

You are doing string concatenation when writing, when doing this, a copy of the string is created in the memory, to be finally thrown away. Change the code to 2 different calls:

writer.write(record);
writer.write(System.lineSeparator());

A stream should be closed in the same scope as where its opened

        try 
        {
            FileWriter writer = new FileWriter(file);
            System.out.print("Writing raw... ");
            write(lines, writer);
        } finally {
        }

By closing a stream at the point where it is opened, you prevent any exception from preventing the closure.

        try(FileWriter writer = new FileWriter(file)) {
            System.out.print("Writing raw... ");
            write(lines, writer);
        }

Inconsistent order of modifiers

public static void main(String[] args) throws FileNotFoundException 
static public void printLines()
static public void writeToOutputfile() throws IOException
private static void write(List<String> records, Writer writer) throws IOException {
static void buildRestOfValues()

Sometimes you use "access modifier", "static" and other times, you do it the other way around. Having a default standard for these kind of things makes your code look better.

Inconsistent output to logger / output to stdout

Logger.getLogger(ScadaParse.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("ex in main loop: " + ex);

Your code is inconsistent when it should output to a logger or to the system out, making code reuse very hard.

Everything's static

My making everything static, your code cannot be properly reused in other places, you should place everything inside a object, and every setting should passed in either via the constructor, or via a design pattern like the builder of factory.

Variables have unnecessary broad scopes

You variables have a unneeded global scope, by limiting the scope of variables to the places they are used makes your code nicer.

Use a imperative for loop instead of a iterator if your needing the index

        int i = 0;
        System.out.println("Printing lines");
        for(String line : lines)
        {
            i++;
            System.out.println("line " + i + ": " + line);
        }

Why not just a simple for loop?

int length = lines.size();
for(int i = 0; i < length; i++) {
    System.out.println("line " + i + ": " + lines.get(i));
}