3
\$\begingroup\$

I have CSV file which has structure like this (separated by tabulators):

VariableName    DateAdded   ValueNumeric
VariableName    DateAdded   ValueNumeric
VariableName    DateAdded   ValueNumeric

Examples of lines in input data: VariableName DateAdded ValueNumeric

St.podatkovnih_blokov   1.12.2015 0:00:21   0,2000
St.podatkovnih_blokov   1.12.2015 0:01:15   0,2000
St.podatkovnih_blokov   1.12.2015 0:02:14   0,2000
.
.
.
St.podatkovnih_blokov   31.12.2015 10:08:02 0,2000
St.podatkovnih_blokov   31.12.2015 22:31:04 0,2000
NAD_krmilnika   1.12.2015 0:00:21   1310,2000
NAD_krmilnika   1.12.2015 0:01:15   1310,2000

... and I am changing it to form where there are all variables with same date in same line:

DateAdded,VariableName,VariableName,VariableName,VariableName,VariableName //this is header and below are values for variables in the header
DateAdded,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric
DateAdded,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric,ValueNumeric

So lines in output data look like this:

datum,St.podatkovnih_blokov,NAD_krmilnika,K5_en_hlajenja_MWh_last_month,K5_en_hlajenja_MWh_this_year,K5_en_hlajenja_MWh_last_year,Rezerva_1,Run_Counter,negative_active_energy_today,negative_active_energy_yesterday,negative_active_energy_this_week,negative_active_energy_last_week,negative_active_energy_this_mont,negative_active_energy_last_mont,negative_active_energy_this_year,negative_active_energy_last_year,strosek_danes_EUR,strosek_vceraj_EUR,K1_en_gretja_kWh_today,K1_en_gretja_kWh_yesterday,K1_en_gretja_MWh_this_month,K1_en_gretja_MWh_last_month,K1_en_gretja_MWh_this_year,K1_en_gretja_MWh_last_year,K1_en_hlajenja_MWh_this_year,K2_en_gretja_kWh_today,K2_en_gretja_kWh_yesterday,K2_en_gretja_MWh_this_month,K2_en_gretja_MWh_last_month,K2_en_gretja_MWh_this_year,K2_en_gretja_MWh_last_year,K2_en_hlajenja_kWh_today,K2_en_hlajenja_kWh_yesterday,K2_en_hlajenja_MWh_this_month,K2_en_hlajenja_MWh_last_month,K2_en_hlajenja_MWh_this_year,K2_en_hlajenja_MWh_last_year,K3_en_gretja_kWh_today,K3_en_gretja_kWh_yesterday,K3_en_gretja_MWh_this_month,K3_en_gretja_MWh_last_month,K3_en_gretja_MWh_this_year,K3_en_gretja_MWh_last_year,K3_en_hlajenja_kWh_today,K3_en_hlajenja_kWh_yesterday,K3_en_hlajenja_MWh_this_month,K3_en_hlajenja_MWh_last_month,K3_en_hlajenja_MWh_this_year,K3_en_hlajenja_MWh_last_year,K5_en_gretja_kWh_today,K5_en_gretja_kWh_yesterday,K5_en_gretja_MWh_this_month,K5_en_gretja_MWh_last_month,K5_en_gretja_MWh_this_year,K5_en_gretja_MWh_last_year,K5_en_hlajenja_kWh_today,K5_en_hlajenja_kWh_yesterday,K5_en_hlajenja_MWh_this_month,K235_en_gretja_kWh_today,K235_en_gretja_kWh_yesterday,K235_en_gretja_MWh_this_month,K235_en_gretja_MWh_last_month,K235_en_gretja_MWh_this_year,K235_en_gretja_MWh_last_year,K235_en_hlajenja_kWh_today,K235_en_hlajenja_kWh_yesterday,K235_en_hlajenja_MWh_this_month,K235_en_hlajenja_MWh_last_month,K235_en_hlajenja_MWh_this_year,K235_en_hlajenja_MWh_last_year,K235_trenutna_moc_kW,tekoce_24h_povprecje_Tzun,tekoce_48h_povprecje_Tzun,sprememba_1h_Tzun,sprememba_3h_Tzun,sprememba_6h_Tzun,sprememba_12h_Tzun,sprememba_24h_Tzun,T23,T51,T52,T
"1.12.2015 0:00:21","0,2000","1310,2000","-0,0200","-32,5900","0,0000","8,5000","2263,6000","0,7000","53,0000","53,0000","459,0000","0,0000","1185,0000","-31491,0000","18347,0000","0,0900","7,1500","0,0000","124,4000","0,0000","9,8100","184,6000","0,0000","-2,5500","0,0000","7,3000","0,0000","0,3600","2,7800","0,0000","0,0000","0,0000","0,0000","0,0000","-11,2800","0,0000","12,1000","307,0000","0,0100","8,8600","64,9400","0,0000","0,0000","-1,7000","0,0000","-0,0200","-40,8700","0,0000","3,5000","116,2000","0,0000","1,6300","6,4800","0,0000","0,0000","-0,4000","0,0000","15,6000","430,7000","0,0000","10,8000","74,2000","0,0000","0,0000","-2,2000","0,0000","0,0000","-84,7000","-84,7000","14,2000","8,4000","8,4000","-0,1000","0,0000","0,1000","999,0000","999,0000","22,2000","16,4000","16,5000"
"1.12.2015 0:01:15","0,2000","1310,2000","-0,0200","-32,5900","0,0000","8,5000","2682,6000","0,8000","53,0000","53,0000","459,0000","0,0000","1185,0000","-31491,0000","18347,0000","0,1000","7,1500","0,0000","124,4000","0,0000","9,8100","184,6000","0,0000","-2,5500","0,0000","7,3000","0,0000","0,3600","2,7800","0,0000","0,0000","0,0000","0,0000","0,0000","-11,2800","0,0000","12,3000","307,0000","0,0100","8,8600","64,9400","0,0000","0,0000","-1,7000","0,0000","-0,0200","-40,8700","0,0000","3,5000","116,2000","0,0000","1,6300","6,4800","0,0000","0,0000","-0,4000","0,0000","15,8000","430,7000","0,0000","10,8000","74,2000","0,0000","0,0000","-2,2000","0,0000","0,0000","-84,7000","-84,7000","13,7000","8,4000","8,4000","-0,1000","0,0000","0,1000","999,0000","999,0000","22,2000","16,4000","16,5000"

... Size of files that I use is more then 160mb. After manipulation is done, file size is a little bit more then 30mb.

When application is finished it also consumes around 180mb of memory (TaskManager). Could I reduce memory consumption any further? Here is the code of application:

static ArrayList<String> lines = new ArrayList();
static String outputFile = "MIC1deci_2015.csv";
static File importFile = new File("export.csv");
static Scanner scanner;
static StringBuilder lineBuilder = new StringBuilder();
static StringBuilder variableBuilder = new StringBuilder();
static int countStolpec = 0;
static int lineCount = 1;
static String variable = "";
static String oldVariable = "";
static int index = 0;
static int multi = 1;
static String value = "";
static String time = "";
static String date = "";
static boolean firstRead = true;  

public static void main(String[] args) throws FileNotFoundException 
        {

            readSettingsFile();
            try
            {
                scanner = new Scanner(importFile); //Scnaer which reads out file
                scanner.next();
                scanner.next();
                scanner.next();

                oldVariable = scanner.next(); //in old variable we save, current variable that we are reading
                variableBuilder.append("datum,").append(oldVariable);
                lines.add(0,variableBuilder.toString());
                try
                {
                    while(scanner.hasNext())
                    {
                            if(firstRead)
                            {
                                buildFirstColumn(); //builds first column which contains date, until it reaches end of first variable
                            }
                            else
                            {
                                buildRestOfValues(); //adds variable values to appropriate date
                            }        
                    }
                }
                catch(Exception ex)
                {
                    System.out.println("ex in main loop: " + ex);
                }
                writeToOutputfile(); //when we have our data ready, we write it to export file, where values are seperated by ','
            }
            catch(IOException ex)
            {
                Logger.getLogger(ScadaParse.class.getName()).log(Level.SEVERE, null, ex);
            }
        }

        static public void printLines()
        {
            int i = 0;
            System.out.println("Printing lines");
            for(String line : lines)
            {
                i++;
                System.out.println("line " + i + ": " + line);
            }
        }

        static public void writeToOutputfile() throws IOException
        {
            File file = new File(outputFile);
            file.createNewFile();
            try 
            {
                FileWriter writer = new FileWriter(file);
                System.out.print("Writing raw... ");
                write(lines, writer);
            } finally {
            }
        }

        private static void write(List<String> records, Writer writer) throws IOException {
        long start = System.currentTimeMillis();
        for (String record: records) {
            writer.write(record + "\n");
        }
        writer.flush();
        writer.close();
        long end = System.currentTimeMillis();
        System.out.println((end - start) / 1000f + " seconds");
        }

        static void buildFirstColumn()
        {
            lineBuilder.append("\"").append(scanner.next()).append(" ").append(scanner.next()).append("\",\"").append(scanner.next()).append("\"");
            lines.add(lineCount,lineBuilder.toString());
            lineCount++;
            lineBuilder.setLength(0);
            variable = scanner.next();
            if(!oldVariable.equals(variable))
            {
                System.out.println("old: " + oldVariable + " new:" + variable);
                oldVariable = variable;
                variableBuilder.append(",").append(variable);
                lineCount = 1;
                firstRead = false;
            }
        }

        static void buildRestOfValues()
        {
            scanner.next();
            scanner.next();
            value = scanner.next();
            lineBuilder.setLength(0);

            try
            {
                String content = lines.get(lineCount);
                lines.set(lineCount,lineBuilder.append(content).append(",\"").append(value).append("\"").toString());
            }
            catch(Exception ex)
            {
                System.out.println("index for lines to big: " + ex);
            }
            lineCount++;
            variable = scanner.next();
            if(!oldVariable.equals(variable))
            {
                System.out.println("old: " + oldVariable + " new:" + variable);
                oldVariable = variable;
                variableBuilder.append(",").append(variable);
                lines.set(0,variableBuilder.toString());
                lineCount = 1;
            }
        }   
\$\endgroup\$
4
  • \$\begingroup\$ In your input files, can the same date be found multiple times? Can variable names be added later, or are there no more names after reaching the first duplicate name? Can the order of variables in the main input file change midway though the file? \$\endgroup\$
    – Ferrybig
    Feb 16 '16 at 11:40
  • \$\begingroup\$ 1. Same date should be found for every variable. Like in the example of input data. Where you can see that both variables have first occurrence for date: 1.12.2015 0:00:21. 2. Lines are sorted by variable name, so if some variable already appeared in upper lines and after that we had read a new variable, we will never read old variable again. 3. If in input file data is sorted by variable names, data in output file is sorted by date. So one instance of date, can appear only once, and we append variable data to those instances. I hope that I made it more clear now. \$\endgroup\$
    – Jure
    Feb 16 '16 at 11:51
  • \$\begingroup\$ Do you have by any change lage versions of the input file? I want to see if a improved version I made has better memory usage. \$\endgroup\$
    – Ferrybig
    Feb 17 '16 at 12:53
  • \$\begingroup\$ You can download it here: drive.google.com/open?id=0B4p4jh0vfEBPZFhwTTFLS3JKN1U \$\endgroup\$
    – Jure
    Feb 17 '16 at 14:28
5
\$\begingroup\$

Never catch exception

        catch(Exception ex)
        {
            System.out.println("index for lines to big: " + ex);
        }

Catching the just the Exception class is a anti-pattern, you don't know what exceptions are going to be catch;t.

In this situation, you should check if the index is valid, and if it isn't valid, then print a message.

Don't do string concatenation when working with a writer

private static void write(List<String> records, Writer writer) throws IOException {
    long start = System.currentTimeMillis();
    for (String record: records) {
        writer.write(record + "\n");
    }
    writer.flush();

You are doing string concatenation when writing, when doing this, a copy of the string is created in the memory, to be finally thrown away. Change the code to 2 different calls:

writer.write(record);
writer.write(System.lineSeparator());

A stream should be closed in the same scope as where its opened

        try 
        {
            FileWriter writer = new FileWriter(file);
            System.out.print("Writing raw... ");
            write(lines, writer);
        } finally {
        }

By closing a stream at the point where it is opened, you prevent any exception from preventing the closure.

        try(FileWriter writer = new FileWriter(file)) {
            System.out.print("Writing raw... ");
            write(lines, writer);
        }

Inconsistent order of modifiers

public static void main(String[] args) throws FileNotFoundException 
static public void printLines()
static public void writeToOutputfile() throws IOException
private static void write(List<String> records, Writer writer) throws IOException {
static void buildRestOfValues()

Sometimes you use "access modifier", "static" and other times, you do it the other way around. Having a default standard for these kind of things makes your code look better.

Inconsistent output to logger / output to stdout

Logger.getLogger(ScadaParse.class.getName()).log(Level.SEVERE, null, ex);
System.out.println("ex in main loop: " + ex);

Your code is inconsistent when it should output to a logger or to the system out, making code reuse very hard.

Everything's static

My making everything static, your code cannot be properly reused in other places, you should place everything inside a object, and every setting should passed in either via the constructor, or via a design pattern like the builder of factory.

Variables have unnecessary broad scopes

You variables have a unneeded global scope, by limiting the scope of variables to the places they are used makes your code nicer.

Use a imperative for loop instead of a iterator if your needing the index

        int i = 0;
        System.out.println("Printing lines");
        for(String line : lines)
        {
            i++;
            System.out.println("line " + i + ": " + line);
        }

Why not just a simple for loop?

int length = lines.size();
for(int i = 0; i < length; i++) {
    System.out.println("line " + i + ": " + lines.get(i));
}
\$\endgroup\$
4
  • \$\begingroup\$ Just to comment on everything is static. I didn't place it in some object, because this won't be used/reused anywhere else. Thank you for your response. \$\endgroup\$
    – Jure
    Feb 16 '16 at 11:21
  • \$\begingroup\$ I implemented your corrections, but memory footprint is almost the same. \$\endgroup\$
    – Jure
    Feb 16 '16 at 11:27
  • \$\begingroup\$ The main cause of memory is the lines list in the application, if we know more about the input data, we may suggest a better approach to this \$\endgroup\$
    – Ferrybig
    Feb 16 '16 at 11:29
  • \$\begingroup\$ I had added to my question examples of lines in input and output files. \$\endgroup\$
    – Jure
    Feb 16 '16 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.