3
\$\begingroup\$

I am getting Out of Memory errors when I try to parse through a large CSV file (2.5Gb). My computer has 32Gb of Memory but Perl uses all of it up. The CSV has 2 columns. The first is the time in epoch and the second is a 10000+ line XML file as a single-line. There are around 13k rows. I then use XML::XPath to retrieve the customer name and save the XML to [customername]-[time].xml. If there is an error, it is because the XML is invalid and I skip it.

Is there any way to make my code run more efficiently and look better?

#!/usr/bin/perl

use strict;
use warnings;
use XML::XPath;
use XML::XPath::XMLParser;
use File::Slurp;

my $file = '../FILENAME.csv';
open my $info, $file or die "Could not open $file: $!";
my $count = 0;
$| = 1;

while( my $line = <$info>)  {
    next if ++$count == 1; #Ignore headers
    my ($time, $report) = ($line =~ /(\d+),(.*)$/); # time, XML file
    eval {
        my $xp = XML::XPath->new(xml => $report);
        our $ext = $xp->getNodeText('/report/customer') . "-" . $time . ".xml";
        write_file($ext, $report);
    };
    if ( $@ ) {printf "ERROR ";}
    else {printf "$count ";}
}

close $info;
\$\endgroup\$
1
  • 1
    \$\begingroup\$ Have you looked at Parse::CSV for the CSV loading? From the POD: "A number of other modules have attempted to put usability wrappers around [Text::CSV_XS], but they have all focused on parsing the entire file into memory at once. This method is fine unless your CSV files start to get large. [...] Parse::CSV fills this functionality gap." \$\endgroup\$
    – jja
    May 12, 2015 at 16:15

1 Answer 1

2
\$\begingroup\$

I am suspicious of this line here:

my ($time, $report) = ($line =~ /(\d+),(.*)$/); # time, XML file

That line needs to scan and group the entire XML doc, but my instinct is that a 2-limited split (see note on LIMIT) would be more efficient:

my ($time, $report) = split /,/, $line, 2;

Apart from that, really, everything looks sane, and far better than a typical perl hack. I am impressed.

About perl using all your memory... are you sure? perl may just be struggling to allocate more than it is allowed. Do you have a ulimit condition that is constraining memory allocation? Are there problems with heavily fragmented memory? (You're not running 32-bit, are you... just checking?)

As it happens, for a task like this, you may find that everything is better with a language other than perl. Not to dismiss perl as incapable, but with the XPath expressions and other items your code looks surprisingly complicated. 2.5gig files are way beyond the normal I would consider for perl performance.

\$\endgroup\$
7
  • \$\begingroup\$ Well the XML report MAY contain commas. But the files are always in the format 1234567890,<XML Report> so I guess i just want the epoch time to be $time and everything else to be $report. \$\endgroup\$
    – Bijan
    May 11, 2015 at 18:26
  • 1
    \$\begingroup\$ @Bijan - Note where I said the 2-limit split... if the XML contains commas that's fine. See the doc: perldoc.perl.org/functions/split.html (Look for the LIMIT paragraph) \$\endgroup\$
    – rolfl
    May 11, 2015 at 18:28
  • \$\begingroup\$ I should have taken a screenshot but Perl was using 31GB of my memory. Windows was creating an extremely large paging file (90Gb) and i was running out of room on my SSD so I limited the paging file to 40Gb. Would my code run much more efficiently if I got rid of the XPath to get customer name? I can always name file to the epoch time, it's not a big deal to have customer name \$\endgroup\$
    – Bijan
    May 11, 2015 at 18:31
  • 1
    \$\begingroup\$ @Bijan - it is very hard to imagine why your code is exploding in memory like that. It makes no sense, and is going in to the realm of perl bugs, not program bugs. From a review perspective, though, your code looks fine. I would consider slurping the entire input file, and manually identifying line-breaks. That would be my next course of action. \$\endgroup\$
    – rolfl
    May 11, 2015 at 18:35
  • 1
    \$\begingroup\$ @Bijan - let's take this to the 2nd monitor chat room \$\endgroup\$
    – rolfl
    May 11, 2015 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.