I am getting Out of Memory
errors when I try to parse through a large CSV file (2.5Gb). My computer has 32Gb of Memory but Perl uses all of it up. The CSV has 2 columns. The first is the time in epoch and the second is a 10000+ line XML file as a single-line. There are around 13k rows. I then use XML::XPath to retrieve the customer name and save the XML to [customername]-[time].xml
. If there is an error, it is because the XML is invalid and I skip it.
Is there any way to make my code run more efficiently and look better?
#!/usr/bin/perl
use strict;
use warnings;
use XML::XPath;
use XML::XPath::XMLParser;
use File::Slurp;
my $file = '../FILENAME.csv';
open my $info, $file or die "Could not open $file: $!";
my $count = 0;
$| = 1;
while( my $line = <$info>) {
next if ++$count == 1; #Ignore headers
my ($time, $report) = ($line =~ /(\d+),(.*)$/); # time, XML file
eval {
my $xp = XML::XPath->new(xml => $report);
our $ext = $xp->getNodeText('/report/customer') . "-" . $time . ".xml";
write_file($ext, $report);
};
if ( $@ ) {printf "ERROR ";}
else {printf "$count ";}
}
close $info;