3
\$\begingroup\$

Here are all the threads write the log file. I want to know if the below script is fine, if it needs any modifications, and whether I am utilising threads properly to execute this code quickly.

open(LOG,"+>","log.txt")or die "cant open the file";


my $file_total= scalar @listOfFiles;

    my $div = $file_total/$thread_count;
    $div = ceil($div);
    my $start = 0;
    my $end = $div;

    for ($count = 1; $count <=$thread_count ; $count++) 
    {
        my $thread = threads->new(\&register, $start, $end );
        push(@threads,$thread);        
        $start = $end;
        $end = $end + $div;
        if ($end > $file_total)
        {
            $end = $file_total;
        }
    }

    foreach (@threads) 
    {
       $_->join;
    }
    close(LOG);
    sub register
    {
         my $lstart = shift;
         my $lend = shift;

         for (my $index = $lstart; $index < $lend; ++$index) 
         {
             print LOG $listOfFiles[$index];
             process......
             ..........
             .......
         }
     }

process... placeholder: I have a huge number of documents, say, 1 million. I am collecting all the 1 million filenames, storing them in the listOfFiles array, and dividing the array index based on the number of threads the user wants. I am passing every thread and the start and end indices to fetch the document name, process the document, and store the status of the output to a log file.

\$\endgroup\$
  • \$\begingroup\$ I have some huge amount of document say 1 million.so iam collecting all the 1 million filenames and stored in the listOfFiles array.And dividing the array index based on no of threads the user wants.Iam passing every thread the start and end index to fetch the document name and process the document and store the status of the output to a log file \$\endgroup\$ – Venkatesan May 14 '14 at 9:49
2
\$\begingroup\$

You can atomically write to a file if you are working with a threadsafe queue.

#!/usr/bin/env perl

use warnings FATAL => 'all';
use strict;
use Time::HiRes qw(time nanosleep);
use threads;
use Thread::Queue;

my $logQ = Thread::Queue->new();

my $consumer = threads->new(\&_logConsumer);

my @producer;
push(@producer, threads->new(\&_logProducer)) for (1..10);
$_->join() foreach @producer;

$logQ->enqueue('STOP');

$consumer->join();

sub _logProducer {

  for (1..10) {
    $logQ->enqueue(sprintf('[%-17s:%04i]', time(), threads->tid()));
    nanosleep(int(rand(10_000)));
  }

  return;
}

sub _logConsumer {

  open(my $fh, '>', '/tmp/atomic.log') 
     or die 'Cannot open log file, stopped';

  my $running = 1;
  while($running) {
    if (defined(my $item = $logQ->dequeue())) {
      $running = undef if $item eq 'STOP';
      print $fh "$item\n";
    }
  }
  close($fh);

  return;
}

The output of /tmp/atomic.log looks like:

[1526568957.83079 :0011]
[1526568957.8308  :0006]
[1526568957.83081 :0009]
[1526568957.83082 :0010]
[1526568957.83087 :0008]
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.