Writing a log file with threads

Question

Here are all the threads write the log file. I want to know if the below script is fine, if it needs any modifications, and whether I am utilising threads properly to execute this code quickly.

open(LOG,"+>","log.txt")or die "cant open the file";


my $file_total= scalar @listOfFiles;

    my $div = $file_total/$thread_count;
    $div = ceil($div);
    my $start = 0;
    my $end = $div;

    for ($count = 1; $count <=$thread_count ; $count++) 
    {
        my $thread = threads->new(\&register, $start, $end );
        push(@threads,$thread);        
        $start = $end;
        $end = $end + $div;
        if ($end > $file_total)
        {
            $end = $file_total;
        }
    }

    foreach (@threads) 
    {
       $_->join;
    }
    close(LOG);
    sub register
    {
         my $lstart = shift;
         my $lend = shift;

         for (my $index = $lstart; $index < $lend; ++$index) 
         {
             print LOG $listOfFiles[$index];
             process......
             ..........
             .......
         }
     }

---

process... placeholder: I have a huge number of documents, say, 1 million. I am collecting all the 1 million filenames, storing them in the listOfFiles array, and dividing the array index based on the number of threads the user wants. I am passing every thread and the start and end indices to fetch the document name, process the document, and store the status of the output to a log file.

Answer 1

You can atomically write to a file if you are working with a threadsafe queue.

#!/usr/bin/env perl

use warnings FATAL => 'all';
use strict;
use Time::HiRes qw(time nanosleep);
use threads;
use Thread::Queue;

my $logQ = Thread::Queue->new();

my $consumer = threads->new(\&_logConsumer);

my @producer;
push(@producer, threads->new(\&_logProducer)) for (1..10);
$_->join() foreach @producer;

$logQ->enqueue('STOP');

$consumer->join();

sub _logProducer {

  for (1..10) {
    $logQ->enqueue(sprintf('[%-17s:%04i]', time(), threads->tid()));
    nanosleep(int(rand(10_000)));
  }

  return;
}

sub _logConsumer {

  open(my $fh, '>', '/tmp/atomic.log') 
     or die 'Cannot open log file, stopped';

  my $running = 1;
  while($running) {
    if (defined(my $item = $logQ->dequeue())) {
      $running = undef if $item eq 'STOP';
      print $fh "$item\n";
    }
  }
  close($fh);

  return;
}

The output of /tmp/atomic.log looks like:

[1526568957.83079 :0011]
[1526568957.8308  :0006]
[1526568957.83081 :0009]
[1526568957.83082 :0010]
[1526568957.83087 :0008]