Run 'N' Number of PHP Scripts on shell forever in a loop

Question

My objective is to run some 'N' number of PHP Scripts in cron.

Since these are web scrapers coded in PHP and hits database on my server, each script runs for like 1 hour and they take a lot of CPU and memory. That's why I want to run one script at a time.

My first approach was to run each script at specific hour of the day but then there are only 24 hours in a day, so I can't really go with that way.

I have prepared a Unix script which runs in an infinite loop and runs the entire script. Assuming the shell will execute one script, wait for some time and then execute the next script. This is my script 1, say.

Then there is another script, Script 2 which runs every 3 hours and checks if Script 1 is running. If Script 1 is not running it will execute Script 1.

My concerns:

1. Is it feasible to run like 100 PHP scripts from one Unix script? If one fails, will it fail the whole script?
2. Is there any better approach?

Script 1 (MasterCronCarlos.sh):

#!/bin/bash
# Description = Script to check cpu usage and if above threshold issue a linux based command.
# This Script only checks "user cpu time (us - userspace (doing userspace stuff))
#


SUBJECT="CRON Status | Jai Carlos"
TO="myemail@gmail.com"

while true
do

echo -en "Starting fresh loop of cron" | mail -s "$SUBJECT" "$TO"


    php /home/site1/public_html/scrapperM1.php
    sleep 300
    php /home/site1/public_html/scrapperM2.php
    sleep 300
    php /home/site1/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site1.com" | mail -s "$SUBJECT" "$TO"

    php /home/site2/public_html/scrapperM1.php
    sleep 300
    php /home/site2/public_html/scrapperM2.php
    sleep 300
    php /home/site2/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site2.tv" | mail -s "$SUBJECT" "$TO"

    php /home/site3/public_html/scrapperM1.php
    sleep 300
    php /home/site3/public_html/scrapperM2.php
    sleep 300
    php /home/site3/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site3" | mail -s "$SUBJECT" "$TO"

    php /home/site4/public_html/scrapperM1.php
    sleep 300
    php /home/site4/public_html/scrapperM2.php
    sleep 300
    php /home/site4/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site4" | mail -s "$SUBJECT" "$TO"

    php /home/site5/public_html/scrapperM1.php
    sleep 300
    php /home/site5/public_html/scrapperM2.php
    sleep 300
    php /home/site5/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site5" | mail -s "$SUBJECT" "$TO"

    php /home/site6/public_html/scrapperM1.php
    sleep 300
    php /home/site6/public_html/scrapperM2.php
    sleep 300
    php /home/site6/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site6" | mail -s "$SUBJECT" "$TO"

    php /home/site7/public_html/scrapperM1.php
    sleep 300
    php /home/site7/public_html/scrapperM2.php
    sleep 300
    php /home/site7/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site7" | mail -s "$SUBJECT" "$TO"

done

Script 2 (mastercron.sh):

#!/bin/sh

RESULT=`ps axf | grep "MasterCronCarlos.sh" | grep -v grep | awk '{print  $1}'`

if [ $RESULT -ge 0 ]; then

echo "Running"

else

echo "Not Running"

sh /home/backups/MasterCronCarlos.sh

fi;

Answer 1

I think more than even a code review, what you are really needing to think about is your architecture. In the modern age of event-based workflows and parallel computing should you really be managing and executing a complex set of scripts in this manner (i.e. via shell automation)?

My advice is to look for better tooling that will allow you to orchestrate your pieces of logic via an automated workflow of some sort. Even consider things like map reduce tools (Hadoop, Elastic Map Reduce, etc.) which are ideal for parallel workload execution with aggregation of results.

Answer 2

As said by another comment, your question is more of an architectural question than code review.

I will still answer. Yes it is possible to run an enormous number of scripts on a machine. Make sure that your machine can handle the the work.

$ w

on linux will show you recent load average, if the averages are above 1 then you might upgrade your hardware.

1. If you can't upgrade your hardware - distribute your work to multiple machines. Using Digital Ocean or AWS you can create multiple machines that do the job of running the scripts. At the same time scale your database to multiple machines as well, you can set-up a master-slave or a master-master.

2. If you are stuck on one machine, queue your scripts. You can store your script paths into a no-sql database like redis. Every time you run a job, either remove it from the queue, or place it that the end of the queue.

   function runJobs(){ $jobs <- getJobs(); while($jobs->current() != null){ $jobs->current()->run(); $jobs->next(); } }