3
\$\begingroup\$

My objective is to run some 'N' number of PHP Scripts in cron.

Since these are web scrapers coded in PHP and hits database on my server, each script runs for like 1 hour and they take a lot of CPU and memory. That's why I want to run one script at a time.

My first approach was to run each script at specific hour of the day but then there are only 24 hours in a day, so I can't really go with that way.

I have prepared a Unix script which runs in an infinite loop and runs the entire script. Assuming the shell will execute one script, wait for some time and then execute the next script. This is my script 1, say.

Then there is another script, Script 2 which runs every 3 hours and checks if Script 1 is running. If Script 1 is not running it will execute Script 1.

My concerns:

  1. Is it feasible to run like 100 PHP scripts from one Unix script? If one fails, will it fail the whole script?
  2. Is there any better approach?

Script 1 (MasterCronCarlos.sh):

#!/bin/bash
# Description = Script to check cpu usage and if above threshold issue a linux based command.
# This Script only checks "user cpu time (us - userspace (doing userspace stuff))
#


SUBJECT="CRON Status | Jai Carlos"
TO="myemail@gmail.com"

while true
do

echo -en "Starting fresh loop of cron" | mail -s "$SUBJECT" "$TO"


    php /home/site1/public_html/scrapperM1.php
    sleep 300
    php /home/site1/public_html/scrapperM2.php
    sleep 300
    php /home/site1/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site1.com" | mail -s "$SUBJECT" "$TO"

    php /home/site2/public_html/scrapperM1.php
    sleep 300
    php /home/site2/public_html/scrapperM2.php
    sleep 300
    php /home/site2/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site2.tv" | mail -s "$SUBJECT" "$TO"

    php /home/site3/public_html/scrapperM1.php
    sleep 300
    php /home/site3/public_html/scrapperM2.php
    sleep 300
    php /home/site3/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site3" | mail -s "$SUBJECT" "$TO"

    php /home/site4/public_html/scrapperM1.php
    sleep 300
    php /home/site4/public_html/scrapperM2.php
    sleep 300
    php /home/site4/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site4" | mail -s "$SUBJECT" "$TO"

    php /home/site5/public_html/scrapperM1.php
    sleep 300
    php /home/site5/public_html/scrapperM2.php
    sleep 300
    php /home/site5/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site5" | mail -s "$SUBJECT" "$TO"

    php /home/site6/public_html/scrapperM1.php
    sleep 300
    php /home/site6/public_html/scrapperM2.php
    sleep 300
    php /home/site6/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site6" | mail -s "$SUBJECT" "$TO"

    php /home/site7/public_html/scrapperM1.php
    sleep 300
    php /home/site7/public_html/scrapperM2.php
    sleep 300
    php /home/site7/public_html/scrapperM3.php
    sleep 300

    echo -en "Script completed for site7" | mail -s "$SUBJECT" "$TO"

done

Script 2 (mastercron.sh):

#!/bin/sh

RESULT=`ps axf | grep "MasterCronCarlos.sh" | grep -v grep | awk '{print  $1}'`

if [ $RESULT -ge 0 ]; then

echo "Running"

else

echo "Not Running"

sh /home/backups/MasterCronCarlos.sh

fi;
\$\endgroup\$
10
  • \$\begingroup\$ Is there a reason you go this way instead of cron jobs? \$\endgroup\$
    – Mast
    Aug 20, 2017 at 19:38
  • \$\begingroup\$ @Mast Hey, The reason is, there are 105 jobs in total and running then on fixed time is not feasible. I need to run them as much as possible. If i assume one script takes 1 hour and run all 105 on 1 hour gap. I need 4 days. But in real some jobs takes 1 hour and sometimes they take 15 mins to completed. Depending on the data they are scrapping. \$\endgroup\$ Aug 20, 2017 at 19:41
  • \$\begingroup\$ @Mast Can you suggest me a better way ? \$\endgroup\$ Aug 20, 2017 at 19:52
  • \$\begingroup\$ Why can't 2 jobs run at the same time? If the jobs take this long, shouldn't you simply optimize the longest running ones so you no longer have this problem? If they're scraping, likely the host on the other site is (at least partially) limiting the throughput. Running multiple jobs at the same time would be more efficient in that case. Your current approach looks pretty risky. \$\endgroup\$
    – Mast
    Aug 20, 2017 at 20:05
  • \$\begingroup\$ The job(s) are already optimized to maximum. And since it is taking a lot of CPU and Memory from ram, i don't want to run 2 scripts at a time. The requirement here is to run one script and then free the resources and then run another. And it should repeat the whole process again and again. \$\endgroup\$ Aug 21, 2017 at 7:15

2 Answers 2

1
\$\begingroup\$

I think more than even a code review, what you are really needing to think about is your architecture. In the modern age of event-based workflows and parallel computing should you really be managing and executing a complex set of scripts in this manner (i.e. via shell automation)?

My advice is to look for better tooling that will allow you to orchestrate your pieces of logic via an automated workflow of some sort. Even consider things like map reduce tools (Hadoop, Elastic Map Reduce, etc.) which are ideal for parallel workload execution with aggregation of results.

\$\endgroup\$
1
\$\begingroup\$

As said by another comment, your question is more of an architectural question than code review.

I will still answer. Yes it is possible to run an enormous number of scripts on a machine. Make sure that your machine can handle the the work.

$ w

on linux will show you recent load average, if the averages are above 1 then you might upgrade your hardware.

  1. If you can't upgrade your hardware - distribute your work to multiple machines. Using Digital Ocean or AWS you can create multiple machines that do the job of running the scripts. At the same time scale your database to multiple machines as well, you can set-up a master-slave or a master-master.

  2. If you are stuck on one machine, queue your scripts. You can store your script paths into a no-sql database like redis. Every time you run a job, either remove it from the queue, or place it that the end of the queue.

    function runJobs(){ $jobs <- getJobs(); while($jobs->current() != null){ $jobs->current()->run(); $jobs->next(); } }

\$\endgroup\$
2
  • \$\begingroup\$ Thanks ! I have followed the 2nd approach only. Running one script at a time and maintaining the status. \$\endgroup\$ Sep 6, 2017 at 5:09
  • \$\begingroup\$ @TusharKesarwani sounds good - please accept this as the answer \$\endgroup\$ Sep 6, 2017 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.