Bash script to process Kafka messages

Question

I want to use shell scripts to consume kafka messages and return to status. I found that my script wasn't efficient enough.

Can I write this script better?

I want to output kafka-console-consumer.sh execution time, how do I write it?

The input result can be kafka_Check|consumer:0|consumer_time:0.3s

#!/bin/bash
result=""
vesrion=`ps -ef|grep -v grep|grep -o "kafka.\{,20\}[0-9]-sources.jar"|awk -F '-' '{print $2}'|cut -d . -f 1,-3|tr -d .`
OUT_FLAG() {
        local check_value=$1
        local result_value=$2
        if [ "${check_value}" -eq 0 ];then
             result="$result|${result_value}:0"
        else
             result="$result|${result_value}:1"
        fi
}

get_kafka_home(){
        local kafka_home_count=`ps -ef|grep kafka-tool|grep -v grep| wc -l`
        if [ $kafka_home_count -ge 1 ];then
        kafka_home=`ps -ef|grep kafka\.kafka|grep -v grep | awk -F '/bin/../libs' '{print $2}'|awk -F ':' '{print $2}'`
        else
          echo "could not found kafka_home."
          exit 1
        fi
}
consumer(){
        local consumer_mes=0
        TOPIC=`$kafka_home/bin/kafka-topics.sh --zookeeper localhost:2181 --list|grep -v "__consumer_offsets"|grep -m 1 "[a-zA-Z]"`
        if [ ${vesrion} -gt 0102 ]; then
                timeout -k 3s 3s $kafka_home/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic $TOPIC --from-beginning --max-messages 1 &>/dev/null
                consumer_mes=$?
                OUT_FLAG ${consumer_mes} "consumer_flag"
        else
                timeout -k 3s 3s $kafka_home/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic $TOPIC --from-beginning --new-consumer --max-messages 1 &>/dev/null
                consumer_mes=$?
                OUT_FLAG ${consumer_mes} "consumer_flag"
        fi

}

main(){
        get_kafka_home
        config_file="$kafka_home/config/server.properties"
        if [[ $(ss -ntupl|grep -w '9092' &>/dev/null) -ne 0 ]]&&[[ $(ss -ntupl|grep -w '2181' &>/dev/null) -ne 0 ]]
            then
                exit 1
        else
            consumer
        fi
        echo "Kafka_Check$result"
}

main

Answer 1

I'm not good enough with shell programming to really dig into your code, but I have some suggestions nonetheless!

• Make sure you know where the inefficiency is.
  There's no bennefit from optimizing stuff that isn't the bottleneck. For example, if most of the time that your script takes to run is spent running the stuff from $kafka_home/bin/, then optimizing your own code won't speed things up.
• Avoid redundancy.
  You're calling ps -ef three times. Can't you just call ps -ef|grep -v grep once and save it to a variable for use in subsequent commands?
• Consider swapping in more modern tools.
  I don't understand what you're doing well enough to question your choice to use bash, but grep looks like low hanging fruit. Several alternatives exist; I've heard ripgrep is the best.
• Consider using $() instead of ``.