0
\$\begingroup\$

Is this efficient way to split the record which is comes from database? So that the thread will process the record based on the start and end points.

Step 1: Get the total number of records from the Database. ie (Select count(*) from employee) eg (example: totalRecordsFromDB=121);
Step 2 :Get the id from database ie Select empid from employee;
Step 3 :Equally share the data to thread (example: totalThreadSize=3)
Step 4 :Calculate the start/end point and trigger the thread.
Step 5 : Based on the start/end I will trigger the thread. ie (select * from employee where empid >=start and empid<=end)

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * @author dhava
 *
 */
public class CoreLogic {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub

        int totalRecordsFromDB=121;
        int totalThreadSize=3;
        int spanStart=1;
        int spanLimit=totalRecordsFromDB/totalThreadSize;
        int counter=1;
        Map<String,List<Integer>> result=new HashMap<String,List<Integer>>();
        List<Integer> sublist=new ArrayList<Integer>();
        int totalRecordForSpan=1;
        int startNode=0;
        int endNode=0;
        int spliter=1;

        //while(rs.next())
        for(int i=1;i<=totalRecordsFromDB;i++) {

            if(spanStart==totalRecordForSpan) {
                startNode=i;
                sublist=new ArrayList<Integer>();
                sublist.add(startNode);
            }
            else if(spanLimit==totalRecordForSpan && spliter!=totalThreadSize) {
                endNode=i;

                totalRecordForSpan=0;
                sublist.add(endNode);
                result.put("Split_"+spliter,sublist);
                spliter++;
                // <<start the thread >>
            }
            else if(totalRecordsFromDB==counter) {
                endNode=i;
                sublist.add(endNode);
                result.put("Split_"+spliter,sublist);
                //<< start the thread >>


            }
            totalRecordForSpan++;
            counter++;
        }
            System.out.println(result);
    }

}

The given below is output.(start/end point in arraylist)

{Split_3=[81, 121], Split_2=[41, 80], Split_1=[1, 40]}

Please refer the given below queries.

  1. Is this design/code good? - I have thread Limit.
  2. Is any other approach or design for better performance?
  3. Is there any way in spring-batch to configure the thread?

Note: I am going to handle the huge volume of data with limited number of thread.

\$\endgroup\$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.