Skip to main content
edited tags; edited title
Link
200_success
  • 144.2k
  • 22
  • 188
  • 473

Python script to find employees who are in one PostgreSQL data query from two tables based on a conditiontable but not in python takes too long time?another

edited title
Link

Querying PostgreSQL data query from two tabletables based on a condition in python takes too long time?

Source Link

Querying data from two table takes long time?

I'm writing a python script and do database query to get the ids of employees who are in one table but are not in another. I'm using psycopg2 module for python 2 and PostgreSQL as database. After querying from one table based on a condition, I do another query on another table to get the difference between these tables using the result of previous query. The problem is the full procedure takes a long time. I want to know is there any other method or technique which can make the entire procedure faster? Below is the code I used for doing my feature:

def find_difference_assignment_pls_count(self):
        counter = 0
        emp_ids = []
        self.test_cursor.execute("""Select id,emp_id from test_test where flag=true and emp_id is not null and ver_id in(select id from test_phases where state='test')""")
        matching_records = self.test_cursor.fetchall()

        for match_record in matching_records:
            self.test_cursor.execute("""Select id from test_table where test_emp_id=%s and state='test_state'""",(match_record['emp_id'],))
            result = self.test_cursor.fetchall()

            if result:
                continue
            else:
                emp_ids.append(match_record['emp_id'])
                counter +=1

        print "Employees of test_test not in test_table: ", counter
        return emp_ids

I run these queries on two tables which at least have more than 500000 records. Therefore the performance is really slow.