# Categorization algorithm for discrete variables

I am trying to categorize some data. For that I check the distribution of the data. Then I split based on the number of appearance of each value. The algorithm I have is working so far but really slow. I am looking to improve the speed. The speed is important on this one because I treat a lot of different data using the same structure and the data is a bit large (140k rows)

def RamsesIdCategory(data):
# handling Ramses Id:
print('Starting Ramses Id')
for value in valueRamses:
if countRamses.get(value) < 2:
elif 2 <= countRamses.get(value) < 5:
elif 5 <= countRamses.get(value) < 10:
elif 10 <= countRamses.get(value) < 20:
elif 20 <= countRamses.get(value) < 32:
else:
print('finished Ramses Id')
return data


EDIT : I reworked my code as I knew there was the problem with the loop doing too much iterations over my rows. Here is the new version :

def RamsesIdCategory(data):
# handling Ramses Id:
print('Starting Ramses Id')
for i in data.index:
elif 5 <= valueRamses.get(data.get_value(i, 'Ramses Trade Id')) < 10:
elif 10<= valueRamses.get(data.get_value(i, 'Ramses Trade Id')) < 20:
elif 20 <= valueRamses.get(data.get_value(i, 'Ramses Trade Id')) < 32:
else:
return(data)


I iterrate over my whole dataset once and do a single select and modification instead of trying to do a multiple modification on the entire dataframe for each different value. It is 100 time faster as it ran in few sec vs 50min

The code can be largely simplified using apply. But first, you need a better way to test your values and assign them an id:

def convert_count_to_id(count, limits=(2, 5, 10, 20, 32)):
for id, limit in enumerate(limits, 1):
if count < limit:
return id
return id + 1


This is equivalent to your elifs chain but harder to get wrong.

def ramses_id_category(data):

Note that I removed the return data at the end. Since you are mutating the parameter in place, there is no need to return it back since the caller will already be able to see the changes on the reference they hold when calling this function.
• How does the switch to apply affect the speed? codereview.stackexchange.com/questions/157797/… uses apply but worries about the speed. – hpaulj Mar 15 '17 at 18:10
• @Mayeulsgc Yes, and you can even change the number of limits. You can change them at the function definition or at the calling point. You can even use functools.partial to be able to change them in an apply call. – Mathias Ettinger Mar 16 '17 at 7:22