A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight IDs, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first occurrence in the dictionary. I changed it to now also check if that value has already occurred before, and if so, to skip it.

def DiscoverEarliestIndex(self, number):
    thegoodshit = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in thegoodshit.values():
                i += 1
                continue
            else:
                thegoodshit[i+1]= next
                i += 1
    else:
        return thegoodshit

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?

A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight IDs, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first occurrence in the dictionary. I changed it to now also check if that value has already occurred before, and if so, to skip it.

def DiscoverEarliestIndex(self, number):
    result = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in result.values():
                i += 1
                continue
            else:
                result[i+1]= next
                i += 1
    else:
        return result

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?

A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight ID's, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first ococurance in the dictionary. I changed it to now also check if that value has already occured before, and if so, to skip it. Here's my code:

def DiscoverEarliestIndex(self, number):
    thegoodshit = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in thegoodshit.values():
                i += 1
                continue
            else:
                thegoodshit[i+1]= next
                i += 1
    else:
        return thegoodshit

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?

A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight IDs, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first occurrence in the dictionary. I changed it to now also check if that value has already occurred before, and if so, to skip it.

def DiscoverEarliestIndex(self, number):
    thegoodshit = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in thegoodshit.values():
                i += 1
                continue
            else:
                thegoodshit[i+1]= next
                i += 1
    else:
        return thegoodshit

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?

A large csv I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight ID's, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first ococurance in the dictionary. I changed it to now also check if that value has already occured before, and if so, to skip it. Here's my code:

def DiscoverEarliestIndex(self, number):
    thegoodshit = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in thegoodshit.values():
                i += 1
                continue
            else:
                thegoodshit[i+1]= next
                i += 1
    else:
        return thegoodshit

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?

A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight ID's, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.

Dictionary = { Index: FID, ... }

This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).

Example:

20110117559515, ... 
20110117559515, ... 
20110117559515, ...                     
20110117559572, ...   
20110117559572, ...   
20110117559572, ...                               
20110117559574, ...                               
20110117559587, ...                             
20110117559588, ...

and so on for 5.3 million some rows.

Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first ococurance in the dictionary. I changed it to now also check if that value has already occured before, and if so, to skip it. Here's my code:

def DiscoverEarliestIndex(self, number):
    thegoodshit = {}
    columnvalues = self.column(number)
    column_enum = {}
    for a, b in enumerate(columnvalues):
        column_enum[a] = b
        i = 0
    while i < (len(columnvalues) - 1):
        next = column_enum[i+1]
        if columnvalues[i] == next:
            i += 1
        else:
            if next in thegoodshit.values():
                i += 1
                continue
            else:
                thegoodshit[i+1]= next
                i += 1
    else:
        return thegoodshit

It's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.

Is there a more efficient way to write this function?