You've given us a small piece of your problem to review, and Janne's suggestion seems reasonable if that piece is considered on its own. But I have the feeling that this isn't the only bit of analysis that you are doing on your data, and if so, you probably want to think about using a proper database.

Python comes with a built-in relational database engine in the form of the sqlite3 module. So you could easily read your CSV directly into a SQLite table, either using the .import command in SQLite's command-line shell, or via Python if you need more preprocessing:

import sqlite3
import csv

def load_flight_csv(db_filename, csv_filename):
    """
    Load flight data from `csv_filename` into the SQLite database in
    `db_filename`.
    """
    with sqlite3.connect(db_filename) as conn, open(csv_filename, 'rb') as f:
        c = conn.cursor()
        c.execute('''CREATE TABLE IF NOT EXISTS flight
                     (id INTEGER PRIMARY KEY AUTOINCREMENT, fid TEXT)''')
        c.execute('''CREATE INDEX IF NOT EXISTS flight_fid ON flight (fid)''')
        c.executemany('''INSERT INTO flight (fid) VALUES (?)''', csv.reader(f))
        conn.commit()

(Obviously you'd need more fields in the CREATE TABLE statement, but since you didn't show them in your question, I can't guess what they might be.)

And then you can analyze the data by issuing SQL queries:

>>> db = 'flight.db'
>>> load_flight_csv(db, 'flight.csv')
>>> conn = sqlite3.connect(db)
>>> from pprint import pprint
>>> pprint(conn.execute('''SELECT MIN(id), fid FROM flight GROUP BY fid''').fetchall())
[(1, u'20110117559515'),
 (4, u'20110117559572'),
 (7, u'20110117559574'),
 (8, u'20110117559587'),
 (9, u'20110117559588')]

You've given us a small piece of your problem to review, and Janne's suggestion seems reasonable if that piece is considered on its own. But I have the feeling that this isn't the only bit of analysis that you are doing on your data, and if so, you probably want to think about using a proper database.

Python comes with a built-in relational database engine in the form of the sqlite3 module. So you could easily read your CSV directly into a SQLite table, either using the .import command in SQLite's command-line shell, or via Python if you need more preprocessing:

import sqlite3
import csv

def load_flight_csv(db_filename, csv_filename):
    """
    Load flight data from `csv_filename` into the SQLite database in
    `db_filename`.
    """
    with sqlite3.connect(db_filename) as conn, open(csv_filename, 'rb') as f:
        c = conn.cursor()
        c.execute('''CREATE TABLE IF NOT EXISTS flight
                     (id INTEGER PRIMARY KEY AUTOINCREMENT, fid TEXT)''')
        c.execute('''CREATE INDEX IF NOT EXISTS flight_fid ON flight (fid)''')
        c.executemany('''INSERT INTO flight (fid) VALUES (?)''', csv.reader(f))
        conn.commit()

(Obviously you'd need more fields in the CREATE TABLE statement, but since you didn't show them in your question, I can't guess what they might be.)

And then you can analyze the data by issuing SQL queries:

>>> db = 'flight.db'
>>> load_flight_csv(db, 'flight.csv')
>>> conn = sqlite3.connect(db)
>>> from pprint import pprint
>>> pprint(conn.execute('''SELECT MIN(id), fid FROM flight GROUP BY fid''').fetchall())
[(1, u'20110117559515'),
 (4, u'20110117559572'),
 (7, u'20110117559574'),
 (8, u'20110117559587'),
 (9, u'20110117559588')]

You've given us a small piece of your problem to review, and Janne's suggestion seems reasonable if that piece is considered on its own. But I have the feeling that this isn't the only bit of analysis that you are doing on your data, and if so, you probably want to think about using a proper database.

Python comes with a built-in relational database engine in the form of the sqlite3 module. So you could easily read your CSV directly into a SQLite table, either using the .import command in SQLite's command-line shell, or via Python if you need more preprocessing:

import sqlite3
import csv

def load_flight_csv(db_filename, csv_filename):
    """
    Load flight data from `csv_filename` into the SQLite database in
    `db_filename`.
    """
    with sqlite3.connect(db_filename) as conn, open(csv_filename, 'rb') as f:
        c = conn.cursor()
        c.execute('''CREATE TABLE IF NOT EXISTS flight
                     (id INTEGER PRIMARY KEY AUTOINCREMENT, fid TEXT)''')
        c.execute('''CREATE INDEX IF NOT EXISTS flight_fid ON flight (fid)''')
        c.executemany('''INSERT INTO flight (fid) VALUES (?)''', csv.reader(f))
        conn.commit()

And then you can analyze the data by issuing SQL queries:

>>> db = 'flight.db'
>>> load_flight_csv(db, 'flight.csv')
>>> conn = sqlite3.connect(db)
>>> from pprint import pprint
>>> pprint(conn.execute('''SELECT MIN(id), fid FROM flight GROUP BY fid''').fetchall())
[(1, u'20110117559515'),
 (4, u'20110117559572'),
 (7, u'20110117559574'),
 (8, u'20110117559587'),
 (9, u'20110117559588')]

You've given us a small piece of your problem to review, and Janne's suggestion seems reasonable if that piece is considered on its own. But I have the feeling that this isn't the only bit of analysis that you are doing on your data, and if so, you probably want to think about using a proper database.

Python comes with a built-in relational database engine in the form of the sqlite3 module. So you could easily read your CSV directly into a SQLite table, either using the .import command in SQLite's command-line shell, or via Python if you need more preprocessing:

import sqlite3
import csv

def load_flight_csv(db_filename, csv_filename):
    """
    Load flight data from `csv_filename` into the SQLite database in
    `db_filename`.
    """
    with sqlite3.connect(db_filename) as conn, open(csv_filename, 'rb') as f:
        c = conn.cursor()
        c.execute('''CREATE TABLE IF NOT EXISTS flight
                     (id INTEGER PRIMARY KEY AUTOINCREMENT, fid TEXT)''')
        c.execute('''CREATE INDEX IF NOT EXISTS flight_fid ON flight (fid)''')
        c.executemany('''INSERT INTO flight (fid) VALUES (?)''', csv.reader(f))
        conn.commit()

(Obviously you'd need more fields in the CREATE TABLE statement, but since you didn't show them in your question, I can't guess what they might be.)

And then you can analyze the data by issuing SQL queries:

>>> db = 'flight.db'
>>> load_flight_csv(db, 'flight.csv')
>>> conn = sqlite3.connect(db)
>>> from pprint import pprint
>>> pprint(conn.execute('''SELECT MIN(id), fid FROM flight GROUP BY fid''').fetchall())
[(1, u'20110117559515'),
 (4, u'20110117559572'),
 (7, u'20110117559574'),
 (8, u'20110117559587'),
 (9, u'20110117559588')]