#Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your assignments (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

#Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you wish to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.

Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your assignments (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you wish to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.

#Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your affectations (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

#Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you wish to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.

#Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your assignments (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

#Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you wish to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.

#Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your affectations (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

#Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you which to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.

#Coding style

1. get consistent: most of your functions and variable names are lower_snake_case except direcToData (why not directToData?), you should stick to lower_snake_case which is recommended by PEP8.

• get consistent: you should have spacing around all your affectations (=).
• use except ValueError as e:, the form using a coma is being deprecated and a syntax error in Python 3;
• use pass instead of {} to create a no-op statement;
• try to be consistent: indent with 4 spaces anywhere, not 9 as in your except;
• myloop (why not my_loop?) is set to True and never changed, thus if myloop is unnecessary;
• two blank lines better separate functions and classes visually. But within the same block it feels weird, you should remove one between pool = and direcToData =.

Oh, and you use both ' and " as string delimiter. You should try to use only one for consistency.

#Directory traversal

You may be able to simplify your file management using os.walk. If "/FullData/RowData" recursively contains only the files you wish to process, then you can write:

for path, dirnames, filenames in os.walk('/FullData/RowData'):
    if filenames: # files exist in traversed directory
        full_names = map(lambda f, path=path: os.path.join(path, f), filenames)
        pool.map(process_file, full_names, 16) # Using @holroy advices

The process_file function would then be:

def process_file(filename):
    producer = SimpleProducer(client)
    with open(filename) as source_file:
        for line in source_file:
            try:
                jrec = json.loads(line.strip())
                producer.send_messages('twitter2613', json.dumps(jrec))
            except ValueError: # `as` nothing since we don't use the value
                pass

This will also help process things faster since it will create only one producer per file instead of one per line.