1
\$\begingroup\$

I am collecting tweets using a Python script and I want to store them in a MongoDB database.

I want to find out the lowest tweet id and for each tweet, I want to change the id field to an _id field.

Is it possible not having to iterate over all the tweets with a for loop in order to do these 2 things.

search_result = #search result from Twitter API

min_id = float('inf')

result = []
for tweet in search_result:
    if tweet['id'] < min_id:
        min_id = tweet['id']
    tweet['_id'] = tweet.pop('id')
    result.append(tweet)
\$\endgroup\$
5
  • 1
    \$\begingroup\$ As I said on your StackOverflow question, "Is there any way I optimize" is way too vague of a question. Do you want it to be faster, take less memory, be more maintainable, or some other metric? If it's speed you're worried about, which part is too slow? How much data do you have? How are you data stored? \$\endgroup\$
    – abarnert
    Nov 24, 2014 at 20:55
  • \$\begingroup\$ min_id would likely decrease as you iterate through search_result. Is that intentional? \$\endgroup\$ Nov 24, 2014 at 21:04
  • \$\begingroup\$ Yes, because I want to find the lowest id \$\endgroup\$
    – JNevens
    Nov 24, 2014 at 21:31
  • \$\begingroup\$ Why do you want to rename "id" to "_id"? Are you using this to find out which tweets you have already "processed"? \$\endgroup\$
    – Nihathrael
    Nov 25, 2014 at 8:57
  • \$\begingroup\$ Because Google tells us that MongoDB uses _id as the primary key for documents. Without the unnecessary result/result.append this is fine. \$\endgroup\$
    – ferada
    Nov 29, 2014 at 23:37

1 Answer 1

3
\$\begingroup\$

Three parts to your questions: 1) find lowest tweet ID and 2) assign id: to _id 3) can (1) and (2) be done without iterating over the list.

To answer (3) first, (1) definitely can be done without iteration, but I am unaware of how to do (2) without iteration.

(1) Lowest Tweet ID The sort function is great for this. This question has more on sorting. To create a new list with the lowest id in the first position:

results = sorted(search_results, key=lambda x: x.id)

To do the same thing replacing the list:

search_results.sort(search_results, key=lambda x: x.id)

Simply access the "0"th result in the list and you have your lowest tweet id.

2) Assign id: to _id

Perhaps someone can answer this better, but the best way I know of is how you've done it:

for tweet in search_results:
    tweet['_id'] = tweet.pop('id')

Your final code would look something like:

search_result = #search result from Twitter API
results = sorted(search_results, key=lambda x: x.id)
min_id = results[0]['id']
for tweet in search_results:
    tweet['_id'] = tweet.pop('id')
\$\endgroup\$
6
  • \$\begingroup\$ So using sort instead of iterating through the list would not gain me anything, because I still have to iterate through the list to change the id field. \$\endgroup\$
    – JNevens
    Nov 25, 2014 at 9:02
  • \$\begingroup\$ sort likely has a performance increase as it only accesses the elements in the array once, while iterating through accesses each element at least once on top of checking that element against a variable, and assign several of those to that variable. \$\endgroup\$
    – Jim
    Nov 25, 2014 at 15:32
  • \$\begingroup\$ Do you mean sorted instead of sort? I only know of the builtin sorted and list.sort. \$\endgroup\$
    – sebix
    Nov 28, 2014 at 17:01
  • \$\begingroup\$ Your right @sebix. I corrected the error. \$\endgroup\$
    – Jim
    Nov 29, 2014 at 23:28
  • \$\begingroup\$ So now it's n*log(n), before it was linear, that's not an improvement. And an empty result is not handled. \$\endgroup\$
    – ferada
    Nov 29, 2014 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.