1
\$\begingroup\$

I am collecting tweets using a Python script and I want to store them in a MongoDB database.

I want to find out the lowest tweet id and for each tweet, I want to change the id field to an _id field.

Is it possible not having to iterate over all the tweets with a for loop in order to do these 2 things.

search_result = #search result from Twitter API

min_id = float('inf')

result = []
for tweet in search_result:
    if tweet['id'] < min_id:
        min_id = tweet['id']
    tweet['_id'] = tweet.pop('id')
    result.append(tweet)
\$\endgroup\$
  • 1
    \$\begingroup\$ As I said on your StackOverflow question, "Is there any way I optimize" is way too vague of a question. Do you want it to be faster, take less memory, be more maintainable, or some other metric? If it's speed you're worried about, which part is too slow? How much data do you have? How are you data stored? \$\endgroup\$ – abarnert Nov 24 '14 at 20:55
  • \$\begingroup\$ min_id would likely decrease as you iterate through search_result. Is that intentional? \$\endgroup\$ – 200_success Nov 24 '14 at 21:04
  • \$\begingroup\$ Yes, because I want to find the lowest id \$\endgroup\$ – JNevens Nov 24 '14 at 21:31
  • \$\begingroup\$ Why do you want to rename "id" to "_id"? Are you using this to find out which tweets you have already "processed"? \$\endgroup\$ – Nihathrael Nov 25 '14 at 8:57
  • \$\begingroup\$ Because Google tells us that MongoDB uses _id as the primary key for documents. Without the unnecessary result/result.append this is fine. \$\endgroup\$ – ferada Nov 29 '14 at 23:37
3
\$\begingroup\$

Three parts to your questions: 1) find lowest tweet ID and 2) assign id: to _id 3) can (1) and (2) be done without iterating over the list.

To answer (3) first, (1) definitely can be done without iteration, but I am unaware of how to do (2) without iteration.

(1) Lowest Tweet ID The sort function is great for this. This question has more on sorting. To create a new list with the lowest id in the first position:

results = sorted(search_results, key=lambda x: x.id)

To do the same thing replacing the list:

search_results.sort(search_results, key=lambda x: x.id)

Simply access the "0"th result in the list and you have your lowest tweet id.

2) Assign id: to _id

Perhaps someone can answer this better, but the best way I know of is how you've done it:

for tweet in search_results:
    tweet['_id'] = tweet.pop('id')

Your final code would look something like:

search_result = #search result from Twitter API
results = sorted(search_results, key=lambda x: x.id)
min_id = results[0]['id']
for tweet in search_results:
    tweet['_id'] = tweet.pop('id')
\$\endgroup\$
  • \$\begingroup\$ So using sort instead of iterating through the list would not gain me anything, because I still have to iterate through the list to change the id field. \$\endgroup\$ – JNevens Nov 25 '14 at 9:02
  • \$\begingroup\$ sort likely has a performance increase as it only accesses the elements in the array once, while iterating through accesses each element at least once on top of checking that element against a variable, and assign several of those to that variable. \$\endgroup\$ – Jim Nov 25 '14 at 15:32
  • \$\begingroup\$ Do you mean sorted instead of sort? I only know of the builtin sorted and list.sort. \$\endgroup\$ – sebix Nov 28 '14 at 17:01
  • \$\begingroup\$ Your right @sebix. I corrected the error. \$\endgroup\$ – Jim Nov 29 '14 at 23:28
  • \$\begingroup\$ So now it's n*log(n), before it was linear, that's not an improvement. And an empty result is not handled. \$\endgroup\$ – ferada Nov 29 '14 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.