8
\$\begingroup\$

Given a list of tuples of the form, (a, b, c), is there a more direct or optimized for calculating the average of all the c's with PySpark? Below is what I have, but I feel like there is a more direct/optimized approach?

Classic movie recommendation example, where each tuple is (userID, movieID, rating). How do we get the average of all of the ratings in a direct/optimized fashion?

ds_movie = sc.parallelize([(1,1,2.25), (1,2,3.0), (2,1,4.5)])
total = (ds_movie
         .map(lambda (userid, movieid, rating): rating)
         .reduce(lambda x, y: x + y))
num = ds_movie.count()
average = total / num
# in this example, average = 3.25
\$\endgroup\$
  • \$\begingroup\$ average for all c? i think your answer is the fastest unless you want to calculate average rating for each movie. \$\endgroup\$ – inyoot Jun 30 '15 at 17:24
6
\$\begingroup\$

I would recommend using mean method:

ds_movie.map(lambda (userid, movieid, rating): rating).mean()

It is not only more concise but should have much better numerical properties (it is using a modified version of the online algorithm).

On a side note it is better to avoid tuple parameter unpacking which has been removed in Python 3. You can check PEP-3113 for details. Instead you can use Rating class as follows:

from pyspark.mllib.recommendation import Rating

ratings = ds_movie.map(lambda xs: Rating(*xs))
ratings.map(lambda r: r.rating).mean()

indexing (arguably much uglier than unpacking):

ds_movie.map(lambda r: r[2]).mean()

or standard function instead of lambda expression (kind of verbose for such a simple use case):

def get_rating(rating):
    userid, movieid, rating = rating
    return rating

ds_movie.map(get_rating).mean()
\$\endgroup\$
0
\$\begingroup\$

It may be more efficient to use pairs:

ds_movie = sc.parallelize([(1,1,2.25), (1,2,3.0), (2,1,4.5)])
sum_ratings, num_movies = ds_movie \
    .map(lambda (userid, movieid, rating): (rating, 1)) \
    .reduce(lambda x, y: (x[0] + y[0], x[1] + y[1]))
average = sum_ratings / num_movies
\$\endgroup\$
  • 5
    \$\begingroup\$ Welcome to Code Review! You have presented an alternate solution, but haven't really explained why your suggestion is better than the original. Could you add some benchmarks or non-performance-related justification? \$\endgroup\$ – 200_success Jul 14 '15 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.