Given a list of tuples of the form,
(a, b, c), is there a more direct or optimized for calculating the average of all the
PySpark? Below is what I have, but I feel like there is a more direct/optimized approach?
Classic movie recommendation example, where each tuple is
(userID, movieID, rating). How do we get the average of all of the ratings in a direct/optimized fashion?
ds_movie = sc.parallelize([(1,1,2.25), (1,2,3.0), (2,1,4.5)]) total = (ds_movie .map(lambda (userid, movieid, rating): rating) .reduce(lambda x, y: x + y)) num = ds_movie.count() average = total / num # in this example, average = 3.25