Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

Filter by
Sorted by
Tagged with
4 votes
0 answers

Subclass of Python's multiprocessing.Pool which allows progress reporting

For context, the whole of the project code can be found here. This question was created specifically for the file. The goal behind it is to allow ...
gabe appleton's user avatar
4 votes
2 answers

Put every object in a specified bucket

I have this array of objects: ...
Katherine Maurus's user avatar
0 votes
2 answers

Javascript + Filter object of values

I have the object with values. I trying to filter based on values. ...
RSKMR's user avatar
  • 101
2 votes
1 answer

The best way for inserting multiple objects into array

I have a transformer helper function. It reduces over the array and transform key/value pairs. At the end of the loop there is the key 'EXAMPLE1' exists and I should insert two objects after the first ...
user215631's user avatar
2 votes
1 answer

Efficiently aggregating nested data

Problem Given the following data: ...
Dominic Campos's user avatar
1 vote
2 answers

Extract unique words from given text and group by letter count

The task is for training go-lang. The idea is to extract unique words sorted and grouped by length. Might be useful in learning new words. The program uses command line argument assuming it's a file ...
triclosan's user avatar
  • 254
2 votes
1 answer

Looping over a bidimensional array and extract data to a new one

I have a bidimensional array like this: ...
robe007's user avatar
  • 123
2 votes
1 answer

Building objects in javascript, without "if(!a[k]) a[k] = []"

When building objects using reduce, I often have crappy code like this: ...
Sophie McCarrell's user avatar
2 votes
0 answers

Groovy Map/Reduce for Jenkins DSL

Jenkins DSL doesn't support collect and inject from what I can tell (I get missing method exceptions when I try), so I ...
RubberDuck's user avatar
  • 30.8k
3 votes
2 answers

Summarizing the score of a personality quiz

This function takes a list of questions and list of answers provided by the user. The list of answers is always a list of booleans (for true and false) and the list of questions takes the following ...
Calin Leafshade's user avatar
6 votes
0 answers

Pyspark Solver for Tiered Board Games

I've written a Pyspark program that will completely solve a tiered board game (no loops, each game position is a member of only one tier) and writes each tier to a file. It also determines the ...
Michael's user avatar
  • 61
8 votes
3 answers

Find Top 10 IP out of more than 5GB data

I have a few of files, and total size of them is more than 5 GB. Each line of the files is a IP address, looks like: reset success ... reset success how can i find Top10 ...
Cherlex's user avatar
  • 145
3 votes
1 answer

Classifying and counting database entries using Scala map and flatMap

I am new to Spark and Scala and I have solved the following problem. I have a table in database with following structure: ...
Shams Tabraiz Alam's user avatar
5 votes
2 answers

Accepting user defined functions for custom map reduce functionality in C++

I am implementing map and reduce - style functions for processing geospatial raster datasets. I would like the ...
jramm's user avatar
  • 133
4 votes
1 answer

Finding the similarity between the two movies using Pearson correlation coefficient

I am trying to find the similarity between the two movies using Pearson correlation coefficient. The programs is working well for small inputs but for large inputs (like 100000 lines) it takes forever....
Praveen Allam's user avatar
15 votes
1 answer

Generic “reduceBy” or “groupBy + aggregate” functionality with Spark DataFrame

Maybe I totally reinvented the wheel, or maybe I've invented something new and useful. Can one of you tell me if there's a better way of doing this? Here's what I'm trying to do: I want a generic <...
Nathaniel's user avatar
  • 253
4 votes
1 answer

MongoDB - Find records in one collection that match string in 2nd collection

I have a MongoDB collection of product IDs/unique products attributes and a second collection of codes that relate to attributes common for products whose IDs are prefixed with those codes. For ...
getglad's user avatar
  • 229
5 votes
1 answer

Weighted Probability Problem in Swift

I was asked a weighted probability question in a technical interview a few months ago that went something like this: Given an input of colors and an integer "weight" value, randomly return a color ...
JAL's user avatar
  • 571
3 votes
3 answers

MapReduce in Erlang

I'm doing a comparison of Erlang, Haskell, Elixir and ES6, and I'm less farmiliar with Erlang and Elixir, but I want to represent all of these languages fairly, so is this good Erlang code? ...
Alexis Purslane's user avatar
8 votes
2 answers

Average movie rankings

Given a list of tuples of the form, (a, b, c), is there a more direct or optimized for calculating the average of all the c's ...
JasonAizkalns's user avatar
2 votes
1 answer

Mergesort using map-reduce, multithreads, buffers and condition variables

I wrote a map reduce program which uses multi threads, bounded buffers, condition variables. It works perfectly for some types of inputs. In the program there are N mappers, R reducers, 1 merger. ...
user4751640's user avatar
2 votes
1 answer

CSMR for large-scale text-prcessing

I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating ...
IrishDog's user avatar
  • 131
5 votes
2 answers

Map reduce tester ported from bash to Python

My MapReduce tester is clearly ported from Shell, short of args=None for line in args or read_input(), what's a better way of ...
A T's user avatar
  • 485
2 votes
1 answer

MapReduce program for finding low value hashes

For class we were to make a MapReduce program in Python to find low value hashes of our name. I have completed the assignment but want to try and speed it up. The program currently takes about 45s to ...
KDecker's user avatar
  • 225
8 votes
1 answer

Calculate min, max, average, and variance on a large dataset

I got a piece of Java code using Hadoop to calculate min, max, average and variance on a large dataset made of (index value) couples separated by a newline: ...
Agostino's user avatar
  • 283
3 votes
1 answer

Customer MapReduce implementation

I would love a second opinion / another pair of eyes on this. ...
ThomasReggi's user avatar
8 votes
1 answer

Map-reduce implementation for splitting strings

I have been changing this code and I don't get to make it much better. I changed a little bit the structure, reimplemeted a new function for splitting Strings which is more efficient, etc. I have been ...
Guille's user avatar
  • 183