Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

Filter by
Sorted by
Tagged with
4 votes
0 answers
429 views

Subclass of Python's multiprocessing.Pool which allows progress reporting

For context, the whole of the project code can be found here. This question was created specifically for the progress.py file. The goal behind it is to allow ...
gabe appleton's user avatar
4 votes
2 answers
53 views

Put every object in a specified bucket

I have this array of objects: ...
Katherine Maurus's user avatar
0 votes
2 answers
5k views

Javascript + Filter object of values

I have the object with values. I trying to filter based on values. ...
RSKMR's user avatar
  • 101
2 votes
1 answer
197 views

The best way for inserting multiple objects into array

I have a transformer helper function. It reduces over the array and transform key/value pairs. At the end of the loop there is the key 'EXAMPLE1' exists and I should insert two objects after the first ...
user215631's user avatar
2 votes
1 answer
57 views

Efficiently aggregating nested data

Problem Given the following data: ...
Dominic Campos's user avatar
1 vote
2 answers
138 views

Extract unique words from given text and group by letter count

The task is for training go-lang. The idea is to extract unique words sorted and grouped by length. Might be useful in learning new words. The program uses command line argument assuming it's a file ...
triclosan's user avatar
  • 254
2 votes
1 answer
61 views

Looping over a bidimensional array and extract data to a new one

I have a bidimensional array like this: ...
robe007's user avatar
  • 123
2 votes
1 answer
65 views

Building objects in javascript, without "if(!a[k]) a[k] = []"

When building objects using reduce, I often have crappy code like this: ...
Sophie McCarrell's user avatar
2 votes
0 answers
319 views

Groovy Map/Reduce for Jenkins DSL

Jenkins DSL doesn't support collect and inject from what I can tell (I get missing method exceptions when I try), so I ...
RubberDuck's user avatar
  • 30.8k
3 votes
2 answers
778 views

Summarizing the score of a personality quiz

This function takes a list of questions and list of answers provided by the user. The list of answers is always a list of booleans (for true and false) and the list of questions takes the following ...
Calin Leafshade's user avatar
6 votes
0 answers
80 views

Pyspark Solver for Tiered Board Games

I've written a Pyspark program that will completely solve a tiered board game (no loops, each game position is a member of only one tier) and writes each tier to a file. It also determines the ...
Michael's user avatar
  • 61
8 votes
3 answers
1k views

Find Top 10 IP out of more than 5GB data

I have a few of files, and total size of them is more than 5 GB. Each line of the files is a IP address, looks like: 127.0.0.1 reset success ... 127.0.0.2 reset success how can i find Top10 ...
Cherlex's user avatar
  • 145
3 votes
1 answer
104 views

Classifying and counting database entries using Scala map and flatMap

I am new to Spark and Scala and I have solved the following problem. I have a table in database with following structure: ...
Shams Tabraiz Alam's user avatar
5 votes
2 answers
651 views

Accepting user defined functions for custom map reduce functionality in C++

I am implementing map and reduce - style functions for processing geospatial raster datasets. I would like the ...
jramm's user avatar
  • 133
4 votes
1 answer
1k views

Finding the similarity between the two movies using Pearson correlation coefficient

I am trying to find the similarity between the two movies using Pearson correlation coefficient. The programs is working well for small inputs but for large inputs (like 100000 lines) it takes forever....
Praveen Allam's user avatar
15 votes
1 answer
36k views

Generic “reduceBy” or “groupBy + aggregate” functionality with Spark DataFrame

Maybe I totally reinvented the wheel, or maybe I've invented something new and useful. Can one of you tell me if there's a better way of doing this? Here's what I'm trying to do: I want a generic <...
Nathaniel's user avatar
  • 253
4 votes
1 answer
333 views

MongoDB - Find records in one collection that match string in 2nd collection

I have a MongoDB collection of product IDs/unique products attributes and a second collection of codes that relate to attributes common for products whose IDs are prefixed with those codes. For ...
getglad's user avatar
  • 229
5 votes
1 answer
3k views

Weighted Probability Problem in Swift

I was asked a weighted probability question in a technical interview a few months ago that went something like this: Given an input of colors and an integer "weight" value, randomly return a color ...
JAL's user avatar
  • 571
3 votes
3 answers
528 views

MapReduce in Erlang

I'm doing a comparison of Erlang, Haskell, Elixir and ES6, and I'm less farmiliar with Erlang and Elixir, but I want to represent all of these languages fairly, so is this good Erlang code? ...
Alexis Purslane's user avatar
8 votes
2 answers
3k views

Average movie rankings

Given a list of tuples of the form, (a, b, c), is there a more direct or optimized for calculating the average of all the c's ...
JasonAizkalns's user avatar
2 votes
1 answer
314 views

Mergesort using map-reduce, multithreads, buffers and condition variables

I wrote a map reduce program which uses multi threads, bounded buffers, condition variables. It works perfectly for some types of inputs. In the program there are N mappers, R reducers, 1 merger. ...
user4751640's user avatar
2 votes
1 answer
207 views

CSMR for large-scale text-prcessing

I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating ...
IrishDog's user avatar
  • 131
5 votes
2 answers
588 views

Map reduce tester ported from bash to Python

My MapReduce tester is clearly ported from Shell, short of args=None for line in args or read_input(), what's a better way of ...
A T's user avatar
  • 485
2 votes
1 answer
161 views

MapReduce program for finding low value hashes

For class we were to make a MapReduce program in Python to find low value hashes of our name. I have completed the assignment but want to try and speed it up. The program currently takes about 45s to ...
KDecker's user avatar
  • 225
8 votes
1 answer
28k views

Calculate min, max, average, and variance on a large dataset

I got a piece of Java code using Hadoop to calculate min, max, average and variance on a large dataset made of (index value) couples separated by a newline: ...
Agostino's user avatar
  • 283
3 votes
1 answer
97 views

Customer MapReduce implementation

I would love a second opinion / another pair of eyes on this. ...
ThomasReggi's user avatar
8 votes
1 answer
2k views

Map-reduce implementation for splitting strings

I have been changing this code and I don't get to make it much better. I changed a little bit the structure, reimplemeted a new function for splitting Strings which is more efficient, etc. I have been ...
Guille's user avatar
  • 183