Questions tagged [mapreduce]

MapReduce is an algorithm for processing huge datasets on certain kinds of distributable problems using a large number of nodes

Filter by
Sorted by
Tagged with
4 votes
0 answers
165 views

Subclass of Python's multiprocessing.Pool which allows progress reporting

For context, the whole of the project code can be found here. This question was created specifically for the progress.py file. The goal behind it is to allow ...
user avatar
4 votes
2 answers
46 views

Put every object in a specified bucket

I have this array of objects: ...
user avatar
0 votes
2 answers
2k views

Javascript + Filter object of values

I have the object with values. I trying to filter based on values. ...
user avatar
  • 101
2 votes
1 answer
135 views

The best way for inserting multiple objects into array

I have a transformer helper function. It reduces over the array and transform key/value pairs. At the end of the loop there is the key 'EXAMPLE1' exists and I should insert two objects after the first ...
user avatar
2 votes
1 answer
53 views

Efficiently aggregating nested data

Problem Given the following data: ...
user avatar
1 vote
2 answers
108 views

Extract unique words from given text and group by letter count

The task is for training go-lang. The idea is to extract unique words sorted and grouped by length. Might be useful in learning new words. The program uses command line argument assuming it's a file ...
user avatar
  • 232
2 votes
1 answer
58 views

Looping over a bidimensional array and extract data to a new one

I have a bidimensional array like this: ...
user avatar
  • 123
2 votes
1 answer
62 views

Building objects in javascript, without "if(!a[k]) a[k] = []"

When building objects using reduce, I often have crappy code like this: ...
user avatar
2 votes
0 answers
262 views

Groovy Map/Reduce for Jenkins DSL

Jenkins DSL doesn't support collect and inject from what I can tell (I get missing method exceptions when I try), so I ...
user avatar
  • 30.7k
3 votes
2 answers
695 views

Summarizing the score of a personality quiz

This function takes a list of questions and list of answers provided by the user. The list of answers is always a list of booleans (for true and false) and the list of questions takes the following ...
user avatar
6 votes
0 answers
75 views

Pyspark Solver for Tiered Board Games

I've written a Pyspark program that will completely solve a tiered board game (no loops, each game position is a member of only one tier) and writes each tier to a file. It also determines the ...
user avatar
  • 61
8 votes
3 answers
1k views

Find Top 10 IP out of more than 5GB data

I have a few of files, and total size of them is more than 5 GB. Each line of the files is a IP address, looks like: 127.0.0.1 reset success ... 127.0.0.2 reset success how can i find Top10 ...
user avatar
  • 145
3 votes
1 answer
100 views

Classifying and counting database entries using Scala map and flatMap

I am new to Spark and Scala and I have solved the following problem. I have a table in database with following structure: ...
user avatar
5 votes
2 answers
634 views

Accepting user defined functions for custom map reduce functionality in C++

I am implementing map and reduce - style functions for processing geospatial raster datasets. I would like the ...
user avatar
  • 133
4 votes
1 answer
1k views

Finding the similarity between the two movies using Pearson correlation coefficient

I am trying to find the similarity between the two movies using Pearson correlation coefficient. The programs is working well for small inputs but for large inputs (like 100000 lines) it takes forever....
user avatar
15 votes
1 answer
35k views

Generic “reduceBy” or “groupBy + aggregate” functionality with Spark DataFrame

Maybe I totally reinvented the wheel, or maybe I've invented something new and useful. Can one of you tell me if there's a better way of doing this? Here's what I'm trying to do: I want a generic <...
user avatar
  • 253
4 votes
1 answer
324 views

MongoDB - Find records in one collection that match string in 2nd collection

I have a MongoDB collection of product IDs/unique products attributes and a second collection of codes that relate to attributes common for products whose IDs are prefixed with those codes. For ...
user avatar
  • 219
5 votes
1 answer
2k views

Weighted Probability Problem in Swift

I was asked a weighted probability question in a technical interview a few months ago that went something like this: Given an input of colors and an integer "weight" value, randomly return a color ...
user avatar
  • 571
3 votes
3 answers
505 views

MapReduce in Erlang

I'm doing a comparison of Erlang, Haskell, Elixir and ES6, and I'm less farmiliar with Erlang and Elixir, but I want to represent all of these languages fairly, so is this good Erlang code? ...
user avatar
8 votes
2 answers
3k views

Average movie rankings

Given a list of tuples of the form, (a, b, c), is there a more direct or optimized for calculating the average of all the c's ...
user avatar
2 votes
1 answer
301 views

Mergesort using map-reduce, multithreads, buffers and condition variables

I wrote a map reduce program which uses multi threads, bounded buffers, condition variables. It works perfectly for some types of inputs. In the program there are N mappers, R reducers, 1 merger. ...
user avatar
2 votes
1 answer
203 views

CSMR for large-scale text-prcessing

I'm working on a project for large-scale text-processing, which is a first implementation of the basic idea of CSMR. CSMR is an algorithm that measures the similarity between documents by calculating ...
user avatar
  • 131
5 votes
2 answers
564 views

Map reduce tester ported from bash to Python

My MapReduce tester is clearly ported from Shell, short of args=None for line in args or read_input(), what's a better way of ...
user avatar
  • 503
2 votes
1 answer
155 views

MapReduce program for finding low value hashes

For class we were to make a MapReduce program in Python to find low value hashes of our name. I have completed the assignment but want to try and speed it up. The program currently takes about 45s to ...
user avatar
  • 225
8 votes
1 answer
28k views

Calculate min, max, average, and variance on a large dataset

I got a piece of Java code using Hadoop to calculate min, max, average and variance on a large dataset made of (index value) couples separated by a newline: ...
user avatar
  • 293
3 votes
1 answer
96 views

Customer MapReduce implementation

I would love a second opinion / another pair of eyes on this. ...
user avatar
8 votes
1 answer
2k views

Map-reduce implementation for splitting strings

I have been changing this code and I don't get to make it much better. I changed a little bit the structure, reimplemeted a new function for splitting Strings which is more efficient, etc. I have been ...
user avatar
  • 183