2022 Developer Survey is open! Take survey.

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

Filter by
Sorted by
Tagged with
0 votes
0 answers
40 views

dataset anonymizing algorithm implementation with Spark

I am trying to implement the following algorithm in Spark / Java: https://en.wikipedia.org/wiki/K-anonymity. My code class is the following: ...
user avatar
-1 votes
1 answer
79 views

Graph coloring problem with Spark (JAVA)?

I am trying to create the algorithm described in the image from this link Simply put any two adjacent nodes must not share tuples and if they do they are colored with the same color. If there's a ...
user avatar
3 votes
2 answers
153 views

PySpark SCD Type 1

I am using PySpark in Azure DataBricks to try to create a SCD Type 1. I would like to know if this is an efficient way of doing this? Here is my SQL table: ...
user avatar
  • 31
1 vote
1 answer
76 views

Group events close in time into sessions and assign unique session IDs

The following is a trimmed-down example of my actual code, but it suffices to show the algorithmic problem I'm trying to solve. Given is a DataFrame with events, each with a user ID and a timestamp. <...
user avatar
3 votes
1 answer
134 views

Rewriting scala code in object-oriented style style to reduce repetitive use of similar functions

I need help in rewriting my code to be less repetitive. I am used to coding procedural and not object-oriented. My scala program is for Databricks. how would you combine cmd 3 and 5 together? Does ...
user avatar
2 votes
0 answers
609 views

Spark Scala: SQL rlike vs Custom UDF

I've a scenario where 10K+ regular expressions are stored in a table along with various other columns and this needs to be joined against an incoming dataset. Initially I was using "spark sql rlike" ...
user avatar
  • 43
2 votes
2 answers
613 views

Scala app to transpose columns into rows

This is the first application or really any Scala I have every written. So far it functions as I would hope it would. I just found this community and would love some peer review of possible ...
user avatar
2 votes
1 answer
36 views

Filtering and creating new columns by condensing the lists for each item information

I am trying to improve my programming skills at work (analyst) and one of the engineering projects I worked is around ETL. Essentially, we roll up all individuals account information to a single row ...
user avatar
  • 121
2 votes
0 answers
194 views

Quantiles calculation in Pyspark MLib

I am trying to find out quantiles for each column on the table for various firms using spark 1.6 I have around 5000 entries in firm_list and 300 entries in attr_lst...
user avatar
3 votes
0 answers
35 views

spark takes long time for checking an array of items present in another array

I am new to spark. I have two dataframes df1 and df2. df1 has three rows. df2 has more than few million rows. I want to check whether all items in df2 are in transaction of df1, if so sum up the costs....
user avatar
  • 131
4 votes
3 answers
614 views

Binary check code in pyspark

I am trying to find out if a column is binary or not. If a column is only having 1 or 0 then I am flagging it as binary, else ...
user avatar
1 vote
0 answers
106 views

Managing PySpark DataFrames

I was successfully able to write a small script using PySpark to retrieve and organize data from a large .xml file. Being new to using PySpark, I am wondering if there is any better way to write the ...
user avatar
  • 111
2 votes
0 answers
521 views

Function to generate pyspark diff and return differences in line

I have written a function that takes two pyspark dataframes and creates a diff in line. I am struggling to get it to scale with 100s of columns. I get around the for...
user avatar
3 votes
1 answer
367 views

Large dataset with pyspark - optimizing join, sort, compare between rows and group by with aggregation

I have a csv file with more than 700,000,000 records in this structure: ...
user avatar
  • 33
1 vote
1 answer
5k views

Joining Apache Spark data frames, with many conditional substitutions

I am joining two data frame in spark using scala . My code looks very ugly because of the multiple when condition . Can somebody please help me simplify my code? Here is my existing code . ...
user avatar
  • 115
3 votes
0 answers
2k views

Apache spark compaction script to handle small files in hdfs

I have some use cases where I have small parquet files in Hadoop, say, 10-100 MB. I would to compact them so as to have files at least say 100 MB or 200 MB. The logic of my code is to: * find a ...
user avatar
  • 149
3 votes
0 answers
2k views

Adding columns in Spark dataframe based on rules

I have a dataframe df, which contains below data: ...
user avatar
2 votes
0 answers
156 views

Reduce sample rate of GPS data based on distance between points

The algorithm needs to reduce an RDD[GPSRecord] based on the distance between several points, e.g. "give me only GPS records when the distance between them exceeds ...
user avatar
6 votes
0 answers
75 views

Pyspark Solver for Tiered Board Games

I've written a Pyspark program that will completely solve a tiered board game (no loops, each game position is a member of only one tier) and writes each tier to a file. It also determines the ...
user avatar
  • 61
3 votes
1 answer
98 views

Classifying and counting database entries using Scala map and flatMap

I am new to Spark and Scala and I have solved the following problem. I have a table in database with following structure: ...
user avatar
0 votes
1 answer
6k views

Unit testing Spark transformation on DataFrame

Looking for suggestions on how to unit test a Spark transformation with ScalaTest. The test class generates a DataFrame from static data and passes it to a transformation, then makes assertion on the ...
user avatar
4 votes
0 answers
569 views

PySpark Dataframes program to process huge amounts of server data from a parquet file

I'm new to spark and dataframes and I'm looking for feedback on what bad or inefficient processes might be in my code so I can improve and learn. My program reads in a parquet file that contains ...
user avatar
  • 365
1 vote
1 answer
830 views

Python + spark to parse and save logs

I need to parse logs and have got following code. I can see two problems: map().filter() may induce some performance penalties and copy-paste block parser.py: <...
user avatar
  • 555
3 votes
1 answer
2k views

Implementing an inner product using pyspark

I'm trying to implement a dot product using pyspark in order to learn pyspark's syntax. I've currently implemented the dot product like so: ...
user avatar
2 votes
0 answers
358 views

Increase performance of Spark-job Collaborative Recommendation.

This is my first Spark Application. I am using "ALS.train" for training the model - Model Factorization. The total time that the Application takes is approx 45 mins. Note: I think takeOrdered is the ...
user avatar
1 vote
0 answers
444 views

collect and parallelize in spark performance

the below code is working fine, but as this is new for me, please help to improve performance. as I have not included yet my complex logic but confused over collect and parallelize. Aim is to collect ...
user avatar
  • 111
15 votes
1 answer
35k views

Generic “reduceBy” or “groupBy + aggregate” functionality with Spark DataFrame

Maybe I totally reinvented the wheel, or maybe I've invented something new and useful. Can one of you tell me if there's a better way of doing this? Here's what I'm trying to do: I want a generic <...
user avatar
  • 253
5 votes
0 answers
713 views

RandomForest multi-class classification

Below is the code I have for a RandomForest multiclass-classification model. I am reading from a CSV file and doing various transformations as seen in the code. I ...
user avatar
  • 153
8 votes
2 answers
3k views

Average movie rankings

Given a list of tuples of the form, (a, b, c), is there a more direct or optimized for calculating the average of all the c's ...
user avatar
0 votes
1 answer
606 views

Class for finding the median of a two-dimensional space

I have a simple static class that it's purpose is given an RDD of Point to find the median of each dimension and return that as a new ...
user avatar
  • 1,319
11 votes
2 answers
28k views

Producing a sorted wordcount with Spark

I'm currently learning how to use Apache Spark. In order to do so, I implemented a simple wordcount (not really original, I know). There already exists an example on the documentation providing the ...
user avatar
  • 213
5 votes
1 answer
2k views

Why does the LR on spark run so slowly?

Because the MLlib does not support the sparse input, I ran the following code, which supports the sparse input format, on spark clusters. The settings are: 5 nodes, each node with 8 cores (all the ...
user avatar
  • 151