1
\$\begingroup\$

I am working out some PySpark exercises and trying to write efficient and best practice code to be ready to work in a production environment.

I could use some feedback with respect to:

  1. Is the code structured well?
  2. Is the code efficient?
  3. Any other best practices I missed but should make habits of.
import findspark
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
import pyspark.sql.types as T

findspark.init()

spark = SparkSession.builder.getOrCreate()

# Sample Data

data = [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)]

# Create DataFrame

df = spark.createDataFrame(data, ["col1", "col2", "col3"])

df.show()

import math

def smallest_column(smallest, col):
    return F.when(smallest <= col, smallest).otherwise(col)
 
def biggest_column(biggest, col):
    return F.when(biggest >= col, biggest).otherwise(col)       
        

df_small_big = (
    df
    .withColumn("smallest", F.col(df.columns[0]))
    .withColumn("biggest", F.col(df.columns[0]))
)

for c in df.columns:
    df_small_big = (
        df_small_big
        .withColumn("smallest", smallest_column(F.col("smallest"), F.col(c)))
        .withColumn("biggest", biggest_column(F.col("biggest"), F.col(c)))
    )


df_small_big = df_small_big.withColumn("min_by_max", F.col("smallest")/F.col("biggest"))

df_small_big.select(*df.columns, "min_by_max").show()

OUTPUT

+----+----+----+------------------+
|col1|col2|col3|        min_by_max|
+----+----+----+------------------+
|   1|   2|   3|0.3333333333333333|
|   4|   5|   6|0.6666666666666666|
|   7|   8|   9|0.7777777777777778|
|  10|  11|  12|0.8333333333333334|
+----+----+----+------------------+
\$\endgroup\$
0

1 Answer 1

1
\$\begingroup\$

For the most part, the code layout is good, and you chose meaningful names for variables and functions.

When I run pylint, it complains about these unused imports:

import pyspark.sql.types as T
import math

It's usually a good idea to remove code that isn't used.

pylint also identifies "Trailing whitespace". It is a good idea to remove that as well.

It would be good to add comments at the top to describe what the code does.

I recommend moving the functions to the top, after the import statements. Having them in the middle of the code interrupts the natural flow of the code (from a human readability standpoint).

'''
PySpark: Create a column containing the minimum divided by maximum of each row
'''

import findspark
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

def smallest_column(smallest, col):
    return F.when(smallest <= col, smallest).otherwise(col)

def biggest_column(biggest, col):
    return F.when(biggest >= col, biggest).otherwise(col)

findspark.init()

spark = SparkSession.builder.getOrCreate()

# Sample Data
data = [(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)]

# Create DataFrame
df = spark.createDataFrame(data, ["col1", "col2", "col3"])

df.show()

df_small_big = (
    df
    .withColumn("smallest", F.col(df.columns[0]))
    .withColumn("biggest", F.col(df.columns[0]))
)

for c in df.columns:
    df_small_big = (
        df_small_big
        .withColumn("smallest", smallest_column(F.col("smallest"), F.col(c)))
        .withColumn("biggest", biggest_column(F.col("biggest"), F.col(c)))
    )

df_small_big = df_small_big.withColumn("min_by_max", F.col("smallest")/F.col("biggest"))

df_small_big.select(*df.columns, "min_by_max").show()
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.