2
\$\begingroup\$

I am trying to apply a function to two Spark Dataframes (in Zeppelin):

def RemoveStopwords (dfin : dataframe) : Dataframe = {
    val stopwords = 

    sc.textFile("/media/ed/Seagate/data/stopwords.txt")

    val tokenizer = new Tokenizer().setInputCol("text").setOutputCol("words")
    val wordsData = tokenizer.transform(dfin)

    // remove stop words
    val remover = new StopWordsRemover().setInputCol("words").setOutputCol("NoStop")
    val dfNoStop= remover.transform(wordsData)

}

and I call it:

Array(dfHive, dfHive1).map(RemoveStopwords)

Is this the right way of doing things? Should I be passing in the stopwords.txt file rather than looking it up twice? Also, is it OK to have : Dataframe (some functions I have seen are declared name : type not just : type)?

\$\endgroup\$
1
\$\begingroup\$

In case it helps anybody, I needed to do the following to fix my code:

  1. add import org.apache.spark.sql.DataFrame
  2. Make sure my DataFrame's were capitalized correctly.
  3. add dfNoStop at the end of the function, to stop it returning unit.
  4. make the call val Array(dfNoStop, dfNoStop1)=Array(dfHive, dfHive1).map(RemoveStopwords)
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.