last day (15 days later) » 

07:15
0
Q: Getting Unsupported literal class in UDF spark scala

AnupamI want to perform null check using below code . import org.apache.spark.sql.expressions._ val windowSpec = Window.partitionBy("FundamentalSeriesId", "FundamentalSeriesPeriodEndDate", "FundamentalSeriesPeriodType") val windowSpec2 = Window.partitionBy("FundamentalSeriesId", "FundamentalSe...

otherwise should contain only one column
@RameshMaharjan Yes I agree but what if we have more than one column to check same condition and add in the Otherwise part ?
you should write the logics then
@RameshMaharjan sorry did not get you ...You mean i need to remove the udf and write the logic ?
all I meant is otherwise should contain only one column. thats all. if you want to check for the same condition multiple times then pass all the columns to udf and do the computation there and return a column only. but keep in mind that inbuilt functions should be used as much as possible than using udf function
07:15
@RameshMaharjan if there is only one column in the otherwise then sorting will not work properly .
what do you mean by "sorting will not work properly" ? can you explain that and update the question? I would suggest you to explain all the details in the question more clearly
@RameshMaharjan updated my question with logic to use this ..
your explanation is getting more complicated
@RameshMaharjan i am sorry.. But in sort i want to include more than one columns in the ` val latestForEachKey1 = tempReorder .withColumn("group", when(containsUdf(collect_list("FundamentalSeriesStatementTyp‌​eCode").over(windo‌​wS‌​pec)), lit("same")).otherwise($"FundamentalSeriesStatementTypeCode"‌​)) ` here ..
I guess you can get enough information on how to do that in stackoverflow . for example stackoverflow.com/questions/46640862/…
07:15
@RameshMaharjan it looks like but in that case i have to change UDF ...Now this is tested so i do not want to make more changes to it ....Is there no way to pass all columns in the udf ?
@RameshMaharjan i have added what i am trying ..When you have time can you please have a look ?
whats wrong with the approach?
I mean what i have updated with that ? Or what answer you have suggested with that ?
I did like this .
import org.apache.spark.sql.expressions._
val windowSpec = Window.partitionBy("FundamentalSeriesId", "FundamentalSeriesPeriodEndDate", "FundamentalSeriesPeriodType")
val windowSpec2 = Window.partitionBy("FundamentalSeriesId", "FundamentalSeriesPeriodEndDate", "FundamentalSeriesPeriodType", "group", "group1", "group2", "group3").orderBy(unix_timestamp($"TimeStamp", "yyyy-MM-dd'T'HH:mm:ss").cast("timestamp").desc)

def containsUdf = udf { (array: Seq[String]) => array.contains("null") || array.contains("NULL") || array.contains(null) }

last day (15 days later) »