last day (15 days later) » 

10:21 PM
0
A: spark structured streaming exception : Append output mode not supported without watermark

thebluephantomThe manual states: withWatermark must be called on the same column as the timestamp column used in the aggregate. For example, df.withWatermark("time", "1 min").groupBy("time2").count() is invalid in Append output mode, as watermark is defined on a different column from the aggreg...

 
i need to group by based on year and taking sum out of that...i am getting error if i ddnt use watermark... is there any way to tackle this ?
 
Manual says not possible. Have not tried if you can have another query after, will look tomorrow.
 
you mean can't we apply spark structured streaming for aggregation for columns other than watermarking? very sad
 
his issue is multiple aggregation... mine is only 1 aggregation
 
10:21 PM
yes but you have a group by in the watermark thing to consider and then another aggr
 
i have tried below df_agg_with_time.writeStream.outputMode("append").partitionB‌​y("year").format("‌​cs‌​v"). option("path", "hdfs://doxdt/apps/hive/warehouse/rtdv_dev_landing_initial_a‌​rea.db/sample_mov/‌​")‌​.start() getting error below: org.apache.spark.sql.AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark
 
I will look tomorrow with an example But I think you will need to write to Kafka and read that topic and AGG again. can you see if you can run without partitionBy
please update your question
 
ok let me check
 
slide unsupported operations, or do you want to group time, year?
 
i want to group only year
Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark
if i use water mark..then following error showing
CSV data source does not support struct<start:timestamp,end:timestamp> data type
 
10:36 PM
but we cannot always do what we want
suggest kafka approach, it is a bit limited
 
even 1 group by is not possible in spark stream ...really bad :(
 

last day (15 days later) »