last day (42 days later) » 

7:24 PM
Hey Chris
 
Hey
 
Thanks for joining
Do you live in UK now?!
 
Always have!
 
Not planing on coming to US/NYC?!
:)
 
I am there on the 19th August! eventbrite.com/e/…
But just for a few days
 
7:27 PM
Cool, you should come and visit Grovo (The company I am working for)
lot of exciting things over here
 
Happy to. What are you currently doing with Spark on C*? Attending the C* day?
 
We are using Spark Streaming / C* to pre-aggregate reports data
running in Dev now
 
Having issues?
 
checked out C* aggregates in 2.2? If it is aggregates within a partition may get to remove some of the streaming
 
7:29 PM
I will give you some context
I did
 
tho 2.2 long way off prod ready
 
but spark connector does not support 2.2 yet
and php driver will not support it till later
 
Have to run in about 5 minutes but can chat tomorrow
btw one of the main devs for the c* connector is based in NYC (helena)
 
I will write couple things here so you can read it once you have time
Cool, didn't know that
Context:
Web logs being submitted to kafka shipped by logstash and SparkStreaming reading the data
data sample:
{"version":"1.1","timestamp":1435000679,"subject":{"User":549431},"verb":["Viewe‌​d"],"direct_object":{"Material":"4806"},"indirect_objects":{"Chapter":"289693","E‌​nterprise":26862,"Training":"25678","Quiz":null},"prep_objects":[],"properties":{‌​"subject":{"ForcedLoggedIn":false,"SessionId":"5847ehmsa07mk6itvfa6k6v9n4"},"verb‌​":[],"direct_object":{"PreviouslyCompleted":false}}}
Running SparkSQL over it
after aggregation process I call the following:
val cassandraRDD = rdd.map(view => new LessonViewsByDayToCassandra(view._1.user, view._1.enterprise, view._1.day, view._2))
log.trace("Calling saveToCassandra")
cassandraRDD.saveToCassandra("dashboard", "counter_lesson_views_per_day")
log.trace("Finished calling saveToCassandra")
saveToCassandra works sometimes and most of the times it just doesn't
!!!!!
It does not even throw and exception
I believe it is a Cassandra configuration issue on write but can't be sure though
 
all the PK columns non null?
 
7:36 PM
Yes
 
tried saving the rdd to a file too just to make sure?
 
I did
and it is working
 
just not seeing the data in C*?
 
Yep
It is killing me over here
 
Try up the CL to see if it then causes a failure
tho i'd be very surprised if it did
 
7:37 PM
what do you mean by CL?
 
tried inserting similar data manually? just to sanity check the cluster
consistency level
 
over cqlsh Command Line!
 
or a standard java app using the driver
 
Consistency Level tried (LOCAL_ONE, Any, ALL)
 
try a QUORUM
 
7:38 PM
Tried LOCAL_QUORUM
tried to set #spark.cassandra.output.throughput_mb_per_sec
 
what data sizes are we dealing with?
 
nothing almost (15 KB)
is there a way to debug on the connector level!?
And are there any configuration that I should be aware of that I am probably missing
 
i have to run now, can you just paste the problem in an issue @ datastax-oss.atlassian.net/secure/… and i'll check in the morning that someone is on it?
 
Already did
I will give you the link
Thank you Chris
 
ah it is already assigned to your local NYC developer :)
I will chase her up tomorrow
 
7:42 PM
is the connector version might be affecting the process
1.4.0-M1
spark version 1.4
 
That is a early developer release :)
tho i'd expect something as basic as this to always work
 
But there aren't any other versions to deal with spark 1.4
Totally
@ChrisBatey Thanks
 
np, chat tomorrow
 
sure thing, enjoy your day
 
Not much left in the UK!
 
7:48 PM
hahha
 
8:30 PM
Does saveToCassandra work with empty counter tables?!
 

  last day (42 days later) »