last day (15 days later) » 

07:00
0
A: Printing data normally without Array of arrays in spark RDD

Ramesh MaharjanFor simplicity you can just add one more && expression in your initial expression as val employeesRdd=sc.textFile("file:///home/cloudera/Desktop/Employees/employees.txt").filter(p=>{p.length > 5 && p!=null && !p.substring(p.length-5).contains("2001")}) employeesRdd.foreach(println) will giv...

What does p.length - 5 indicates ?? Can you elaborate it in detail.. why -5 ??
thats the begin index for substring. so p.substring(p.length-5) should give you the last digit i.e. 2001 or 1001 and so on. you better look at api doc of subString
Ohhh ! Understood but is it a good practice ??
Yeah sure. Thanks for your answer but from certification perspective which approach is better according to you ?? Array of arrays or the normal form like you suggested above ?
if you want it as in the input data then my solution is better but if you require further computation with arrays by splitting the lines then arrays would be better. so it really depends on your requirement
Okay . Thank you so much for your detailed explanation but do you have any idea from certification perspective ?? How do they expect output in exams ??
07:00
they will just look at the question requirements . if you fulfill all the requirements then you get full marks
Yes, Ramesh . Sure I haven't tried yet . Let me try and I will upvote and mark the answer :)
I tried your code but it throws exception along with the data because of which when I try to save data in to text file it throws error and doesn't write data. Please refer imgur.com/a/EEkDNtz
can you update the question with the code you've tried and the error message please
I tried the same code answered by you. Nothing modified
its a data issue. you must have lines with only one character and thats where the error message throws up. just change p.length > 0 filter to p.length > 5 and you should be fine. I have updated my answer as well
yeah it worked now but can you just explain me why did you change p.length > 0 to p.length > 5 ?? in layman terms
p.length > 5 means number of records should be greater than 5 . correct ??
07:00
p.length is the length of each line that is read from the file as string. And because we have to check for 2001 in each line and 2001 is of length 4 so p.length > 5 is used. Hope its clear now
So initially i used p.length > 0 so is it checking that there should not be any empty records ??
and you used p.length > 5 for last column ..so If i want to deal say suppose first column so what should be the length criteria ??p.length > 6 ?? since its length is 5 ?? Am i correct with this ??
p.length > 5 is for each records in the file . so now if you have less than 6 characters in a record then that record is filtered out. did you understand?
okay that is why i get last column and criteria is matched for that column since it has less than 5 characters . Am i correct this time ?? so i can ask few other doubts as well. I am sorry but I am a newbie and trying my best to understand basics.sorry for bothering you,sir
you are almost there. if you have 67832,CLARE,MANAGER,68319,1991-06-09,2550.00,,1001 the length is 50 and so p.length is 50 but if you have 678 then p.length is 3 . did you get the point now? so the whole point is for !p.substring(p.length-5).contains("2001") condition a record must have at least 5 characters as p.length-5 will be 0 when the length of a record is 5 but if a record as 3 characters as 678 then p.length-5 would be -2 which is a wrong index and in your error message you had -4 arrayindexoutofbound error. I hope its clear now
Yes, Ramesh. Now I understood each and every point. So now suppose if i want to access the second column from start i wrote !p.substring(p.length-44).contains("FRANK") and it worked..It excluded all the records containing "FRANK"..so is it correct ??
07:00
Yes you are correct but its not efficient as the length of the records needs to be greater than 44 otherwise you will encounter the same error as before. better solution would be to just split the records by , and check the second or any other columns . I hope its clear .
I see that you have already got your own version of answer. so it would be helpful if you upvote the answer. thanks

last day (15 days later) »