Ramesh Maharjan

Sep 27, 2018 07:00
I see that you have already got your own version of answer. so it would be helpful if you upvote the answer. thanks
Sep 27, 2018 07:00
Yes you are correct but its not efficient as the length of the records needs to be greater than 44 otherwise you will encounter the same error as before. better solution would be to just split the records by , and check the second or any other columns . I hope its clear .
Sep 27, 2018 07:00
you are almost there. if you have 67832,CLARE,MANAGER,68319,1991-06-09,2550.00,,1001 the length is 50 and so p.length is 50 but if you have 678 then p.length is 3 . did you get the point now? so the whole point is for !p.substring(p.length-5).contains("2001") condition a record must have at least 5 characters as p.length-5 will be 0 when the length of a record is 5 but if a record as 3 characters as 678 then p.length-5 would be -2 which is a wrong index and in your error message you had -4 arrayindexoutofbound error. I hope its clear now
Sep 27, 2018 07:00
p.length > 5 is for each records in the file . so now if you have less than 6 characters in a record then that record is filtered out. did you understand?
Sep 27, 2018 07:00
p.length is the length of each line that is read from the file as string. And because we have to check for 2001 in each line and 2001 is of length 4 so p.length > 5 is used. Hope its clear now
Sep 27, 2018 07:00
its a data issue. you must have lines with only one character and thats where the error message throws up. just change p.length > 0 filter to p.length > 5 and you should be fine. I have updated my answer as well
Sep 27, 2018 07:00
can you update the question with the code you've tried and the error message please
Sep 27, 2018 07:00
they will just look at the question requirements . if you fulfill all the requirements then you get full marks
Sep 27, 2018 07:00
if you want it as in the input data then my solution is better but if you require further computation with arrays by splitting the lines then arrays would be better. so it really depends on your requirement
Sep 27, 2018 07:00
thats the begin index for substring. so p.substring(p.length-5) should give you the last digit i.e. 2001 or 1001 and so on. you better look at api doc of subString
 
Sep 26, 2018 07:15
whats wrong with the approach?
Sep 26, 2018 07:15
I guess you can get enough information on how to do that in stackoverflow . for example stackoverflow.com/questions/46640862/…
Sep 26, 2018 07:15
your explanation is getting more complicated
Sep 26, 2018 07:15
what do you mean by "sorting will not work properly" ? can you explain that and update the question? I would suggest you to explain all the details in the question more clearly
Sep 26, 2018 07:15
all I meant is otherwise should contain only one column. thats all. if you want to check for the same condition multiple times then pass all the columns to udf and do the computation there and return a column only. but keep in mind that inbuilt functions should be used as much as possible than using udf function
Sep 26, 2018 07:15
you should write the logics then
Sep 26, 2018 07:15
otherwise should contain only one column
 
Jul 31, 2018 12:43
thanks
Jul 31, 2018 12:43
great
Jul 31, 2018 12:42
hehe
Jul 31, 2018 12:42
thanks
Jul 31, 2018 12:42
that would be a help to me
Jul 31, 2018 12:42
you can accept the answer and upvote too
Jul 31, 2018 12:42
ok great
Jul 31, 2018 12:41
I guess you solved it didn't you?
Jul 31, 2018 12:34
just remove the math imports and you should be fine
Jul 31, 2018 12:31
let me debug
Jul 31, 2018 12:31
I got the error replicated
Jul 31, 2018 12:31
its one of your imports
Jul 31, 2018 12:16
can you paste the rest of the codes too?
Jul 31, 2018 12:16
no errors
Jul 31, 2018 12:13
let me try that way too
Jul 31, 2018 12:10
?
Jul 31, 2018 12:10
/
Jul 31, 2018 12:10
how did you create the dataframe
Jul 31, 2018 12:09
I just got surprised
Jul 31, 2018 12:09
i am not saying you are lieing
Jul 31, 2018 12:07
really?
Jul 31, 2018 12:00
if not try on that first
Jul 31, 2018 12:00
are you doing the same?
Jul 31, 2018 12:00
I am running on the dataset you have provided in the question
Jul 31, 2018 11:57
well its working for me
Jul 31, 2018 11:56
yes I saw that
Jul 31, 2018 11:55
can you update the question with the schema of the dataframe?
Jul 31, 2018 11:55
I just tested in 2.0.2 and its working normal as in 2.3.0. so i guess you have to see where the error is generating at
Jul 31, 2018 11:55
spark 2.3 and yours?
 
Jul 25, 2018 10:58
the query is not copy pastable here
Jul 25, 2018 10:58
check your gmail chatt
Jul 25, 2018 10:56
do you have gmail
Jul 25, 2018 10:53
its the same string