last day (14 days later) » 

06:09
0
A: Select substring from an RDD in pyspark

user1952500How about: re.findall("[A-Za-z]+", "Toronto, Ontario, Canada", 0)[1] 'Ontario' re.findall("[A-Za-z]+", "California - LA", 0)[1] 'LA' This basically finds words composed only of alphabets and outputs the second element in the list.

is there a function called findall in pyspark ?
findall is a function from the regular expression module re of python.
oh okay, how do i pass my RDD to that function ?
The above will be the lambda in the map function. Eg: rdd.map(lambda x: re.findall("[A-Za-z]+", x, 0)[1])
So i did this ne = city.map(lambda x:re.findall("[A-Za-z]+",x,0)[1]) and when i tried to view the elements of ne i get only 2 elements [u'LA', u'TN']
06:09
what do you get when you view city ?
i get PythonRDD[43] at RDD at PythonRDD.scala:43
Your question mentioned cities as the rdd. Is city the same as cities ?
yep
what is city.count() ?
10000
thats my city.take(10000)
06:14
in your list there are some unclean elements that you'll have to eliminate before doing the lambda above
use the filter function to eliminate null strings
or rather do this
actually the filter seems to be the way to go since the data is not cleaned
okay i replaced u'' with empty and u` with '
something like rdd.filter(lambda x:len(x) > 0).map(lambda x: re.findall("[A-Za-z]+", x, 0)[1])
does the above work ?
okay
now the length is 3
[u'LA', u'TN', u'England']
the length is 3 with the filter statement is it ?
yep
06:22
well there needs to be more cleaning of data i presume. you can use a big hammer and do the following:
rdd.filter(lambda x:len(re.findall("[A-Za-z]+", x, 0)) > 1).map(lambda x: re.findall("[A-Za-z]+", x, 0)[1])
this will apply the same function twice, so it's inefficient, but does the job and is not too bad since the individual records in the rdd are tiny
haha ill try
else you'll have to clean the data in a better way...without knowing the source of data it is tough to predict all the cleaning that is needed
okay now the count works normal
could you link me to any spark resource ?
i mean for reference
06:25
well i use the pyspark reference in general: spark.apache.org/docs/1.1.1/api/python/…
yeah, it is not that detailed ..
yeah unfortunately it's problematic...but as you go along using it it gets better
hope so :)
thanks @user1952500
i don't know of any other one...
np, have fun with pyspark
again thanks :)

last day (14 days later) »