last day (15 days later) » 

11:55
0
A: TypeError: a float is required pyspark

Ramesh MaharjanThe error didn't get replicated in mine but the output was all null too. So I passed columns instead of string and it worked as gps_data.withColumn("dist", dist( col("longitude"), col("latitude"), lag("longitude", 1).over(my_window), lag("latitude", 1).over(my_window) ).alias("dist")).sh...

still getting the error ,which spark version are you using?
spark 2.3 and yours?
I am using spark 2.0.2
I just tested in 2.0.2 and its working normal as in 2.3.0. so i guess you have to see where the error is generating at
ohh...I will just run the whole script again and acknowledge you if it works.
11:55
can you update the question with the schema of the dataframe?
I have updated the question
yes I saw that
well its working for me
same error.. Give me sometime then I will have to check again. Ran the script but no luck
I am running on the dataset you have provided in the question
are you doing the same?
if not try on that first
I am using that dataset only
12:07
really?
yes,Why will I lie
i am not saying you are lieing
I just got surprised
how did you create the dataframe
/
?
gps_data=spark.read.csv("/home/experiment/Desktop/GPS Trajectory/go_track_trackspoints.csv"
, header=True)
let me try that way too
yes please
12:16
no errors
can you paste the rest of the codes too?
import pandas as pd
import numpy as np
from pyspark.sql.functions import *
from pyspark.sql.types import *
from pyspark.sql.window import Window
from pyspark.sql.functions import stddev,avg,broadcast
from pyspark.sql.functions import max,min
from pyspark.sql import functions as F
from pyspark.sql.functions import stddev_pop, avg, broadcast,abs
from math import radians, cos, sin, asin, sqrt, atan2, pi
from pyspark.sql.functions import acos, cos, sin, lit, toRadians
gps_data=spark.read.csv("/home/deevisha/Desktop/GPS Trajectory/go_track_trackspoints.csv"
, header=True)
def dist(lon2, lat2,lon1, lat1):
phi_1=toRadians(lat1)
phi_2=toRadians(lat2)
delta_phi=toRadians(lat2-lat1)
delta_lambda=toRadians(lon2-lon1)


a=sin(delta_phi/2.0)**2+cos(phi_1)*cos(phi_2)*sin(delta_lambda/2.0)**2
c=2*atan2(sqrt(abs(a)),sqrt(abs((1-a))))
return c * 6372.8
gps_data.withColumn("dist", dist(
col("longitude"), col("latitude"),
lag("longitude", 1).over(my_window), lag("latitude", 1).over(my_window)
).alias("dist")).show(truncate=False)
the indentation is poor in chat
12:31
its one of your imports
I got the error replicated
let me debug
just remove the math imports and you should be fine
yes you are right!! Thanks for your help!
I guess you solved it didn't you?
ok great
you can accept the answer and upvote too
that would be a help to me
thanks
hehe
already done (y) :D
great
thanks
Was so caught up in the code ,was not able to figure what really was going wrong .Silly me

last day (15 days later) »