last day (14 days later) » 

1:21 PM
0
Q: Error using python to import data in hive

mdivkI am learning using python to import data into Hive on hadoop, here is the python code: import sys import datetime for line in sys.stdin: line = line.strip() userid, movieid, rating, unixtime = line.split('\t') weekday = datetime.datetime.fromtimestamp(float(unixtime)).isoweekday() ...

 
Your data is JSON, so FIELDS TERMINATED BY '\t' doesn't make much sense
 
I have no clue why it is like that, what's your suggestion to this? Thanks.
 
I can't follow what the data flow here is... Is the Python reading in JSON data from stdin? It doesn't look like it. Please edit your question to show the data.
 
sample data posted in OP, thank you.
 
Okay, that does look like tab delimited data, so where are you getting JSON? What's in u_data?
 
1:21 PM
here are the description of the data: u.data -- The full u data set, 100000 ratings by 943 users on 1682 items. Each user has rated at least 20 movies. Users and items are numbered consecutively from 1. The data is randomly ordered. This is a tab separated list of user id | item id | rating | timestamp. The time stamps are unix seconds since 1/1/1970 UTC
 
Hmm, that also says tab separated... I doubt the python is the problem, but it isn't required, actually. I believe you can convert time strings within Hive alone
 
can you show me the right steps you would take to finish this import? what would be the best practice? I just started hadoop learning. Thank you very much.
Good morning
 
Morning. I'm actually on my phone... So this may be hard to help
 
oh, maybe teamviewer? please......
I can wait for your convenience time
 
Essentially, you have the timestamp, you only need a date format string
for the day of the week
Very bottom of the table... dayname(date) is date_format(date, 'EEEE')
 
1:31 PM
so maybe this is all I need?
INSERT OVERWRITE TABLE u_data3
SELECT
userid, movieid, rating, from_utc_timestamp(unixtime,'EST').isoweekday()
AS (userid, movieid, rating, weekday)
FROM u_data;
and in this case the python script is not necessarily needed, right?
 
Where do you see that isoweekday function?
 
from the python code
 
And, correct, all your python was doing was converting that one column
You should be using Hive functions, not python functions
 
thanks. but you actually just reminded me that isoweekday if a python function, will it be accepted by hive? if not, ....?
 
It won't because hive doesn't run python
 
1:35 PM
so it's gonna fail...:(....
what's wrong with the python script though?
 
date_format(from_unixtime(unixtime), "EEEE"))
 
found a solution to get the day of week in hive
8
Q: Hive date function to achieve day of week

user3279189I'm looking for a workaround or hive date functions that gives day of the week , Sunday - 1 Monday - 2 Tuesday - 3 Wednesday - 4 Thursday - 5 Friday - 6 Saturday - 7 Requirement in detail : I'm looking for a function that takes date string (YYYYMMDD) as input and outputs the day of the week as ...

 
Something like that
Yeah, found that question too
 
ok, I will try that out, but what's wrong with the original python script?
why it doesn't get working in hive?
 
Other than it isn't entirely needed, I didn't see anything wrong with it
 
1:38 PM
hmmm
 
The real problem is your getting JSON data without any tabs at some point
You're*
 
ok, thank you very much for your enlightening.
can you reach out to you later when I have other question on my hadoop journey?
please..........
 
Ha, sure. That's kinda my job... Hadoop contracting
 
wonderful, please tell me how to get you? email? skype?
 
cricket.007@outlook.com should work
I'm not on Skype too frequently though
 
1:42 PM
that's enough, thank you soooooo much, have an excellent day, btw, what is your timezone?
 
CST. I'm much more active during the week on StackOverflow (though I actually should be doing work related tasks)
 
ok, I am on EST, one hour ahead of you. :)
bye for now, thanks again and have a good rest of the day, talk to you later
 
Welcome and likewise
 

last day (14 days later) »