last day (14 days later) » 

2:51 PM
1
A: python pandas - how to merge date from one and time from another column and create new column

jezraelYou can use strftime with to_datetime: df['date'] = pd.to_datetime(df.FltDate.dt.strftime('%Y-%m-%d ') + df.ESTAD2.dt.strftime('%H:%M:%S')) print (df) Airport FltDate Carrier ESTAD2 date 0 EDI 2017-06-18 BACJ 1899-12-30 05:35:00...

 
thk u. just wondering why can't we use df.FltDate.dt.date and df.ESTAD2.dt.time directly with some addition or datetime concat function. without the use of stftime. Is strftime the most optimal solution? please shed some light on this. i am asking this because i already have datetime object right.. then why bother to convert to string and back to datetime again.
 
My first solution was with convert to date, time, timedelta, but i think this is much faster and simplier.
Yes, but date and time in panas are objects, so use function is really problematic. So is necessary convert it to datetime - too complicated I think.
 
ok thk u... for some reason Row 0 shows new col 'date' as 2017-06-19 11:35:00 (instead of 2017-06-18 05:35:00) in my computer. this is unlike ur result. possibly to do with timezones?? i am in UTC+7.
 
Yes, it can be problem with timezones. check docs and try df.Date = df.Date.dt.tz_localize('UTC').dt.tz_convert('Europe/London')
Europe/London change up to you :)
 
i think the problem is not timezones... even df.ESTAD2.dt.strftime('%H:%M:%S') returns.. 1899-12-30 05:35:00 (instead of just the 05:35:00) as our expectation... pls advise.
 
2:51 PM
Problem is with all dates?
 
yes all the ESTAD2... however... df.FltDate.dt.strftime('%Y-%m-%d ') returns correct as expected ..'2017-06-18 '... only the df.ESTAD2.dt.strftime('%H:%M:%S') returns.. 1899-12-30 05:35:00 (instead of just the 05:35:00)... fyi the database is ms access with only timestamps in the fields. that is why the 1899-12-30 hh:mm:ss
 
hi
it is weird
 
hi thanks for all your help
it is in the right direction.. however... i see this issue.. do you need me to check anythingmore?
 
is possible data out of limit?
 
no i don't think so... access dates.. start from 1900
01/01/1900
whenever there is just timestamp... it shows... something like 1899.
 
2:53 PM
what is your version of pandas?
 
latest for sure 0.19..sthg
0.19.2
 
it is really weird
 
data as i said comes from MS Access 64bit
through pyodbc
 
yes, so df.ESTAD2.dtype is datetime?
 
dtype('<M8[ns]')
above is the dataframe after i received code from u for the 'date' column
 
3:03 PM
I dont understand
I nvere see it :(
 
i am sure we can work around it through some deltas... right? but we can't find the cause. is it?
if u can help me to simplify will be great.. or should i check anymore.. please also guide... sorry for occupying ur time for this long.
 
alternative solution
df['date'] = pd.to_datetime(df.FltDate.dt.strftime('%Y-%m-%d ') +
                            df.ESTAD2.astype(str).str.split().str[1])

print (df)
  Airport    FltDate Carrier              ESTAD2                date
0     EDI 2017-06-18    BACJ 1899-12-30 05:35:00 2017-06-18 05:35:00
1     EDI 2017-06-18      BA 1899-12-30 06:40:00 2017-06-18 06:40:00
2     EDI 2017-06-18    BACJ 1899-12-30 07:00:00 2017-06-18 07:00:00
3     EDI 2017-06-18      BA 1899-12-30 07:05:00 2017-06-18 07:05:00
4     EDI 2017-06-18      BA 1899-12-30 09:00:00 2017-06-18 09:00:00
convert to string, split by whitespace and select second values
 
fyi. no matter what i did with ur original solution... it shows like this
df.ESTAD2.dt.strftime('%Y')
even above shows
0 1899-12-30 05:35:00
1 1899-12-30 06:40:00
2 1899-12-30 07:00:00
3 1899-12-30 07:05:00
4 1899-12-30 09:00:00
5 1899-12-30 11:05:00
6 1899-12-30 11:25:00
yes alternate solution works..
 
super.
I dont know what was problem
 
should i file a bug report.. to pandas.. or sthg?
and will you edit solution in SO.. with comments.. or i should do that?
 
3:14 PM
yes, solution was modified.
 
ok thank you so much.
 
yes, bug report is nice, but can you simulate error?
 
i will play with it more... above was the simplistic case..
df.ESTAD2.dt.strftime('%Y')
even above shows
0 1899-12-30 05:35:00
1 1899-12-30 06:40:00
2 1899-12-30 07:00:00
3 1899-12-30 07:05:00
4 1899-12-30 09:00:00
5 1899-12-30 11:05:00
6 1899-12-30 11:25:00
 
it is wrong
I dont understand still
 
3:46 PM
mydate = '1899-12-30 05:35:00'
print pd.to_datetime(mydate).strftime('%Y').head()
print df.ESTAD2.dt.strftime('%Y').head()
print df.FltDate.dt.strftime('%Y').head()
mydate = '1899-12-30 05:35:00'
print pd.to_datetime(mydate).strftime('%Y').head()
print df.ESTAD2.dt.strftime('%Y').head()
print df.FltDate.dt.strftime('%Y').head()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-52-53d0331c4c47> in <module>()
hi this is what it shows... ValueError: year=1899 is before 1900; the datetime strftime() methods require year >= 1900
i tried to do a simple test
 

last day (14 days later) »