last day (16 days later) » 

11:54 AM
0
Q: How to do Join on 3rd data frame?

ArpitI have 2 dataframe (5ml rows) each, DF1 & DF2. Each row in DF1 corresponds to row in DF2 at same index. DF1:- miliSecDF.iloc[:,list(range(1, 81, 2))][:3] Out[25]: B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11 B12 B13 B14 B15...

 
Use df = pd.concat([DF1, DF2], axis=1)
 
@jezrael, In DF3 values from DF1 become index and it might be having missing values.
 
So you need s = pd.Series(DF2['col'], index=DF1['col']) ?
 
@jezrael, I have 40 columns in DF1 & DF2.
 
So maybe not understand question, can you create minimal, complete, and verifiable example?
 
11:54 AM
@jezrael, I just edited the question.. please have a look.
 
hmmm, is possible create 3-4 columns for each and add expected output?
 
@jezrael, edited the question.
 
Thank you for minimal, complete, and verifiable example. So it means there are same values for index? And same values per columns?
Why is for expcted output for second row 404700 500 2000 2000 ?
 
We can ignore index.. In expected output first column will be index (which will come from DF1 (with range max, min value of dataframe)) corresponding values @ DF2.
It's B2 column
 
Sorry, but I am a bit confused
 
11:56 AM
Ok.. Wait let me explain again
I have 2 data frames
columns B1,,,B20 & A1...A20
 
ok
 
otherone BQ1..BQ20..AQ1...AQ20
B1 is key Value is BQ1
B2is key BQ2 is value
 
I am lost, sorry
is possible some sample data
like
change this
df = pd.DataFrame({
'A':list('abcdef'),
'B':[4,5,4,5,5,4],
'C':[7,8,9,4,2,3],
'D':[1,3,5,7,1,0],
'E':[5,3,6,9,2,4],
'F':list('aaabbb')
})
like you need
 
Ok
 
Because Not sure if understand it well
 
11:59 AM
Do you want to see over anydesk ?
I can show & explain the issue..
 
ya
but I have experience
always working with samll data sample
 
The issue is I am able to do it.. with Join.. but since it's huge number of rows.. it's taking very long time to complete
 
for easy working, verifying
and then apply solution to large data
ok
so what is your solution?
 
489932819
anydesk
join operation working fine with small data
but with large it's going on for 4-5hrs
 
ok
I think maybe your code help me understand what you need
 
12:01 PM
Yes, that's why anydesk
for i in range(1, 3000):
print(i)
d = {'Price' : np.concatenate(miliSecDF.iloc[:,list(range(2, 81, 2))][:1].values, 0), 'Qty': np.concatenate(miliSecDF.iloc[:,list(range(1, 81, 2))][:1].values, 0)}
master1 = pd.DataFrame(d)
master1.set_index('Price', inplace=True)
master = master.join(master1, rsuffix=i)
del master1
I did this for 3000 rows
 
hmmmm
 
d is dict 'Price' from DF1 & 'Qty' from DF2
master = pd.DataFrame(index=list(range(minValue, maxValue, 100)))
master['Qty'] = 0
 
so data are changed in time
 
yes
 
so is necessary join with original?
 
12:04 PM
Nopes
DF1 contains prices
Df2 Qty
both has index time
at different time DF1 price can have different qty values
but price will remain same
maxValue = miliSecDF.ix[:,list(range(1, 81, 2))].max().max()
minValue = miliSecDF.ix[:,list(range(1, 81, 2))].min().min()
New data frame will have index as price which is in DF1
 
hmmm
is possible create list of DataFrames first and use concat?
 
It's possible but how will it help
 
out = []
for i in range(1, 3000):
    print(i)
    d = {'Price' : np.concatenate(miliSecDF.iloc[:,list(range(2, 81, 2))][:1].values, 0), 'Qty': np.concatenate(miliSecDF.iloc[:,list(range(1, 81, 2))][:1].values, 0)}
    out.append(d)

df = pd.DataFrame(out)
Not sure if working, because I have no your data, but yiou can try
 
Let me try 2 min
 
ok
idea is create list of dictionaries
and then only once DataFrame constructor
 
12:11 PM
This has created DF but each column has lists
master1.head()
Out[31]:
Qty
Price
[500, 2000, 2000, 9000, 1500, 3500, 2000, 2500,... [404800, 404700, 404600, 404500, 404400, 40430...
[500, 2000, 2000, 9000, 1500, 3500, 2000, 2500,... [404800, 404700, 404600, 404500, 404400, 40430...
[500, 2000, 2000, 9000, 1500, 3500, 2000, 2500,... [404800, 404700, 404600, 404500, 404400, 40430...
[500, 2000, 2000, 9000, 1500, 3500, 2000, 2500,... [404800, 404700, 404600, 404500, 404400, 40430...
[500, 2000, 2000, 9000, 1500, 3500, 2000, 2500,... [404800, 404700, 404600, 404500, 404400, 40430...
 
hmmm
out = []
for i in range(1, 3000):
    print(i)
    d = {'Price' : np.concatenate(miliSecDF.iloc[:,list(range(2, 81, 2))][:1].values, 0), 'Qty': np.concatenate(miliSecDF.iloc[:,list(range(1, 81, 2))][:1].values, 0)}
    out.append(pd.DataFrame(d))

df = pd.concat(out)
 
cannot concatenate object of type "<class 'dict'>"; only pd.Series, pd.DataFrame, and pd.Panel (deprecated) objs are valid
 
do you use out.append(pd.DataFrame(d)) ?
 
2min
This seems to be working
Let me check for larger data
& it's faster as well
anyways we can get rid of for loop.. since it always slows down
Sorry, it's not what I am looking for..
How to should I send data ?
 
hmmm
is possible create sample data?
Because I really avoid working with big data
because really bad teting
testing
 
12:24 PM
I am just uploading data
and my code
my code
data
I have 40files like this
performance bottelneck line 69-75
 
hmmm
a bit complicated
 
little bit
 
btw, why solution with
out = []
for i in range(1, 3000):
print(i)
d = {'Price' : np.concatenate(miliSecDF.iloc[:,list(range(2, 81, 2))][:1].values, 0), 'Qty': np.concatenate(miliSecDF.iloc[:,list(range(1, 81, 2))][:1].values, 0)}
out.append(pd.DataFrame(d))

df = pd.concat(out)
not working?
 
it's giving single row & column
It should be same number of rows as in DF2
Like at T1 <P1> <Q1>
T2 <P1> <Q2>
Time T Price P Qty Q
It should be P1 Q1 Q2
 
ooops
Use df = pd.concat(out, axis=1)
 
12:31 PM
2 min.. let me check
 
ok
 
This is close
Now I need Price as index
df.columns
Out[7]:
Index(['Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty',
...
'Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty', 'Price', 'Qty'], dtype='object', length=5998)
 
hmmm
 
I hope problem is clear now
The issue is the code is working fine.. and as expected.. but for a large number it's taking more time.
 
hmmm
unfortunately not
complicated code
is possible send me it to my email and tomorrow I can check it
 
12:39 PM
Sure.
 
because I will be offline in 15 minutes
 
please share the email id
 
3-4 files and your code, please
 
Ok
 
my email is in my profile
only run code
 
12:41 PM
somehow I am not seeing the mail id
 
it is hidden
in code
run code for email
 
Ok
Very Interesting !!
Got it ! will send mail
files through google drive
 
too much emails, so it was necessary
;)
 
Thanks for your help.
 
ok
 
12:43 PM
really appreciate it
mail sent
will wait for your update
 
ok
I got email
 
Please have a look, whenever you have time.
T1 P1 Q1
T2 P1 Q2
P1 Q1 Q2 Q3..
Just a quick suggestion..
if you can do transpose of DF2
then DF wil be
T1 T2 T3 T4 (Below it Q1 Q2 Q3 Q4..)
and then we join, merge ,... not sure what .. based on T1 T2 T3.. for both dataframes
 

last day (16 days later) »