last day (14 days later) » 

9:10 AM
1
A: Pandas append multiple columns for a single one

jezraelI think you can use concat: df_pivoted = countryKPI.pivot_table(index='country', columns='indicator', values='value', fill_value=0) print (df_pivoted) indicator x z country Austria 7 7 ...

 
That nearly works - but I get the error of incompatible categories in categorical concat
 
Problem is with real data, right? Smale works perfectly, I think.
 
Unfortunately yes.
pandas.pydata.org/pandas-docs/stable/categorical.html#mergin‌​g this one In this case the categories are not the same and so an error is raised: might be a clue
 
One solution is convert category column to string or add missing categories. Which columns are dtype category?
 
the nationality columns used for the join
 
9:10 AM
Try customers.nationality = customers.nationality.astype(str)
I simulated problem, give me a time.
 
thanks a lot - nice to see you again so quickly.
 
;)
yes
 
I just tried the object variant - > that does not (yet) work as reindexing only is valid with uniquely valued index objects
in case I remove the set_index the join works. but only NaNs are joined
However, for the customers_df the set_index works just fine (as long as not used in the concat operation)
 
You working with sample?
 
no with the whole data - have not simulated it yet. but if you already have code for that would you mind sharing it so we work on the same thing?
 
9:16 AM
Yes.
I only add casting to categories
countryKPI = pd.DataFrame({'country':['Austria','Germany', 'Germany', 'Austria'],
'indicator':['z','x','z','x'],
'value':[7,8,9,7]})
customers = pd.DataFrame({'customer':['first','second'],
'nationality':['Slovakia','Austria'],
'value':[7,8]})

customers.nationality = customers.nationality.astype('category')
countryKPI.country = countryKPI.country.astype('category')
Then concat raise erorr
 
really? then I have to restart my jupyter kernel because I tried that before.
 
For me it works
 
Now I can confirm the error.
 
I found solution
 
really? very cool. How?
 
9:21 AM
Give me a sec, I update answer.
 
Thank you.
I edit answer, please check it.
+1 too ;)
 
I get a Name Error 'Index' is not defined
(for the sample code / minimum example)
ok - found my problem
Will try it on the big dataset.
 
Super ;)
 
As you remember from the last question for the countryKPI df I have the country not as a column but rather in the index
How can I "access" the .cat to set the new categories of the index?
 
9:37 AM
df.index = df.index.astype('category') shoulb be works.
 
thanks. For now it is (still) running but already showing some warning messages which were not present for the example.
--> Future warning sort is deprecated use sort_values

For the small example removing the other calls like

`pd.concat([customers.set_index('nationality'), df_pivoted], axis=1).reset_index()`

works just fine and that means that I would not require to specify the names in case of multiple / changing countryKPI_names. Why did you add these lines?
 
You need use instead function sort - sort_values, obviously only overwrite it.
 
sure - but that happend outside my coed in pandas /indices/base.py :)
 
10:10 AM
I have to admit I am not sure if it works on the big dataset as this seems to be a pretty time-consuming operation. I manually stopped it. If I have it run on a small set of data e.g. 10 rows the kernel is crashing repeatedly.
 
I add faster solution to my answer, please check it.
One thing - what is your version of pandas?
0.18.1 ?
 
10:29 AM
indeed 0.18.1
the grouped operation returns a value error: shape of passed values is 43 - 166 but indices imply 43 158
Strange -> on the toy example it fails in case the dataframe to merge contains categories # we've already checked that all categoricals are the same, so if their

ValueError: incompatible categories in categorical concat
I will try the group-by solution on the big dataset without categories and strings only
Indeed the problem persists.
@jezrael Thank you very much for your help again. Unfortunately, I could not get your solution to run on the big data frame I use. I hope you are not too disappointed that I selected the other solution as correct - > but both solutions do work fine on the small example.
 
11:00 AM
no problem
good luck!
 

last day (14 days later) »