last day (14 days later) » 

7:28 AM
1
A: Pandas NaN introduced by pivot_table

jezraelI think the best understand pivoting is in small sample: import pandas as pd import numpy as np countryKPI = pd.DataFrame({'germanCName':['a','a','b','c','c'], 'indicator.id':['z','x','z','y','m'], 'value':[7,8,9,7,8]}) print (countryKPI) ...

 
Indeed this example is good. But how can I prevent the NaN values?
 
Ok, what do you need? NaN replace to 0?
 
I see so the problem is that in my data for some countries not all indicators were reported ... :(
 
Yes, exactly. This is problem why you get NaN.
 
But a call to api.worldbank.org/countries/eg/indicators/… indeed returns some data -> which strangely did not result in my download pd.Dataframe
 
7:28 AM
How do you read data from worldbank?
 
Hi Georg
 
but I already could confirm if only EG (one of the countries with missing values) is used then the data is downloaded correctly. Only if I pass a long list >100 countries this problem seems to occur
 
It seems more complicated, maybe some timeout?
 
I already found out that a download of 'ALL' is getting all the data which is missing ...
 
7:32 AM
Can you post me list of your 100 countriesDL for test?
 
TR;AT;BA;DE;HR;PT;RS;NL;US;HU;IT;SK;RU;VN;RO;SI;NP;EG;PL;FR;IN;NG;ES;GE;SK;CN;MA‌​;UA;CO;EE;SE;AL;MN;AF;TH;CH;ME;MK;BG;GB;MK;AM;BR;PH;FI;SO;DO;BY;CU;CM;KR;GR;BD;IL‌​;LB;LY;IR;AZ;CA;PK;IE;BE;JP;JO;DZ;KG;LT;AU;MY;ID;TN;HK;IQ;MD;ZA;KZ;DK;KE;GM;LV;SA‌​;GH;JM;BT;KH;SY;MG;PE;CG;NO;EC;LI;CL;CI;SC;TM;GT;KP;MM;LU;SG;MX;IS;SN;SD;HT;OM;BO‌​;LK;SM;NZ;ET;AR;NA;CY;PY;MZ;CN;GN;NE;SB;ZW;UZ;VE;BN;BF;NI;AE;DM;SL;TZ;MC;CR;TG;HN‌​;BJ;SV;AO;PS;ZM;TJ;LC;KW;UG;GW;MU;MO;RW;VU;GA;CV;FJ;TT
 
Sorry, imports are misiing in your script
 
they are these here:
import pandas as pd
import numpy as np
import datetime as DT
import requests
from bs4 import BeautifulSoup
import fs
import json
from pandas.io.json import json_normalize
import gettext
import os
 
7:49 AM
@jezarel: can you replicate my problem?
 
It is really weird
I try use firefox and past link
http://api.worldbank.org/countries/
TR;AT;BA;DE;HR;PT;RS;NL;US;HU;IT;SK;RU;VN;RO;SI;NP;EG;PL;FR;IN;NG;ES;GE;SK;CN;MA‌​;UA;CO;
EE;SE;AL;MN;AF;TH;CH;ME;MK;BG;GB;MK;AM;BR;PH;FI;SO;DO;BY;CU;CM;KR;GR;BD;IL‌​;LB;LY;IR;AZ;
CA;PK;IE;BE;JP;JO;DZ;KG;LT;AU;MY;ID;TN;HK;IQ;MD;ZA;KZ;DK;KE;GM;LV;SA‌​;GH;JM;BT;KH;SY;MG;
PE;CG;NO;EC;LI;CL;CI;SC;TM;GT;KP;MM;LU;SG;MX;IS;SN;SD;HT;OM;BO‌​;LK;SM;NZ;ET;AR;NA;CY;PY;
MZ;CN;GN;NE;SB;ZW;UZ;VE;BN;BF;NI;AE;DM;SL;TZ;MC;CR;TG;HN‌​;BJ;SV;AO;PS;ZM;TJ;LC;KW;UG;GW;
MU;MO;RW;VU;GA;CV;FJ;TT/indicators/SP.DYN.LE00.IN
but worldbank api return
{"fault":{"faultstring":"Evaluation of script urlPatternScript.py (py) failed with reason: \"UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)\"","detail":{"errorcode":"steps.script.ScriptEvaluationFailed"}}}
and what is worse, it change link to:
http://api.worldbank.org/countries/%20TR;AT;BA;DE;HR;PT;RS;NL;US;HU;IT;SK;RU;VN;RO;SI;NP;EG;PL;FR;IN;NG;ES;GE;SK;CN;MA
%E2%80%8C%E2%80%8B
;UA;CO
;%20EE;
SE;AL;MN;AF;TH;CH;ME;MK;BG;GB;MK;AM;BR;PH;FI;SO;DO;BY;CU;CM;KR;GR;BD;
IL%E2%80%8C%E2%80%8B;
LB;LY;IR;AZ;
%20CA;
PK;IE;BE;JP;JO;DZ;KG;LT;AU;MY;ID;TN;HK;IQ;MD;ZA;KZ;DK;KE;GM;LV;
SA%E2%80%8C%E2%80%8B;
GH;JM;BT;KH;SY;MG;
%20PE;
CG;NO;EC;LI;CL;CI;SC;TM;GT;KP;MM;LU;SG;MX;IS;SN;SD;HT;OM;
BO%E2%80%8C%E2%80%8B;
LK;SM;NZ;ET;AR;NA;CY;PY;
%20MZ;
CN;GN;NE;SB;ZW;UZ;VE;BN;BF;NI;AE;DM;SL;TZ;MC;CR;TG;
API adds some %20, I dont know why, to link.
 
Ok - so I will go for all countries and adopt my script
Thanks a lot
 
opps, I post bad linjk - I add some line breaks.
So It return only:
http://api.worldbank.org/countries/TR;AT;BA;DE;HR;PT;RS;NL;US;HU;IT;SK;RU;VN;RO;SI;NP;EG;PL;FR;IN;NG;ES;GE;SK;CN;
MA%E2%80%8C%E2%80%8B;
UA;CO;EE;SE;AL;MN;AF;TH;CH;ME;MK;BG;GB;MK;AM;BR;PH;FI;SO;DO;BY;CU;CM;KR;GR;BD;
IL%E2%80%8C%E2%80%8B;
LB;LY;IR;AZ;CA;PK;IE;BE;JP;JO;DZ;KG;LT;AU;MY;ID;TN;HK;IQ;MD;ZA;KZ;DK;KE;GM;LV;
SA%E2%80%8C%E2%80%8B;
GH;JM;BT;KH;SY;MG;PE;CG;NO;EC;LI;CL;CI;SC;TM;GT;KP;MM;LU;SG;MX;IS;SN;SD;HT;OM;
BO%E2%80%8C%E2%80%8B;
LK;SM;NZ;ET;AR;NA;CY;PY;MZ;CN;GN;NE;SB;ZW;UZ;VE;BN;BF;NI;AE;DM;SL;TZ;MC;CR;TG;
But unfortunately it is bad too.
ok.
Good luck! And thank you for accepting!
 
Thanks for your explanation!
One more thing: I just found out that even for /all data for AF Afghanistan is not downloaded correctly
probably will need to go for brute-force and download one by one
 

last day (14 days later) »