« first day (4291 days earlier)      last day (673 days later) » 

3:26 AM
Has anyone in pandas ever imported/handled a dataset with an uneven multiindex on the columns? I'm slowly getting it working semi-manually, but it's a giant pain, looking for some tips from anyone who's done this. The dataset I'm using has 3 levels, like a worse version of Python pandas creating an uneven multiindex
 
3:47 AM
@smci bit more info. would be nice?
 
 
3 hours later…
7:00 AM
jagged multiindex sounds like a bad idea to begin with, you can just use them like normal columns perhaps
 
 
4 hours later…
11:14 AM
why are you trying to self promote your fresh answer to a 7 year old question with no context or relation to any conversation?
 
 
3 hours later…
2:40 PM
@mintorii please don't ask for help here with fresh questions on the main site as per our rules
 
 
1 hour later…
3:50 PM
Found a solution to my question above that works for vscode.
https://stackoverflow.com/a/59717891/15481857
 
@JacobBumgarner good find, thanks for getting back to us
 
 
2 hours later…
5:49 PM
Wow, mypy really just ignores @wraps decorators? What the heck is up with that
def f(x: bool, *extra: int) -> str: ...

@wraps(f)
def test(*args, **kwargs):
    return f(*args, **kwargs)

reveal_type(test)  # Revealed type is "def (*args: Any, **kwargs: Any) -> Any"
 
6:40 PM
@Aran-Fey typeshed has wraps() as the identity function, not copying the signature.
 
Any clue as to why?
 
7:22 PM
@mintorii: for beginner questions about reading CSV dataset files in pandas, mungeing int,float,string,categorical data from records into a strictly numerical (int/float)-only array X_train for training ML, computing features, handling NAs, etc. etc., please go read any of the tons of examples, beginner blogs, boilerplate out there; look at playground or past competitions on Kaggle.com. There's so much existing material out there, just start reading it.
 
7:41 PM
@Kevin ...sounds like a mantra for us idealists...
 
8:14 PM
@JonClements Here's the full example, I can't post it as an SO question yet because I'm not fully sure how to treat the (very uneven) multiindex, as it will be a pain to do aggregations or tabulations:
Ok here's a dataset with a very non-uniform multiindex. It's US Circuit court caseload data Q1/2022, [You can download the dataset 'fjcs_b1_0331.2022.xlsx' from uscourts.gov/statistics/table/b-1/….
Conceptually this is a data cube with 3 dimensions:Circuit, Nature_of_case and Status. Where Circuit∈ {'DC','1st','2nd',...'11th','Fed'}, Nature_of_case ∈ {'Criminal, 'U.S. Prisoner Petitions', ..., 'Original Proceedings and Miscellaneous Applications'} and Status is a very non-uniform 3-level multiindex whose first levels are ∈ {'Filed', 'Terminated', 'Pending'}
So Excel columns B..N contain the column multiindex. Rows 7..14 contain the caseload data for All Circuit courts; rows 15.23 contain the caseload data for 'DC' Circuit, rows 24..32 contain the caseload data for the '1st' Circuit, ... rows 114..123 contain the caseload data for the '11th' Circuit. [They have expanded/munged the cube into a flat 2D table, I will want to extract it back to a cube stucture, among other things]
Here's some boilerplate to read in the raw data:
# from uscourts.gov/statistics/table/b-1/…
cs = pd.read_excel('data/fjcs_b1_0331.2022.xlsx', index_col=[0], header=[2,3,4], skipfooter=2)
#   ignore UserWarning: Workbook contains no default style, apply openpyxl's default
cs = cs.replace('-', 0)
# cs.dtypes should now be all 'int64', confirm with cs.info()

# Some code to fixup names, whitespace
#cs = cs.rename(index=fixup_col_names, columns=fixup_col_names)
Anyone got any ideas how to handle it? How much effort I put into handling the column multiindex structure under 'Status'=='Terminated' will depend on how finely I might ultimately need to aggregate the level-3 values; so until I learn the significance of (say) 'Terminated-On Procedural Grounds-By Judge' vs 'Terminated-On the Merits-After Oral Argument', I'd like to keep all the level-2 and level-3 levels...
... But, it might be less coding grief to string-merge and rename the level-2 and level-3 levels, e.g. 'On Procedural Grounds-By Consolidation'/'...By Judge'/'...By Staff' could become -> 'ProcC'/'ProcJ'/'ProcS', likewise -> 'MeritsC'/'MeritsO'/'MeritsS'. I'm leaning towards that, at least while I do data exploration. Anyone got suggestions?
Here's a clearer PNG (why is my PNG rendering so small?). (Note: The blue rectangle is the table for All Circuits (which is just an aggregation across all Circuits ('DC','1st','2nd',...'11th'), so we could drop the 'All Circuits' table (rows 6..14) and reconstruct it by aggregating the 12 individual Circuits)
 
8:57 PM
@smci only the embedded preview ("onebox") is small, if you click through it's the proper size
 
Okay thx. Anyway, anyone got suggestions on how to handle such a non-uniform multiindex, if I intend to extract the data back to a cube, then do aggregations? (other than flattening/merging levels 2 and 3 of Status -> ['Term', 'ProcC'/'ProcJ'/'ProcS'..., 'MeritsC'/'MeritsO'/'MeritsS']?
 
9:16 PM
(Tried reposting the Excel screenshot with a 'h' suffix but didn't get 'huge' imgur thumbnail
 
I don't want a huge thumbnail. This size seems good to me. It's clear what it depicts, and whoever wants to look at it in detail can click on it. Its native size doesn't even fit my screen.
 
@AndrasDeak--СлаваУкраїні Fine. Its native size is perfect on desktops, which was the intent. Let's get back to the question
 
9:41 PM
Last and least important, the row-index 'Nature of case' names (cells A7:14) also implicitly encode another uneven hierarchical multiindex (although until I know which case Natures the analysis should focus on, I don't care to make it a hierarchical multiindex). Here's their meaning: "U.S. ..." means "federal", "Private" means "State/Local"...
...Hence I simplified the row-index labels to ['Criminal', 'Prisoner-Fed', 'Civil-Fed-Other', 'Prisoner-S/L', 'Civil-S/L-Other', 'Bankruptcy', 'AdminAgyAppeals', 'Misc']. I might further merge (or simply drop) 'Criminal' and 'Prisoner-*'; don't know yet.
In any case, pandas DataFrame is only 2D, it doesn't support data cubes, so either a) I munge into a list/Series of dataframes or b) write some really crufty non-reusable ad-hoc code (I'd rather not) or c) I use one of the third-party data-cube packages (recommendations?)
 
since your data are numbers you could technically use numpy... but managing all those labels would be a bit cumbersome
that might be your b)
 
10:23 PM
Hello fellow comrades!
 

« first day (4291 days earlier)      last day (673 days later) »