I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. Please refer below details:
This is the input data:
d={'Name': ['Ankan', 'Shiv', nan, 'Sandeep']...
Turning strings into functions is tough even when pandas isn't involved
Besides eval(), there's a "dispatch" style solution where you store a dict that maps strings to functions, e.g. {".isna": pandas.DataFrame.isna, ...} but this is only practical if there are a small number of functions that can be called
Perhaps you could create a whitelist-based sandbox by first examining the rule with ast.parse before you call eval on it. That would improve security, if such a thing is important
If you have any leeway on the requirements, perhaps you could allow only one rule per row.
So instead of
Column_Name Rule
0 Name .notna()
1 email .notna() & .str.contains('foo')
You could have
Column_Name Rule
0 Name .notna()
1 email .notna()
2 email .str.contains('foo')
Ok, without ampersands, the basic outline of my sandbox idea is: take the rule, add "my_dataframe" to the front, and call ast.parse on it. Then walk over the node structure and verify that the expression looks like what you expect
but at least you could have [['notna']] for row 0 and [['notna'], ['str', 'contains']] for row1 (and store the arguments somehow in a sane and safe way), meaning there's only one filter for the first row and two filters (& or | has to be assumed I'm afraid)
I a totally in for eliminating the & let them say, I wouldnt bugde, I will see if I can agg the conditions together, now that we have .explode().. haven't explored all possibillities
In related news, this Q&A is a hard-to-notice train wreck. I have no idea how to salvage it. The two answers answers different questions, and OP accepted the wrong answer (as the question stands) with a comment that it's not what they need.
@anky_91 having either all & or all | is fine in my suggestion, because you just have to loop over the list of lists, and gradually apply each list (which means a single lookup like .str.contains) and take, say, numpy.logical_and.reduce or whatever (or just reduce them in a loop).
The problem with my approach is that you can't easily have function calls inside your condition, only at the end, or the logistics of separating methods and args can probably get tedious
I just started (still working on it) creating a super simple library for common tasks in python 3, would really appreciate any thoughts: https://github.com/agamm/flick
I've got one foot out the door so I only have time to look at the README. Some interesting features there. diff seems particularly interesting. memeory and craete are typos.
@roganjosh, hmm I just rewrote the beginning of the README with possibilities. Basically 2 main usages, one for small scripts where you need to write fast and it will help you abstract many common stuff we do in python 3. The second thought was to let users copy implementations of specific functions for production usage thus not adding another dependency.
PS, I just found out that someone took the name flick 4 days ago (!) in PyPI, so I renamed to https://github.com/agamm/eze
Hmm yeah, you are right, I just found that I need to `json` something and it spits out a str, but a byte which in most cases isn't what I need, plus it is shorter to write this way: `f.json(something)` instead of: ``` jsonlib.dumps(something, ensure_ascii=False).encode("utf8") ```
Hmm, yeah I can add one, but I'm using the traditional json lib, so it won't make any difference. (I didn't implement it)