« first day (3254 days earlier)      last day (1919 days later) » 

18:00
other kids are busy playing PUBG here in India ;)
18:24
thank you
Which one? There are a couple questions on that page
OOPs my bad
2
Q: Data profiling in a dataframe based on an input file-pandas

anky_91I have a data file and a rules file. First column of rules file will have columns from the data file and the second column has an pandas operation which has to be performed on the respective Column. Please refer below details: This is the input data: d={'Name': ['Ankan', 'Shiv', nan, 'Sandeep']...

this one
you have had a similar look before @Kevin :)
Turning strings into functions is tough even when pandas isn't involved
Besides eval(), there's a "dispatch" style solution where you store a dict that maps strings to functions, e.g. {".isna": pandas.DataFrame.isna, ...} but this is only practical if there are a small number of functions that can be called
Hmm,, pd.eval is safer so I am using it thus far, though I have a bad feeling about it of something which I might not have explored
@Kevin but it's also safe
18:37
dispatch gets even harder if you want to be able to accept different kinds of arguments, or chain functions together
At some point you're basically just writing your own language parser
My requirements would have something similar
      Column_Name                                          Rule
    0        Name                                      .notna()
    1       email  .str.contains('[^@]+@[^@]+\.[^@]+',na=False)
@Kevin i was inclined for a VBA solution originally , but they didnt agree
How about changing your requirements a bit so that you can use getattr rather than hard-coding entire function calls?
there is a new interface where we are hosting this
this reminds me a lot of an eval-y question from today...
@AndrasDeak i was trying to do that
I failed
:/
@AndrasDeak i see
i wasnt the one :P
18:42
Perhaps you could create a whitelist-based sandbox by first examining the rule with ast.parse before you call eval on it. That would improve security, if such a thing is important
@anky_91 I'm aware
oh yeah, it had a bounty :/ stackoverflow.com/questions/57740110/…
The tricky bit is the rules that you're combining with "&", since it's not entirely easy to split them up again
mycond = Conditions({"(m1)" : "(lev1 < 0)",
                     "(m2)" : "(lev2 > 2)",
                     "(m3)" : "(lev1 == 0)"},
                    ["(m1)", "(m2) & (m3)", "(m2)"],
                    ['A', 'B', 'C'],
                    999)
Can't just use rule.split("&") because what if a regex contains an ampersand?
I was wondering if numexpr could help OP but it just seemed like too much of a wasp's nest to get involved with
18:44
@Kevin if not would that be easier?
If you have any leeway on the requirements, perhaps you could allow only one rule per row.
So instead of

      Column_Name                            Rule
    0        Name                         .notna()
    1       email  .notna() & .str.contains('foo')

You could have

      Column_Name                  Rule
    0        Name              .notna()
    1       email              .notna()
    2       email  .str.contains('foo')
i thought the same actually then perhaps agg them, for now think there is no &
Ok, without ampersands, the basic outline of my sandbox idea is: take the rule, add "my_dataframe" to the front, and call ast.parse on it. Then walk over the node structure and verify that the expression looks like what you expect
Let's see if I can make a little sample...
Thanks that would help :)
I've got a partial result for a potential path which is probably not even in the game
>>> functools.reduce(getattr, [df, 'a', 'str', 'contains']) == df.a.str.contains
True
18:53
@AndrasDeak woah
which might make it possible to store attribute names instead, and the arguments of the call separately...maybe
interesting
but this would probably imply that you'd be storing lists of lists in your dataframe cells, which is ew
i will explore that and let you know sir :)
@AndrasDeak i can manipulate my cell contents
if it fits
:)
but at least you could have [['notna']] for row 0 and [['notna'], ['str', 'contains']] for row1 (and store the arguments somehow in a sane and safe way), meaning there's only one filter for the first row and two filters (& or | has to be assumed I'm afraid)
18:56
I a totally in for eliminating the & let them say, I wouldnt bugde, I will see if I can agg the conditions together, now that we have .explode().. haven't explored all possibillities
In related news, this Q&A is a hard-to-notice train wreck. I have no idea how to salvage it. The two answers answers different questions, and OP accepted the wrong answer (as the question stands) with a comment that it's not what they need.
@anky_91 having either all & or all | is fine in my suggestion, because you just have to loop over the list of lists, and gradually apply each list (which means a single lookup like .str.contains) and take, say, numpy.logical_and.reduce or whatever (or just reduce them in a loop).
The problem with my approach is that you can't easily have function calls inside your condition, only at the end, or the logistics of separating methods and args can probably get tedious
hmm, i understand that , but thanks for sharing that, probably I can use it, yet to test.. I will do so Monday when I have access again on dremIO
sure, have fun
trust me having a lot of fun ;)
Here is a tiny prototype. It's absolutely not a fully enclosed sandbox, but this is the approximate shape of the approach.
The bad guys can still erase your hard drive with a rule like ".str(delete_hard_drive()).contains()", for instance
19:04
thanks for taking our time out @Kevin , really appreciate it both @Kevin and @AndrasDeak :) i will play around a bit with your suggestions
@Kevin ha ha I will provide them a disclaimer
i anyways advised for a powerquery for data transformation and vba and powerbi
but its thier call
apparently that tool supports pandas better
so lets see
:)
19:16
rbrb
19:52
I just started (still working on it) creating a super simple library for common tasks in python 3, would really appreciate any thoughts:
https://github.com/agamm/flick
I've got one foot out the door so I only have time to look at the README. Some interesting features there. diff seems particularly interesting. memeory and craete are typos.
@Kevin, thanks! I'll try to implement them soon + tests. Thanks for the proofreading too, I'll fix them now :)
20:19
If the project goes to version 1, what do you you evisage for its usage? What are you shooting for?
@roganjosh, hmm I just rewrote the beginning of the README with possibilities. Basically 2 main usages, one for small scripts where you need to write fast and it will help you abstract many common stuff we do in python 3. The second thought was to let users copy implementations of specific functions for production usage thus not adding another dependency.

PS, I just found out that someone took the name flick 4 days ago (!) in PyPI, so I renamed to https://github.com/agamm/eze
I just skipped the README and went to the code, sorry
@roganjosh no need to be sorry, I really do appreciate any thought. Anything you would change/add?
20:37
While I was scanning the code, this stood out for me
@roganjosh what stood out exactly?
Deserializaion is covered in many ways already
And your example doesn't use any nested structures
Hmm yeah, you are right, I just found that I need to `json` something and it spits out a str, but a byte which in most cases isn't what I need, plus it is shorter to write this way:
`f.json(something)` instead of:
```
jsonlib.dumps(something, ensure_ascii=False).encode("utf8")
```
Hmm, yeah I can add one, but I'm using the traditional json lib, so it won't make any difference. (I didn't implement it)
In any case, I'm impressed that you went through the whole process of getting your package registered. Nice work :)
21:02
Thanks @roganjosh, I'll update here when I get to 1.0.0 :)
21:39
/@funerr probably missed some context here, but also check out demjson?
We just watched IT (2017). I was rooting for the clown all along.
21:57
cabbage
@ReblochonMasque yeah, unclear or no MCVE maybe

« first day (3254 days earlier)      last day (1919 days later) »