« first day (2808 days earlier)      last day (2132 days later) » 

4:33 AM
cbg
 
cabbage
 
editing my answers from earlier when I was traveling (-:
 
 
1 hour later…
6:01 AM
What is a good way of checking the integrity of an JSON blob? To verify that all the fields I want are there.
 
In python, probably trying to get them all as if they are all safely there with appropriate catch clauses.
(that's for existance of fields, it's not neccessarily going to help from a security perspective)
 
Even if it's a complex data model I'm expecting? There's no "nicer" way to do it?
 
Well, think about it. If the minimum possible time to parse a full blob is the time when it's correct
 
In what way would security have to be considered?
Yeah I guess, I just figured there would be some json utility function for this
 
The only way you can get better performance is to use some cheaper system for guessing that it's not correct (which is highly dependant on the expected structure)
but in all cases that it IS correct, you're actually adding to the runtime by double checking
So it depends on how often you expect it to be wrong really
you could write a regex specific to your structure, probably.
But that will process every character in the string anyway
 
6:15 AM
Ideally it would never be wrong because I will be the only one sending the json blob, so no need to do an overkill. I think I'll use what you proposed, thanks
 
So you may as well break the data up as normal in a structural parser, and defer processing the actual values until after you've parsed it
 
Cabbage
 
as for security, I'm not a security expert but you know, never eval(external string), never directly use strings etc in a raw SQL statement, use whatever mechanism your library has for safely pairing parameterised queries with values, etc
 
@simeg Maybe I misunderstand your question, but I don't see the problem. json.load can read from any file-like object with a .read method, and it won't return until it's decoded a complete object (or raises an Exception caused by malformed JSON).
 
@PM2Ring I want to verify that all the keys in my model are there and that they have a value
 
6:25 AM
@simeg you should probably also verify the value-type (particularly if you have to drill down recursively)
 
@simeg Ok. Does the JSON decode to a simple flat dict, or does it have a nested structure?
 
@Cosmo yes, good idea. @PM2Ring it has a nested structure.
that's why I figured it's annoying to do nested ins
 
you could possibly parameterise your model into your code to make it cleaner (and then when you reach each key, just call a generic check based on the expectations for that key) e.g. uh
 
@simeg Nested dicts makes it a little trickier. I assume that you want to verify that you have all the expected keys, and that they're nested in the right place. But what do you mean by "that they have a value". Of course they'll have a value, otherwise the JSON is malformed, and json.load will raise an Exception.
 
@PM2Ring yes that's true. To clarify: what I want to verify is that it's a value of the type I expect and that it's not empty.
so empty string for example, is still a valid JSON value but I don't want to accept empty values
 
6:38 AM
@simeg Ok. This sounds like an interesting problem. But give me a little while to have a coffee & think about it. :) In the mean time, take a look at stackoverflow.com/a/50090400/4014959 and the other answers I link at the bottom of that one.
 
expected = {'a':{b:int,c:float}}
def verify(obj, key, expectation_map):
  if type(expectation_map[key]) is dict:
    return type(obj) is dict and all(lambda subkey, subobj: verify(subobj, subkey expectation_map[subkey]), obj.items())
  else:
    return type(obj) is expectation_map[key]
the all() on the fourth line can be extracted out so that you only recurse as you parse, efficiency is kind of similar
Also that code has typos lol
 
Wow thanks a lot for the help people, I really appreciate it. Worth mentioning is that I use Python 3.6.
@Cosmo what is the expectation_map in your code? Is expected what you pass in as that param?
 
And you can add any other complexity you want to that verify function as long as you can map it
it's a recursive structure that roughly maps to your json object, except that leaf-node values are replaced by something that reflects the validity of the value
e.g. they could instead by lambdas
such as
lambda val: val and type(val) is list
(note that [] == False)
that lambda would return true only for non-empty json arrays (unless my python is rusty, I don't use type() much)
 
@Cosmo: bool([]) is False, but [] != False.
 
@user2357112 oh. Okay then.
in that case, I'd use len(val)
expected should by used as the expectation-map at the highest level, and then at the next level, expectation_map[key] should be used @simeg. Note that my code wouldn't work as-is for several reasons (the all().. would need to be used at the root for example) but that should give you an idea as long as you know how to implement things recursively etc'
 
6:51 AM
I like both examples you provided, I'll have to think about which one to use
 
alrighty
 
I was thinking of doing something very much like Cosmo's code. Except I wouldn't use a lambda in that all() call: it would be more efficient to define a proper def function outside of verify() that takes subkey, subobj and expectation_map as args.
To check for the presence of a single key somewhere in a nested dict / list, you can use this code that I wrote a couple of weeks ago, with some improvements suggested by Kevin.
def has_key(obj, key):
    if isinstance(obj, dict):
        if key in obj:
            return True
        values = obj.values()
    elif isinstance(obj, list):
        values = obj
    else:
        values = []
    for v in values:
        if has_key(v, key):
            return True
    return False
 
7:07 AM
@PM2Ring yeah I only used the lambda for brevity of example
 
lambda is ok there, but a new lambda gets created on every recursive call, which is a bit wasteful if there's a lot of recursion going on.
 
ah
 
7:34 AM
That applies whenever we define a function inside any kind of looping code. When the interpreter encounters a function definition, whether it's a lambda or def, it has to execute that definition to create a new function object, which can then be called. Eg,
for i in range(3):
    f = lambda x: 2*x+1
    print(i, f(i), f)
# output
0 1 <function <lambda> at 0xb721053c>
1 3 <function <lambda> at 0xb7190c8c>
2 5 <function <lambda> at 0xb721053c>
The memory gets recycled, so the id number is reused, but that 3rd lambda is still a new function object, it's not the same as the 1st one. And of course one shouldn't normally bind a lambda to a name. I only did it in that code so I could get its id.
 
yeah I've done this on purpose before, closures etc
ah in fact my code did use the closure-capacity
 
7:49 AM
Big thanks to both of you, this is more than enough for me to solve my problem :)
@Cosmo & @PM2Ring ^
 
yw! c:
 
@Cosmo Yep. That's why I said the external def version would need to take expectation_map as an arg.
 
yeah, the lambda did initially but I realised that all() didn't work that way haha (it'd need to take a tuple or something which would be like a wrapper function anyway)
 
8:48 AM
Another question: I've defined my type using the native typing library, now how can I construct an object and make sure I have the correct types for all the fields? Can't find anything on google..
CustomObject = {
  Tuple[str, int],
  Tuple[str, str],
  Tuple[str, Optional[List[Tuple[str, str]]]]
}
Something like this
 
class CustomObject:
   attr1: Tuple[str, int]
   attr2: Tuple[str, str]
^ like that?
 
Maybe I've misunderstood the capacity of the type system in Python, but in a static typed language I would expect this to be trivial
oh I'd have to create a class for it?
 
Well, you can add annotations to an instance, but no existing type checker will respect those
 
Hmm, ok. Then I've misunderstood it. That explains a lot :D
Then I know what I need to do, thank you
 
9:18 AM
@simeg you can dynamically attach attributes to objects but if you want to define stiff statically you're options are basically literals (for tuples, arrays, dicts, etc) or as static-attributes on classes like @Aran-Fey's example. Am I correct btw, @Aran-Fey?
 
 
2 hours later…
11:23 AM
hey
 
 
3 hours later…
2:20 PM
Sunday Cabbage =)
 
3:07 PM
A bit late, sorry :D
 
3:20 PM
:(
 
Hey, 2018 isn't over yet!
 
PEP 572 missing
 
I couldn't find Guido's reaction to that PEP, so I guess we'll need to wait and see
But yes, if that PEP and that \other PEP get accepted, they'll be put in the center of the image :D
 
No explicit endorsement, just taking it seriously
 
3:55 PM
cbg
 
cbg @3141 You might find this interesting: chat.stackoverflow.com/transcript/message/29424913#29424913
 
 
3 hours later…
7:17 PM
cbg
 
cbg
 
Can users like their fresh meta questions here?
 
they can like whatever they like
 
lol
 
ffs
I can't fix that now
 
7:27 PM
as long as it's relevant to the room I don't see why not to link
 
Nah, it isn't relevant. But it's a bit weird. The weekly rep tally hasn't reset for the new week for some reason.
It's already Sunday, but the tally from the previous week has carried over.
 
Did it change by this time in previous weeks?
And do weeks start on Sunday or Monday? I presume "yes, it usually changes by now" and "week starts on Sunday" but I want to be sure
 
Yup. It's always been sunday thus far
unless I've been mistaken for the last ~50 odd weeks
 
OK
so etiher that or caching ;)
 
Caching on their side? Possibly.
 
7:34 PM
but I consider you an authority when it comes to looking at rep
 
thanks
 
7:50 PM
@vaultah and that important core dev is RH.
also... if only Python would have interfaces
 
RH negative in this case
 
we wouldn't have this silliness of __yet_another_dunder_method__
 
is there a one-sentence explanation of what an interface is or should I google it?
 
@AndrasDeak the kind of Zope interface or Java interface...
 
i.e. the latter :D
the first google searches make it seem like it's an abstract base class that implements some kind of API
 
8:05 PM
basically an abstract base class with only abstract methods, no actual implementation
 
so very abstract
 
yeah. The sole purpose is to specify the interface that subclasses must implement
 
OK, that's what it came across as
 
Python: If it walks like a duck, quacks like a duck, looks like a duck, it must be a duck.
Java: If it's a duck, it must be a duck.
 
it sounds more like "You must be at least THIS tall if you want to be a duck"
 
8:10 PM
True, you're allowed to add extra features to your duck. Java doesn't care if your duck has a few extra limbs
 
8:35 PM
Hi Guys
Greetings from Germany
 
hello
 
I have to solve a problem, but
1. I do not know how to describe it
2. I do not know how to extract the problem from my big code so it makes sense for someone to watch the whole code
what would you suggest? @AndrasDeak
 
I suggest not pinging users without prompting
if you can't solve #1 first there's no way anybody else can help you
Then I suspect nobody will help you at #2 for free unless you can boil down your problem to something small. But for that you need to start localizing the problem and its possible sources, which comes back to #1 anyway.
 
I'll say it that way. How can you explain a problem in such a way before you begin to present the whole project?
 
Before you explain it you need to understand it first.
If you understand it you can think of what is necessary for others to understand it. See also all the content out there about asking good questions, e.g. codeblog.jonskeet.uk/2010/08/29/writing-the-perfect-question
 
8:46 PM
I will read the blog post, ty
 
A temporary workaround could be to show the problem to a coworker, but in the long run you really need to be able to understand and describe your problems
 
OK, let me know if it helps with your general situation
 
9:41 PM
object oriented programming was a mistake
 
bold statement. Are you swimming in Java right now or something?
 
10:00 PM
yeah... checked in some 1k lines of code this week
and that's the last I'll have to deal with java for the rest of the summer
 
How much did you have to type by hand?
 
Most of the non-test, and test mocking code
so, like ~700 of them
the rest was copy pasting test cases and tweaking stuff for different scenarios
of course, working with intelliJ means tab completion. So that helped
 

« first day (2808 days earlier)      last day (2132 days later) »