« first day (4898 days earlier)      last day (56 days later) » 

5:47 AM
@MisterMiyagi Yes that's specifically why I replied that, and gave an explicit example of with multiple different senses of NA. Yes NA is not equal to itself.
@roganjosh The idea has been floated in the R community on occasions of having multiple distinct values of NA, say NA_1..NA_32. Might have been some RStudio(?) guys. It was never standardized though IIRC. It would have been interesting.
You would have been able to get much smarter NA-aware behavior in say NA-aware decision tree algorithms; remember how C4.5/C5.0 briefly became popular after doing well on Kaggle Titanic and being declared "#1 in the Top 10 Algorithms in Data Mining" in one paper back in 2008.
(You can achieve the same effect by using integer sentinels (e.g. 999, -1) instead of multiple NA values; but R makes code handling NAs cleaner).
The IEEE 754 standard was not at all concerned with data-science considerations like multiple NA values, but if it was formulated today it might be.
Ah, here's part of the historical motivation: Stata and SAS [well-known proprietary statistics tools] have a system of tagged NAs, where NA values are tagged with a letter (from a to z). [26, 32, whatever]. [cran.r-project.org/web/packages/labelled/vignettes/…
Anyway, noone in R has ever standardized an implementation of multiple NA values [can't find a good article why], and C5.0 algorithm stopped being hot post-2016 and was overtaken by other things. But the general concept still has merit.
 
6:24 AM
@smci Just wondering what’s the point then. Even though float already technically has multiple nans they cannot act that way by definition. The entire crux of nan-ness is that one cannot filter by a specific nan-value. Do these extra NAs come with special functions to select them?
 
@MisterMiyagi Just because they don't compare equal with plain old ==, doesn't mean the standard can't define an additional non-default comparison operator where they do compare as equal, which any algorithm (e.g. NA-aware decision tree like C5.0) that wants to use NA values intelligently would use. Just like how Verilog defines a === as distinct to ==, where x === x and z === z). The main intent is to protect NA-type values from being treated as normal values in ordinary +/-/==/</> operations
Like I'm saying, IEEE 754 is overly concerned with almost every representable value must have a numerical value and fitting into some ordering relationship. Just like how it's an accident of history that classical applied math started out being more concerned with particles, projectiles, not with statistics, hypothesis testing, sampling, distributions etc.
Another example: numpy.array_equal(a1, a2, equal_nan=False), where the non-default behavior is NaN's compare as equal.
^^ Call it the 'identicality operator' or 'case-equality operator' (I think that's what Verilog calls it) or whatever.
@MisterMiyagi: hey as a matter of fact, has any language or package ever defined a type for an imprecise quantity, e.g. "If you sell 2000 +/- 50 items at a price of $4.00 +/- $0.25, how much revenue would you make"?
> Do these extra NAs come with special functions to select them? Presumably if someone provided an implementation, that would include an additional non-default case-equality operator.
(Fuzzy logic did try to formalize operations with imprecise quantities, but only for 0..1.0 boolean values, not floats or ints)
 
 
1 hour later…
7:57 AM
@smci I'm not sure what such a type would add but you could probably do that simply enough to draw from a distribution. But prices don't really fluctuate like that, you would want to look at price elasticity and bog-standard stochastics would have you covered in a bulk modelling sense?
Prof. Powell is all over my LinkedIn with his books on this topic and his company is rapdily growing
So, I guess modelling under uncertainty is becoming more popular these days given the huge culture shock of covid demolishing just-in-time manufacturing and supply chains
Speaking of linkedin, GCHQ has just joined and provided a visual puzzle if anyone fancies a distraction :)
 
8:16 AM
@smci Good point. Even for Python that works with good old is instead of ==. I'll have to ponder that, seems useful to know...
@smci I think there have been some questions around that over on langdev.stackexchange. IIRC the problem is that just the standard operators can't express the finer details of error propagation (namely correlation) so you are bound to naive/worst-case errors which quickly spiral out of control.
 
8:44 AM
An language example with [lower, upper] and [lower, best, upper] interval arithmetic is Frink.
 
 
2 hours later…
Cow
10:59 AM
Do you guys know, if I have a list comprehension in Python and check some data vs a unique set that is created inside the comprehension, if Python builds the unique set for every iteration or it saves it in memory or something? Example: [i for i in somdatalist if i[0] in {x[1] for x in someotherdata}] - Will it be faster to create the set outside of the list comprehension?
 
Yes, you should create it outside
 
Cow
@Aran-Fey thank you so much :)
 
11:29 AM
@Cow What Aran said. The set gets re-built for every if test. Here's a demo:
a = [i for i in range(9)
    if i in {(print(j) or j) for j in range(0, 9, 2)}]
print(a)
 
Cow
@PM2Ring Thank you so much for taking to time to write this :)
 
Using print() like that inside comprehensions is a very handy trick. :)
 
Cow
yeah that's pretty cool
 
11:57 AM
Yam, I was just going to suggest throwing dis.dis at that list comprehension - what a massive beast!
Does anyone know what the ExceptionTable of dis.dis encodes?
On 3.12.2 I get the following after the opcodes:
ExceptionTable:
  12 to 36 -> 98 [3]
  38 to 60 -> 88 [7]
  62 to 68 -> 98 [3]
  72 to 78 -> 98 [3]
  88 to 96 -> 98 [3]
Full dis output: dpaste.com/7WJ59M5A8
 
No idea, but for that I get something entirely different
ExceptionTable:
  24 to 58 -> 134 [3]
  60 to 96 -> 124 [7]
  98 to 104 -> 134 [3]
  108 to 114 -> 134 [3]
  124 to 132 -> 134 [3]
That's in 3.12.0 so I wouldn't have expected it to change wildly in minor versions?
I'm guessing they are character ranges?
 
I'm wondering if they are opcode indices, but that only makes little sense. For my dis that would be the list comp setup, set comp setup, the containment and append, and something about assigning x and i.
 
I think they are character indices in the code string since the intervals between ours are the same but I had the original 2-liner. When I run exactly yours I get the same output as you
 
Hm, the tracebacks now have those neat wiggly lines, that could fit to your idea.
 
12:27 PM
In the docs postgres says "Numeric values are physically stored without any extra leading or trailing zeroes. Thus, the declared precision and scale of a column are maximums, not fixed allocations." I'm trying to store Decimals from python
I'm first using a string representation of the value, which has varying numbers of values after the decimal point ( so, Decimal(str(some_value) like Decimal('1.234'). However, when I look at it in pg_admin, it appears it has padded the values out with 0s to the full 18 characters that the column could hold
Is there a way I can look at the size of items on a row level? I know the python version hasn't done the expansion but I can't decide whether this is just some artefact of the way pgadmin displays Numeric values or whether that padding is real
Only trailing 0s, though. Not leading 0s
 
@MisterMiyagi Pretty sure it means "If an error occurs between opcodes 12 and 36, jump to 98". You can see how every opcode that can be targeted by a jump is marked with >>, and nothing except for the exception table jumps to 88 or 98
Absolutely no clue about the numbers in square brackets though
 
Make sense, thanks.
 
(Also no clue why it sets x and i when an error occurs)
 
On a totally unrelated sidenote, does anyone have pointers for class naming conventions of multiple acronyms? I've got an API for a thing called HPSS specifically via PFTP, but I don't really want to call the class HPSSPFTPAPI...
 
Something like HpssViaPftp maybe. Or demote the PFTP part to an implementation detail and just go Hpss
 
12:47 PM
There's going to be other variants, so I can't just drop the PFTP part sadly. :/
 
How about a different design then, something like Hpss(via=Pftp())?
 
You mean hiding it behind some abstraction? That could work, actually.
 
Yeah, basically a strategy pattern
 
@roganjosh it really is being padded out to the full 18 decimal places. Checked when reading back in to a Decimal: Decimal('1.168738538200000000'). Using setcontext()from Decimal on the python side to limit the decimal places before writing to the DB also doesn't help :'(
 
1:06 PM
Hmm, ok, it doesn't seem that bad. Although it's visually represented with the padded zeros and it comes back out of the DB into a Decimal with the full padding, using a modified version of this seems to suggest the physical storage itself on disk indeed doesn't have the additional zeros
 
 
1 hour later…
2:14 PM
hi all, I'm new to Python, and I was wondering about "nested methods"
 
Please note that we have a guide for formatting code in chat
 
thank you, just what I needed
 
:) No worries. Everyone struggles at first because it's anything but intuitive
 
import numpy as np

A = np.array([[2, 5], [3, 3]])
B = np.diag((1,2))

C_1 = np.dot(np.linalg.inv(A),B)
C_2 = np.linalg.inv(A).dot(B)
I want to compute the product of $A^{-1}$ with $B$
the way that I'm familiar with is how I computed C_1
however, I've seen someone else compute the product by C_2
I don't really understand that syntax
I tested if the following would work, trying to pattern match the syntax from C_2:
A_inv = np.linalg.inv(A)
C_3 = np.A_inv.dot(B)
however, this is not valid syntax
I looked in my Python book and I tried googling "nested methods Python"
however, I couldn't find anything
could someone explain the syntax to me, or ideally, point me to a source which explains this?
 
Have you tried replacing the entire thing you assigned to A_inv with it?
 
2:25 PM
ah, without np you mean
 
There is one specific part that you left out
 
I haven't, let me check
 
Yeah, np.
 
oh, how silly, it works
 
There's a few things going on here
 
2:27 PM
I guess it would be easier to explain on some simpler object than numpy arrays
 
sure, a different example is fine :)
 
np.dot() is a method which you can see because of the () after the name. Something like np.nan is an attribute and can be accessed via the . (so, print(np.nan). Both of these (the method, and the attribute) are defined within the numpy package itself without you having done anything
 
aha, I wasn't aware of attributes
 
However, you are doing np.A_inv. on the last line which suggests that you make the assumption that A_inv = np.linalg.inv(A) has created an attribute on np, which you haven't. The previous line is evaluated but it doesn't mean that the result is automatically accessible on the module level
 
right, thank you, it's starting to make sense. I will have a look in my book where it talks about attributes, and then I'll reread what you wrote :)
 
2:33 PM
numpy is actually a bit more complex than other libraries here and I can see where you got this idea. If we just think about sum() as the most basic kind of arithmetic operation, it provides you two ways of doing this. The first is a method on the module level np.sum() and second is a method on the array object itself.
arr = np.array([1, 2, 3])
print(np.sum(arr))
print(arr.sum())
 
usually, if you assign something to a variable, like np.linalg.inv(A) to a variable (A_inv), you should replace that entire thing with this new variable further down the code.
 
I say "on the module level" but I wonder whether I should really say "on the package level". The method is actually buried in the library but keeps getting hoisted up to the top level through multiple __init__.py files importing it
 
@matszwecja that is fair. my reasoning was that we needed to include np because we were using np.dot
 
You are not using np.dot in this case. You are using method of an A_inv array called dot.
 
aha, that's what roganjosh meant
 
2:37 PM
BTW, in Numpy, to solve a matrix equation of the form $ax = b$ it's better to use linalg.solve It's more efficient than doing the matrix inverse & multiplying.
 
thanks for your help! I wasn't aware of this distinction between methods on the module level vs on the object level, but it's clear now :)
 
Well, np.dot is a function, but A_inv.dot is a method of the A_inv object.
A method is a function which is an attribute of an object.
 
In the first case, dot is a standalone function with 2 arguments, like e.g. max in max(1, 2). Second dot is like capitalize method in "foo".capitalize()
 
@PM2Ring ohh! right!
okay then it's clear. I wasn't aware there was both a dot function and method
 
@ShaVuklia it is something like this
def my_sum(vals: list) -> int:
    accumulator = 0
    for value in vals:
        accumulator += value
    return accumulator


class MyObj:

    def __init__(self, vals):
        self.array = vals

    def my_sum(self) -> int:
        accumulator = 0
        for value in self.array:
            accumulator += value
        return accumulator
In reality, I suspect that MyObj.my_sum() would probably just call the function my_sum rather than redefine exactly the same logic, especially as they would be liable to get out of sync if you ever needed to change the logic (i.e. you decide returning a float would be better in the future). But it illustrates that they are separate things and could be defined differently
 
2:46 PM
right, yea I should have a look at classes to better understand methods. that's on my to-do list :)
 
So when you call the np.dot function you have to pass it the two vectors (or whatever) that you're dot-producting. But when you call A_inv.dot() you only pass one arg because the object itself is the first arg. Generally, Numpy methods are also available as functions. But sometimes there are slight differences in additional args that may be given.
@roganjosh I'd say "on the module level". When you're using a library you shouldn't need to know or care how it's packaged.
Apart from knowing when to use np.linalg rather than plain np. ;)
 
In the past where I've had some really tight loops, the array.sum() method is quite a bit more performant. IIRC I even tested it against function_local_sum = np.sum before the loop and it still won out but that was a long time ago
@ShaVuklia if you haven't seen classes before then I guess I should fill in the last step to show how my code block is analogous to what you're doing with numpy
arr = [1, 2, 3]

print(my_sum(arr))

my_array = MyObj(arr)
print(my_array.my_sum())
 
3:06 PM
makes sense!
thank you all for the elaborate response. it is reassuring to know that chats like these exist in case i get stuck in the future again
 
3:44 PM
Huh, unexpected highlight of my day. I wanted to remove the CSRF token expiry time because it causes lots of silent user issues if a form goes stale. I expected to read an answer on Information Security about what a terrible mistake this is and that I should burninate every cache and the HDD on every expiry to be safe. Turns out, apparently it's fine!.
 
3:56 PM
One CSRF token per session? How is that different from the session token, then?
 
I guess because the token is embedded into the page and not in the session itself. If you look at the top answer they actually ripped a load of extra stuff out
 
4:08 PM
I had to re-learn how CSRF works, but now it makes sense. You could even use the session token as the CSRF token, which is pretty funny
CSRF is such a weird problem to have, honestly
 
It's even weirder that it comes in Flask with a default timeout and there are plenty of answers where people are actively asking how to reduce the default timeout. I think it's the first (and probably only) default setting on some security measure where I've found I actually can remove it and not feel like a terrible, lazy human being by not wrestling conflicts between multiple timeouts
 
Finally a dev who solves timeouts properly! There are far too many websites that just break if you let them sit for a couple of hours
"Link has expired" Well then fetch a new one, you lazy piece of tech
 
4:24 PM
Just read through CSRF token description as well, and honestly this sounds a lot like the refresh token versus access token divide.
 
Hahaha, I didn't even get to trying that lazy band-aid. I was testing against loads of different combinations of inputs on a large form and then started hitting 400s, which at that point would be totally silent to the user. I'm obviously still active in the session but because I'm using AJAX and didn't complete the form-submission-loop, the form token expired while my session was valid
 
 
5 hours later…
9:54 PM
@PM2Ring , MisterMiyagi, thanks for the reading links
 
10:25 PM
@roganjosh I wasn't advocating for it especially, I was simply commenting on IEEE 754's attitude to f-p values: it insists that all values are crisp and have a well-defined order relationship wrt. each other, so 3.9 < 4.0 But if you have interval arithmetic, then (2 +/- 0.1) * (1.8 +/- 0.3) < 4.0 is no longer a crisp 'True' value, it's a probability distribution with an expected value below 1.0. It would be possible to define an IEEE 754-type standard: [value, optional precision]
 
This raises lots of questions for me that have been in the back of my mind in some way
I will take an IEEE definition of nan in a strict sense and I can reason about why that is useful. Once you get into the uncertainty that you're describing then I think that prescribed rules don't really apply. If I choose stochastic approaches, are they going to shoot me? Is another user of my library going to shoot me with an IEEE justification?
 
wrt my comment about NAs: NA_1..32 could have a user-defined hierarchy on their equivalency relationships. e.g. survey of political affiliation, some of the NA types belong to the set "respondent was overage, eligible to vote, lived at this address and gave an answer to the enumerator", and other of the NA types don't ("did not answer phone/door" or "bad address").
@roganjosh I'm not beating up on IEEE, I'm just saying that the standard is a product of its time and treats NA like a fairly trivial afterthought which only carries one bit of information ("all NAs equivalent"). Anyway, standards evolve very slowly. Been looking but can't find which R packages tried out the concept of multiple NA values.
 
Sure, I didn't think you were bashing the standards
 
11:02 PM
Here is a deferred Numpy Enhancement Proposal from 2011: NEP 25 — NA support via special dtypes where the second option would support multiple NA values.
Also, Numpy has masked arrays. But not supported by base Python, and when cast to pandas all masked values simply get hammered to NaN.
And a good review of discussions and proposals up to 2012: NEP 26 — Summary of missing data NEPs and discussion
 
11:36 PM
Some R packages which support "tagged" or "labelled" NAs imported from STATA or SAS are haven and labelled
 
11:47 PM
As well as "tagged NAs" NA(a)..NA(z), they also have user-defined NAs which they call "user NAs".
 

« first day (4898 days earlier)      last day (56 days later) »