« first day (4058 days earlier)      last day (907 days later) » 

4:03 AM
@holdenweb thanks, I will ask myself that question the next time I try to sneak in pattern matching :D, also cbg guys
 
4:47 AM
Hey all ! I am new here
 
5:06 AM
Cbg
Don't know how I did not pick this up, for my password manager, I hashed the master password and appended it with salt and put it into the database, little did I know that this master password hash that is used to encrypt the individual passwords stored in the database can be actually acquired if you get hold of the database file very easily and decrypt the password. The only solution here that I can think of is to use a encrypted SQLite for python, or I am missing some logic here. Any ideas?
 
5:36 AM
uh
something is wrong here. you're "encrypting" passwords (ie with a corresponding decrypt) or "hashing" the passwords
 
hashing the master password but encrypting the individual passwords(like for facebook, etc..)
 
i suppose password managers do encrypt, not just hash but hmm..
in that case, instead of me* speculating, i'd say maybe explore what keepass does
 
Yea let me do some more research
 
yeah, i dont know much about this area, but i know for a fact keypass uses a local db concept, so it should hopefully be close to what you're trying
I should also add, if you're doing this for learning: great. but in general, if you're building this for actual use, security related stuff should be delegated to tried and tested libraries. If you do it yourself you will get it wrong.
 
Oh, this is just for learning, nothing else
But at the same time, I do use tested libraries like pycryptodome for encryption and pbkdf hmac for hashing
 
5:50 AM
those names are jargon to me, so im presuming they must be good and tell you to keep it up :P
 
But I am searching for a encrypted extension for encrypted SQLite that is safe and at the same time has python binding, no luck yet
 
you know, i just realized, maybe the trick is, your salt and hash the master password and store in db, but you decrypt using the master password?
 
Let me give that a try then :)
 
i'd still recommend exploring what keypass does though, this is just me speculating
 
I did have that thought before sleep but then the key for AES(in my case) has to be 32 bytes long or something but this will keep varying between passwords aight. So I do not think it is possible, I'll look into some keepass videos meanwhile
 
6:12 AM
i suppose assumption check then...are you encrypting sqlite db?
 
Nope, the db is query-able by anyone who gets hold of the file
 
actually i think that must be the gap then. why does someone having access to the db file make everything fall apart? its not encrypted is it
ok makes sense, i think thats the bottom line issue
 
Hmmm yea same, all the research shows to store the hash in the database too, so maybe I need to encrypt it. I suppose I have to keep looking for a secure encrypted SQLite with python bindings
github.com/leapcode/pysqlcipher is something but not updated in 4-5 years
 
naive question on my part, wont any library that can encrypt arbitrary files work?
 
Wouldn't that still make the database openable, maybe by using a db browser?
 
6:29 AM
well no, you only create the encrypted file on disk
at least, i dont think so
so you essentially switch to using the decrypted database purely in memory
that encrypted file should be impossible to open until decrypted
 
Let me get this straight, the problem is that people can get the hashed master password out of the db, and decrypt the other passwords with that?
 
hey guys, hope you are good
I wanted to know what exactly is a "CPU bound" operation, in terms of asyncio
should async function only have an IO call and no other stuff inside (json manipulation, validation)?
right now CPU bound to me is some huge calculations, should I also include json manipulation as CPU bound? renaming keys, checking schema those kind
 
Depends on how much of it you do. If it uses 100% of your PC's processing power, then it's CPU bound
 
6:44 AM
how do I know that?
 
Maybe think about it this way: asyncio doesn't give you any additional processing power. Multiprocessing does.
@Jake Look at the task manager?
 
@Aran-Fey yep thats right
@ParitoshSingh Hmmmm let me give that a try then
 
So that means you're encrypting the passwords with the hash of the master password, then?
 
@Aran-Fey lol ok, I am using fastapi and I was wondering if I must move my functions to async or not, all that my function would do is receive a request validate it, then make a post to a different api
 
The point of hashing a password is that hashing is a non-reversible operation. So even if someone gets access to the password hash, they still don't have the password. If you use the hash to encrypt the other passwords, then the hash effectively is your master password and you have a security hole. Feed the master password into a key derivation function, and use the output of that for encryption.
@Jake If you're doing network requests, your first thought shouldn't be CPU power, it should be network latency. Asyncio is never a bad idea for networking.
 
6:51 AM
got it, I am just confused with the whole "cpu bound operations block your event loop" thing
anything in a computer uses CPU so I was wondering :D, but I guess the task manager suggestion would do
so at times I get in my output something like "took 5.006 seconds", I guess this only happens if I block the loop?
 
@Aran-Fey Oh, what I do now is, I feed the password into pbkdf and then get that hash then encrypt with it
 
That's not a hash. Don't store that in your DB.
 
Oh okay, can you elaborate other approach?
 
You need to generate 2 things from the master password:
1) A hash. This is stored in the DB, and used to verify that the user entered the correct master password.
2) A derived key. This is used to encrypt/decrypt the other passwords.
 
Oh wow, that makes sense, ill give it a try right now
 
7:21 AM
@Aran-Fey Seems like __init_subclass__ would be useful to switch on/off the "this method is required" check. You might also have success using two Parent class, in which the public concrete class does a switcheroo with the private abstract class when subclassed.
 
Tbh I don't know how to do either of those things. How can you turn off that check? And how do you implement that switcheroo? My best guess is __mro_entries__?
Not that it matters anymore, I've lost sleep over the design and decided to roll back the major refactor I started yesterday...
 
FWIW, I don't think the design sounds unreasonably insane. :P
 
I honestly can't tell anymore haha. I realized that Andras' suggestion to use a factory function would indeed have worked, I just thought it wouldn't because my brain is overloaded
 
nothing like a good night's rest to give fresh perspective on issues
 
Even a terrible night's rest helps, apparently :D
 
7:32 AM
Is using pbkdf_hmac('sha256',...) same as using just sha256()?
 
No. Can I explain what the difference is? Also no.
 
No probs, as long as its not the same, its fine
 
Alright, so one's a key derivation function, the other's a hash function. What's the difference? The KDF allows you choose how many iterations it should do (i.e. how slow it is, to help against brute force attacks) and how many bytes you get as output
 
@Aran-Fey oh yea, thats true. Also using the approach you mentioned, there is no use of storing the salt right, salt from the pbkdf?
 
Salts always have to be stored. You can't reproduce the same output again if you don't remember the salt you used
 
7:42 AM
Okay so I assume, I will create a hash, then use PBKDF and get a salt and append this salt to the created hash and then save that output to the database, so the next time I am decrypting password, I will get the salt from the stored hash, and pass it onto the pbkdf right?
 
From a security point of view that sounds fine, but I think you're still a bit confused because you say you want to store the salt with the hash. I don't understand why you would store those 2 together, because they don't have anything to do with each other. The salt is needed for PBKDF, and the hash isn't. Conversely, the hash is needed to verify the master password, but the salt isn't. So it makes no sense to append the salt to the hash.
Creating a database:
1) Compute `hash(MASTER_PW)` and store that as `MASTER_HASH`.
2) Generate a salt and store that as `MASTER_SALT`.

Opening a database:
1) Compute `hash(MASTER_PW)` and verify it's identical to `MASTER_HASH`.

Saving/loading a password to/from the database:
1) Compute `pbkdf(MASTER_PW, MASTER_SALT)`. This gives you your `MASTER_KEY`.
2) Use `MASTER_KEY` as the key for the encryption algorithm of your choice.
 
Ah but I need to still store it somewhere right? Maybe in a separate column?
 
Yeah, that sounds good
 
8:21 AM
Is there a generic dupe for higher order functions like map, Executor.submit or Thread needing to be used as func(target, *args) instead of func(target(args))?
 
8:31 AM
import subprocess

test=subprocess.check_output( "ls non_existent_file; exit 0",stderr=subprocess.STDOUT,shell=True)
print(test)

output --> b"ls: cannot access 'non_existent_file': No such file or directory\n"

I don't want to use "exit 0" , so if file not found it should not terminate ,instead it should give above output without exit 0
 
@MisterMiyagi Don't think so. And to be honest, specific answers are probably more useful than a generic one. For example, if you're creating a thread, you can pass args=whatever, but you can't do that with map.
 
Anyone have solution regarding subprocess !!
 
I don't understand what you mean by "it should not terminate". You mean it shouldn't raise an exception?
 
Yes it shouldn't raise an exception , instead it should give me the
output --> b"ls: cannot access 'non_existent_file': No such file or directory\n"
 
You can just catch the exception and get the output from its .output attribute
Or use subprocess.run instead of check_output
 
8:45 AM
test=subprocess.check_output( "ls non_existent_file; exit 0",stderr=subprocess.STDOUT,shell=True)

without "exit 0" its should throw output:
b"ls: cannot access 'non_existent_file': No such file or directory\n"
@Aran-Fey I have the intension to use check_output only
 
 
2 hours later…
10:23 AM
I can't believe I was ever excited for dataclasses. I keep trying to use them as a convenient tool to reduce boiler plate, and then it painfully reminds me that that's not what it is.
 
Wow... quite an interesting answer pile on, on: stackoverflow.com/questions/70109321/… :p
 
@Aran-Fey why, what is it?
 
AranCoder9 strikes again!
 
@Aran-Fey this makes me happy, that I did not meddle with them yet
 
Hard to describe... essentially, the dataclass assumes it knows everything about itself - the fields you've annotated determine how the instances are hashed and compared, and what the constructor arguments are. Almost always, one of those things is in conflict with what I'm trying to do. For example, if I annotate a child class with @dataclass, then it stops calling the parent class __init__. So I have to add @dataclass to my parent class as well, even though it's largely pointless.
It's like async, it infects the rest of your code
 
10:33 AM
infectious code is the worst, it's the same with frameworks which are around you instead of libraries
 
Django is a framework. Flask is (more like) a library.
 
It works well if you use it to define one class and that class's purpose is mostly just to hold data. If you add more classes, or behavior, or anything extra, you run into pain.
 
@Aran-Fey clearly the issue is inheritance :P
 
Yeah and I kinda prefer flask. But yeah if you want to do ironman stuff you might need a suit. But so often what tries to be a suit, could just be a tool which I can grab and use and leave again. Getting into and out of a suit is cumbersome
 
I think that's why I've always preferred flask (until I met FastAPI).
@Aran-Fey I cam across this article, which may or may not help. The explanations are kind of Noddy, but it explains the subclassing implementation quite well. This, of course, is not guaranteed to be relevant.
 
10:42 AM
I've settled for NamedTuple and hand-written classes. @dataclass seems to sit neatly at a point where it combines the annoyance of both simple and complex cases.
 
@holdenweb looks neat, so it's flask with automatic docs and faster?
 
@holdenweb I do understand how it works, but honestly, a class whose initializer doesn't call the parent class's initializer is just... broken. You can't not call the parent __init__, that's dumb. That's pretty much the very basics of OOP.
 
Let's just say it has a modern and (IMHO) well-considered feature set, apparently implemented very competently.
@Aran-Fey Fair enough. You understand how it works and that doesn't satisfy your requirements. It does have many satisfied users, though, and as you say it's (maybe too) easy to think of it as a convenient tool to reduce boilerplate. Which is anyway true if you keep things simple.
"This toaster doesn't crush garlic very well"?
 
It's a toaster disguised as a garlic crusher, though :P
Or maybe it's more like, in this universe every toaster should also be able to crush garlic
 
I'm on Arans side, if it has class in the name it should call parents init method. Otherwhise call it datacontainer or something else
 
10:55 AM
There's talk about inheritance on the dataclasses reference page, so it explicitly supports inheritance. But it seems to be geared toward dataclass-only inheritance trees.
 
It's actually even stricter than that. Only dataclasses, and no custom __init__.
 
Since we're talking dataclasses, i've often heard that attrs does everything that dataclass cant. Have you formed any opinions about that yet Aran?
 
@dataclasses.dataclass()
class Parent:
    def __init__(self):
        print('Parent.__init__ called')

@dataclasses.dataclass()
class Child(Parent):
    pass

Parent()  # prints
Child()   # doesn't print
 
@Aran-Fey "keeping it simple" probably includes not defining __init__ for your dataclasses
 
Yo man. I identified the issue, I discussed yesterday. The memory issue. I hope you remember that.
Turned out I was using multiprocessing library and when I removed all those stuff, it is now working fine. So the issue was with it. But one thing is still strange that windows was handling that behavior but Debian environment couldn't
 
10:58 AM
@ParitoshSingh Nope, never used attrs. Maybe I should start
 
@HKay glad to hear you found a culprit/workaround
 
Yeah, Thanks man. I thought I should come here to tell you. You deserve to know it. Take care
 
@Aran-Fey FWIW i think this is ultimately the biggest selling point of the feature. Dataclasses were designed to address this specific use case first and foremost, and i think many folks are happy with this capability.
 
@Aran-Fey When dataclasses were introduced the dev tea were very clear tht they didn't want them to complete with attr.
 
But... they do the same thing though? How can they not compete?
 
11:03 AM
Perhaps in the same way that urllib doesn't compete with requests ;)
 
iirc attr does "more" essentially.
lol AD
 
They are there for people who are happy with the built-in feature set.
It's a "minimalist" offering, and is helpful (e.g.) to some sites where using anything not built in is a pain.
 
Ah, you're saying they never planned to add many features / support complex cases. In a sense, that's reasonable enough, but it annoys me that they identified a common problem in the language, and then decided to solve it poorly
 
FWIW dataclasses do allow the use of a post_init method that can be used to call the superclass's _init__, so you could subclass from a class that did that ...
But I wouldn't call that an elegant solution.
 
@Aran-Fey I also started thinking of this example... don't dataclasses generate inits?
 
11:11 AM
It doesn't if the class already has one
A non-inherited one
 
this essentially seems like it was bound to break the parent's init then, because there's a promise of "init generation" essentially overwriting whatever it would have inherited.
i could see how it would be nice if there was an option to explicitly let the parent's init also be fired, assuming its a chain of variable creation
 
it does take into account parent dataclass-generated fields
according to the docs, that is
that's probably not done through the respective __init__s, especially since children overwrite their parents' definitions
 
Yeah, the child __init__ takes on the responsibility of initializing all parent classes' attributes as well. It does all the work itself without chaining up to the parent __init__
 
 
1 hour later…
12:34 PM
Wow thanks for introducing me to attrs. This is what I wish Pythons default classes would look like
Are there any downsides to it?
 
amusingly enough, attrs was the inspiration for dataclass
external dependency, if you consider that as a downside, and larger learning curve i suppose. that should be about it really
you can genuinely think of dataclasses as a stripped down simplest use case of attrs
 
The most important features I would want took me 5min to learn. So that's already a huge win, what a great day :)
 
For what I end up working with I've always quite liked marshmallow.readthedocs.io/en/stable
 
I vaguely remember a point that attrs would explicitly stay outside stdlib so that it can evolve faster. Much like requests and friends.
 
12:41 PM
and pydantic is also nice...
 
i've heard about pydantic but dont know much more about it. Marshmallow this would be my first time hearing about it o.o
 
12:53 PM
@JonClements They seriously chose a code sample that does literally nothing as the very first thing to show on their docs? Really?
 
doesn't seem too awful to me...
 
Actually, it does convert the date to a string. Not sure why, but ok.
 
serialisation - so something else can load it back...
 
Is there an option in mypy or pylint to warn me when a variable has been re-assigned? So for example if I assign a value to foo I can't later in the same scope re-use the variable name and assign a different value to it.
 
the elusive const keyword...
 
1:05 PM
Would that work on things like say a dataframe?
 
@crypticツ typing.Final should do what you want.
 
The reason I am asking is I found a bug in some code where the variable was being overwritten in a loop, while the loop also needed the original value. I want to try to avoid such errors if possible.
@MisterMiyagi thank you. =o)
 
1:20 PM
@MisterMiyagi does that also work outside an inheritance picture? If so, the docs should be a lot less specific.
OK, the PEP explains that it should work in any context
 
is there a canonical answer for why not to just do try: except: pass? I have a memory of having found a good answer for this question, but I can't find it anymore
 
that doesn't sound like a good Q&A target
 
Ok, I guess I will keep looking. I'm sure I found a good explanation of why not to do that and I don't want to write one myself. Also it seems like my coworker ignores all and any warnings in Pycharm...
 
1:39 PM
well it's poke
 
talking of poke - it's been a while since I've seen 'em around...
 
he poked his nose in here not so long ago
@MisterMiyagi "that's just, like, your opinion, man"
for some values of "not so long ago"
 
@AndrasDeak triggered :D
 
2:06 PM
@AndrasDeak Famous last words before cabbage hits the fan. :P
 
2:31 PM
Hi I've got a pandas question where I'd like to add another condition to the min max condition that I've already got in this line of code:
df['Bonus'] = df['Delivered'].mul(0.14).where(df['Average Parcels'].between(10, 18, "left"),0)

So right now the min is 10 and max is 18. If Average parcels happens to be above 18 it ignores it and returns 0. Is there a way to replace values above 18 with 18 instead of 0 in the same line of code?

Thanks
 
@cchev have you looked at .clip ?
 
I have not, I'll look now. Thanks Jon :)
@JonClements That worked, thanks!
 
3:07 PM
@Aran-Fey Not at the moment, but I've been intending to write up a machine-readable list of recipes for my own purposes. If I make any progress on that, I'll share it with you
And if you want to cut out the middleman, you can examine the game's source. github.com/TBartl/space-station-13-idle/tree/master/src/data has some promising-looking file names. I haven't investigated that option very deeply myself, for fear of spoilers
 
I don't play idle games so let me know if you need a pair of eyes
 
I speculate that the data files won't have separate recipe entries for buff-boosted recipes. For example, all Engineering-class recipes that produce power allow an optional ingredient Energy Drink, and gives you power_amount*4 coins. So the basic recipe "1 Ghost Tear + 5 seconds -> 64 power" can be boosted to "1 Ghost Tear + 1 Energy Drink + 5 seconds -> 64 power + 128 coin"
 
Unfortunately I'm in design hell and my project currently consists of various half-finished bits and pieces. I did have an algorithm I wanted to test, but... not anymore :|
 
It would be rather tedious to manually write up ~12 new recipes for the ~12 existing Engineering recipes, so perhaps the game constructs them dynamically
Hehe. My work on the topic is also half-finished bits and pieces :-)
 
Hello :D, still talking about recipees xD
 
3:22 PM
Shouldn't an energy drink boost the amount of energy you generate, rather than coins? :s
 
Recipes and crafting is the most interesting game mechanic among game mechanics, ever since it was invented by Minecraft in 2010
@Aran-Fey There's another buff called "Social Lubricant" which rewards power every time you sell something in Cargonia. I have never used it, because I don't know how to get into Cargonia.
 
nice, I wish I would be at your level, maybe I learn one or the other thing from justreading your deep deep thoughts :D
 
I'm uncertain of whether my level is particularly enviable, but anybody can reach it if they put in the time and thought
Five minutes ago I wrote that 64*4 == 128, so don't put me on a pedestal that's too tall
 
3:42 PM
@Kevin takes away a lot of glamour of many things. Once I realized I could do X, actually doing X lost it's appeal. Knowing I could do it is enough, which I find a bit odd, but well
 
I know a recipe for disaster if that helps? (sometimes it's letting me near a keyboard :p)
 
4:37 PM
Hi is it okay to share the link to the question that I had posted.?
 
@Tushar have you read through sopython.com/chatroom ?
 
I am sorry I have not....link seems to be blocked as I am on my work laptop...will definitely read through once I connect from the personal one...Thanks.! :)
 
as in... are you referring to this one? - if so, or even not so, it'd be nice if you tidied up the examples :)
(put them in code blocks as well as your code and make sure your code example isn't actually indented unnecessarily...)...
the general rule here is we ask you to wait 48 hours before asking here as the main site is very active and you'll get attention there
 
@JonClements yeah that one.! unfortunately I am having trouble editing with the tables always..
 
looks like MM has been kind enough to do it for you anyway :p
 
4:43 PM
@JonClements yeah.! Thank you @MisterMiyagi :)
 
 
1 hour later…
6:11 PM
cbg!
so, I had a question although, the answer is probably obvious, it kind of didn't jump to me yet: there this module called keyboard, and when trying out one of it's example code, I noticed that it use a list but has space delimiting two value inside of it, without comma?? example here:github.com/boppreh/keyboard/blob/master/examples/…
the list in question is events
when i tried to print it or used repr on the events list/variable, it showed something like this:
[KeyboardEvent(q down), KeyboardEvent(q up)]
but when trying to reassign this to a list, it doesn't work because of the space delimited values, such as q down in this specific case
I'm just wondering how this module make it work in this specific case, since this doesn't make much sense to me
 
6:48 PM
Ok, I see your question now. The __repr__ is free to do string formatting how it wants. KeyboardEvent(q down) is indeed invalid code, it's just the representation that they have chosen for the items in the list
I don't know why they didn't include the comma because some PEP (I forget which) explains that repr should give you something that can reconstruct the object. I don't know how seriously that's actually taken because I don't think I've seen it in the wild
 
@roganjosh it often doesn't work that way, although ideally it should
 
I'll be honest, I'm pretty sure I've broken this myself many times
 
@NordineLotfi you can check events[0] for that specific KeyboardEvent.
 
 
1 hour later…
8:09 PM
@roganjosh that explain why I didn't understand this :o Thanks for the detailed explanation :)
@AndrasDeak that doesn't work, especially if you take into account what roganjosh mentioned earlier :)
 
8:32 PM
Hi, I'm trying to multiply the values of a 2D array, given a 2D mask, I think there's a way to do it elegantly with Numpy but I can't find the documentation
Let's say I have a 5*5 array, and a 2*2 mask, how could I compute all the combinations of multiplications of values with this array and mask
(so I'll get a 4*4 array, kinda like a convolution)
The difference is that a convolution adds the values given the weight of the values in the mask. I'm trying to multiply the values inside the array if they match the mask
 
@BeChillerToo the quickest-to-implement answer is numpy.org/devdocs/reference/generated/…
4x4 arrays of 2x2 windows -> compare with mask using broadcasting -> all() along window dimensions
It will probably be slow.
 
It's not a big array
Thanks for the link!
It's for project euler #11, I spotted the solution using my good old eyes, but I'd like to implement a neat solution
 
No worries. Let me know if anything's unclear. I'm on mobile so few words must do trick.
 
Your words did fine, thanks a lot for your effort!
 
8:58 PM
If there's ever something to make you realise that you're in a losing battle with devops, it has to be this?
 
Hmmm, that looks goooood
 
I unearthed that with some moldy potatoes. Good grief
 
9:13 PM
it's parametrised so it's all good
352 placeholders. And you insert the month at index 224. Wow.
Does the original come with any comments/documentation of what all that crap is?
I also like how str(currPath) suggests an actual Path object, but then they use string concatenation to build the path
 
Yam Laurel
 
@AndrasDeak Of course it comes with comments (though it's clearly self-documenting) "# insert into intermediate table from csv"
 
ah yes
 
I mean, I could try extol the virtues of .join() and tell them about executemany or other methods that might not be cripplingly slow. But what do I know? I'm only DS
 
tell them they should try committing twice per row
 
9:21 PM
Extra Sure
@AndrasDeak You'll love the other part in the same module here
<establishes a crying corner in the room>
 
nice
at least there's a chance that dt can only ever be a datetime
Is that supposed to be a fetchone like the comment suggests?
 
Sorry for the delay, I was trying to get my head around it. It checks that the data value of sqlData = {'data': cursor.fetchall(), 'columns': cursor.description} isn't empty, truncates the table, then tries inserts
columns is never used, but it couldn't be anyway because you'd need [i[0] for i in c.description] to actually do anything useful with it
 
I don't know sql but semantically speaking "is it empty?" sounds like "check one".
 
Yeah, you could LIMIT 1 on the query
That'd be the short-circuit if you wanted to break early
 
So what's fetchone() for?
 
9:36 PM
That'd be .next() on a generator
 
Which is also lazy, right?
 
.fetchall() will expand the whole thing into a list of tuples
 
OK?
 
Can we rewind a bit. I think I've missed the point of the question sorry
 
sure
 
9:42 PM
I've gone back through both links and I don't think I see where you get the idea "is it empty?" from a code comment
Ah!
 
# check file date is valid
and you then said "It checks that the data value ... isn't empty"
perhaps the point is not to check if it's empty, but rather checking emptiness is a necessity before doing further things with the data
 
Yes I did. Sorry. I don't know whether fetchone() will automatically limit the query, I'd use "LIMIT 1" in the query myself
 
Hmm, where do the results of a query live? On the cursor? Does it have some internal object that holds a query's result, which it can feed in response to .fetch* methods?
 
Good question, and one that I realise I forgot to answer to myself
 
If I understand correctly you'd be worried that despite the fetchone the entire queryset (or whatever) would be pulled from the db, just not instantly returned
a bit like having a list iterator rather than a proper generator expression
 
9:47 PM
If it keeps firing queries then that would be... tricky to keep track of. I think it has to do the full query and fetchone() is pretty superficial in the grand scheme of things because it's all there in memory, just not materialised in your program
 
I find that reasonable. Something something caching.
OK, thanks
 
Decent question, though. I forgot that I wanted to dig into this :P That's why I just put LIMIT on my query so I know that SQL can just short-circuit
 
 
1 hour later…
10:55 PM
@roganjosh it doesn't short circuit though - it has to do the whole thing before it limits it to 10
(the RDMS can just as you're only asking for 10 just disregard the rest so it doesn't have to maintain a cursor or some locking stuff but ultimately - it had to do the whole work anyway to do it...)
(select * from some_table limit 10 - it can probably work out, but if you do something like select * from some_table order by something - it stands no chance)
(unless the something happens to be a reasonably cardinable index or something and it might do so)
Loving this courier tracker interface - been waiting up just so I don't miss the delivery and it's now: " delivery in 7 to 47 minutes."
 
11:35 PM
@JonClements Sure. I'm talking about no aggregation or sorting, just to see if data exists. I find it hard to believe that it scans billions of rows to work out if 1 exists with select * from some_table limit 1. Does it?
fetchall() on the code I posted will have to do all the work, not disputing that. It's just awful. But I have a big misunderstanding if you're telling me the cursor is scanning the entire database to give me one, unordered, result back
 
if you're literally talking about select * from some_table limit 1 - then it'd be a fairly bad RDMS that screws that up
 
<wipes brow>
Yes, that is what I was suggesting as the right way. I was talking about what I think that script that I pulled the snippets from was trying to do
I obviously can't share the whole module but you wouldn't want to see that either, unless you're a rubber-necker for car crashes :P
 
on a side note: select count(*) from some_table and checking it against 0 might not always be the most efficient
 
That will pull from table metadata?
Wait, I didn't suggest COUNT()
 
I know... just saying it out there for the newbies to RDMS's :P
 
11:41 PM
Ah, ok
 
rule of paw is - don't expect the DB to be in the state you believe it to be in
 
I was gonna say... the postgres admin dashboard doesn't even try to keep track properly, it only gives an estimate IIRC
 
@roganjosh postgres is very ACID conforming
so you can query its metatables regarding stuff... and its best guesses as to what it'll do... but until you do a query, well...
for all it knows, your query might well be in the middle of an insert/delete query transaction
when curious - always put an EXPLAIN in front of your query and get the system to show you what its query plan is
(and if you see a lot of SCAN TABLE - something's wrong)
 
Does this hold for Redshift do you know?
 
11:57 PM
haven't used redshift in 4/5 years? So I'd have to pass I'm afraid
 
We stack a lot of CTEs (that probably do need to scan the table)
 
I'm sure putting an EXPLAIN in front of your "heaviest" queries would give you an idea
it's rare that a DB should have to table scan
 

« first day (4058 days earlier)      last day (907 days later) »