« first day (4723 days earlier)      last day (445 days later) » 

02:47
Hey. I was.curious if stackexchange is a good site to post my program I made to get feedback. It works as I intend it to but I would just like feedback on my actual construction. I think I'm getting better at programming. In this particular program I made it is the first time i used functions. Any other feedback would be welcomed. I do not have a formal education in programming or computer science.
 
1 hour later…
03:53
@randomgirl CodeReview.SE is more on-topic if it's working code rather than specific code snippets that have issues.
 
3 hours later…
06:27
@Arne oki doki. I've added my other code just for posterity but I misunderstood the requirement; I still think it's useful to keep around in the case where the CSV has to be read multiple times. Gonna give DuckDB a go now and add that to your timing script.
07:06
Anyone using AWS Timestream? Did you experience an outage this morning?
07:20
Ok, I'm officially completely baffled by DuckDB. I don't understand the docs at all; this shows you reading directly from an existing db, but this shows creating a table. And what magic is going on where my_df knows where the data is in the second case?
I guess the query engine must look up locals() but it looks a bit arcane
My birds eye picture of databases is that they are 100% magic and the only difference is the relative amount of dark, voodoo and eldritch magic in that mix.
Doubly true in this case. When you create a table, that exists in the ether for the duration of the program, which means that on big projects you're not familiar with, there could be databases lurking in memory that you know nothing about... don't look under the bed. I guess you can do in memory sqlite databases too but you still need a conn/cursor object
07:39
"Mommy, there's a scary database in my closet!" -- Terminator, programmer edition
Olaf's sinister brother Olap
@MisterMiyagi I wanted to thank you for your help with my Futures question. It's interesting to reflect on the concurrent.futures design decision that a future represents the result, not the computation. I am glad to learn how to dive into the future's subprocess directly, and about alternatives like pebble, which have the futures semantic I was hoping for.
@MaxSuica Thanks to you too for asking an interesting question. Even if the answer was out there already, I didn't know it then and your question got me thinking and looking for it.
@MisterMiyagi it is exactly your ability to find that answer that brings me here! First I want to say how impressed I am. Before asking my question I did search SO, but could not find the answer. Yet you were able to find the answer!

And I want to ask if you could can share some of the terms or strategies you used in your search - if I can become more effective at finding existing answers, I believe that will improve the quality of the questions I /do/ end up asking.

thanks again for your guidance and patience!
07:54
By "search SO" do you mean using the search bar inside SO itself?
Indeed as roganjosh hints at, my main strategy is just not to use SO search. General purpose search engines usually support searching specific sites only.
Using SO search is a critical error. It is, and will forever be, utter garbage. It is known.
For example, for your situation searching site:stackoverflow.com python future cancel on Google had the duplicate Q&A in my top five results.
@roganjosh yes - I searched terms like "concurrent.futures" "result(timeout" that should show up in questions similar to mine. While I found a few results pertaining to concurrent futures, they were not examples of my problem. And the results had lots of "noise" - results that didn't have my terms at all.


Ps. My question was here:
https://stackoverflow.com/questions/77147531/concurrent-futures-future-cancel-ineffectively-terminates-running-threads-an

The answer to my question was here:
https://stackoverflow.com/questions/42782953/python-concurrent-futures-how-to-make-it-cancelable
@MisterMiyagi noted! will never ever again use SO search, and will always defer to site:stackoverflow.com on google or ddg.

I just tested `site:stackoverflow.com python future cancel` in ddg. The 2nd result was the answer you suggested!
FWIW I can't remember the last time I used site:stackoverflow.com in my search. Typically SO results will rise to the top anyway but sometimes I snag a decent tutorial or blog about my issue so I don't start by excluding them. I guess that's just personal preference though
08:04
@roganjosh I agree. Usually works for me too. But in this case, not using site:stackoverflow was causing a bunch of spurious results
As much as I am dissatisfied by SO recently, it's still got much better average quality than many blogs and such (for my topics of interest). Searching SO first isn't a silver bullet but usually keeps my blood pressure lower.
I'm particularly happy that ddg's site:stackoverflow.com works here, since it lets me continue my google boycott even while doing SO research. (By my estimate, each search taken away from google removes something like ~$0.11 of ad and placement revenue from the pockets of the greedy search giant!)

Thanks once again for the help, @MisterMiyagi, @roganjosh.
Oh, for sure. I imagine your searches are usually much more targeted than mine for the answers you need. I've trained my eyes to block out things like GeeksforGeeks or W3Schools etc. Sometimes I might skim a Medium article for 10 secs
I just don't do what the radical kids these days blog on what they do these days while riding their hoverboards these days.
@MaxSuica no worries. There's always this room too if you get stuck and your search doesn't yield anything :)
08:17
@roganjosh oh, it absolutely is useful. I had no good example of polars that I had really understood yet, so having one at hand is great
08:36
It doesn't even just use locals(), it must use globals()
import duckdb as ddb
import pandas as pd

df = pd.DataFrame({'a': [1, 2, 3]})

def create_table():
    ddb.sql("CREATE TABLE tbl AS SELECT * FROM df;")

create_table()
print(ddb.sql("select * from tbl"))
Only locals and globals? Pathetic!
__builtins__.df = pd.DataFrame({'a': [1, 2, 3]})
ddb.sql("CREATE TABLE tbl AS SELECT * FROM df;")
# Throws Catalog Error: Table with name df does not exist!
Write a bug report right now!
I think my DuckDB dabbling days are done :P
09:02
it does check nonlocals though, preference being local -> global -> nonlocal
i didn't know yet you can access nonlocals in a scope where they weren't declared as such.
It's the same as with global, you can read but not write
it's the same as with globals: the variable scope is determined at compile time, and the lookup occurs at runtime
and that scope resolution rule works the same way, allowing read-only access to the outer scope.
@Arne Are you sure? nonlocals should have priority over globals, and it's easier to implement it that way, too.
it's in locals if it's accessed there
I would expect nonlocals to be extremely hard to fetch unless there is an actual nonlocal access. Python won't create a Cell for nonlocals unless you at least lexically statically read from the variable.
09:08
def foo():
    x = 1
    def bar():
        print([*globals()])  # ['__name__', '__doc__', ...] no x
        print([*locals()])  # []
    bar()
foo()
CPython doesn't even create a closure there, since bar never accesses x. If duckdb can access that x, they must be metaprogramming wizards
right, "i didn't know yet you can access nonlocals in a scope where they weren't declared as such or accessed."
@Aran-Fey definitely one of the darker shades of black magic. and all because they thought that ddb.sql("SELECT foo FROM df") was a lot better than ddb.sql("SELECT foo FROM df", {"df": df})
Wow, duckdb can actually do that. My mind is blown
Ah, they're doing it wrong. I too made this mistake...
def foo():
    df = pd.DataFrame({'a': [1, 2, 3]})
    def bar():
        ddb.sql("CREATE TABLE tbl AS SELECT * FROM df;")
        print(ddb.sql("select * from tbl"))
    return bar

bar = foo()
df = pd.DataFrame({'b': [9, 8, 7]})
bar()  # Prints the global df
@Aran-Fey I thought foo still makes the stack frame you can access from raising a traceback
That's exactly the mistake! Stack frames do not equal scopes
09:18
Is the frame's dict different to scope?
@MisterMiyagi, I've written up a pebble based solution. It remarks on why we can't cancel native futures if they're running, and shows us how to create and cancel pebble's ProcessFutures.

The interface is very simple! If you're interested, check it out here: https://colab.research.google.com/drive/14XDhhFHF8wA4va6NTjpm_WgONx_DUGtk?usp=sharing
@Aran-Fey ok that's an insidious setup though
True. It's honestly a reasonable design decision if they did it on purpose
@Peilonrayz Yesn't. Depends on what exactly you mean by "scope". If we're talking about runtime, then a "scope" is essentially a dict like what you get from globals() and locals(). But when talking about nonlocal and closures, what matters is the syntactical scope, i.e. where the function is defined
Getting the dict is probably easy. The ChainedDict not so much, yeah
FWIW, it would be quite a waste if inner functions forced all variables of containing functions to stick around.
09:24
how do you even get the syntactical scope during runtime. re-parse the module where the call happens?
Going up the call stack in order to resolve a variable (i.e. what duckdb is doing) would only be correct if python worked like this
def foo():
    print(x)  # Prints 'hello' ?!

def bar():
    x = 'hello'
    foo()

bar()
By the time bar runs, both foo() and it's df variable are long gone – and that's a good thing.
@Arne Can't think of another way. You can do some dark magic via the import machinery so that your module can influence how importing modules are parsed, but it's a hell of a ride.
macropy works that way, IIRC.
@Aran-Fey The code there would work with DuckDB?! I'm not a fan
It does
def foo():
    ddb.sql("CREATE TABLE tbl AS SELECT * FROM df;")
    print(ddb.sql("select * from tbl"))

def bar():
    df = pd.DataFrame({'a': [1, 2, 3]})
    foo()

bar()  # prints the df
Oh wow, they are really doing that? D:
09:29
you can just do print(ddb.sql("select * from df")) to test
Oh, thanks
Well that's even worse
It's stuff like this that makes me think maybe macros aren't such a bad idea after all.
Because now I don't know whether it copied the df or it's using the existing memory
@MisterMiyagi It's an easy mistake to make... I'm speaking from experience
09:31
oh, maybe they just go up the callstack until they find a variable with the right name?
Looks like it. :/
checking my debugger if that replicates the behavior
09:45
df = pd.DataFrame({'a': [1, 2, 3]})

def create_table():
    ddb.sql(f"CREATE TABLE df (b INT)")
    ddb.sql("INSERT INTO df VALUES (1)")

create_table()
print(ddb.sql("select * from df"))
I hate this library
Do they also offer a non-magic API? I'm usually happy if it's at least possible to write sane code...
I don't understand that snippet. doesn't it just ignore the df variable because you declare your own table of the same name before the df is consulted?
Yeah, that exactly true
that was kind of what i expected when I read the code ^^
Now in my mega project, are you going to know whether I defined a table called df or it's looking in literally every scope and running over something called "df"?
09:53
ahh
If you hash out create_table() it'll give you a completely different result. Still runs just fine, though
yeah, would be cool if ddb.sql accepted kwargs which, when used, disable any scope lookups
But then you would have to know this stuff was happening anyway. Look at the debate we've been having here so far between people who actually know python internals (I'll exclude myself there). Why should it be this complicated? What has it achieved other than complete ambiguity?
10:06
Unexpected design consequence: choosing def as the keyword for functions means that tons of beginners randomly conclude that "defined function" is meaningful terminology (it comes across that they think there are other kinds of functions, but Python code uses "defined functions" instead... ?)
(or perhaps that their code could contain other kinds of functions that they haven't learned about yet.... ???)
well, lambdas kinda are different
@roganjosh given how often python-beginners expect variables to be aware of their name, ddb's behavior might be intuitive for them.
if the target group uses mostly notebooks, things like runtime scope <-> lexical scope discrepancy should also be more or less nonexistent
You already gave the answer yourself, though. "and all because they thought that ddb.sql("SELECT foo FROM df") was a lot better than ddb.sql("SELECT foo FROM df", {"df": df})". That's how every other SQL process works, I don't see how this setup is useful for anyone
Tbh, I personally feel like in-process SQL DB is flawed concept in and of itself
@KarlKnechtel I treat (and teach) def as a verb, not an adjective or noun. def foo isn't a defined function foo, it means you define function foo. But my target audience may be more open to that picture...
10:13
@matszwecja you might need to substantiate that claim a bit more :P
@matszwecja it's more an SQL interface to python objects
in that sense it can be quite useful
@Aran-Fey Ah I see what you mean now. The value isn't bound to <function bar>.__closure__ if the value isn't used in bar. And if the value is bound to <function bar>.__closure__ the values will be available in f_locals.
@roganjosh nah, that's nothing more than my gut feeling.
@MisterMiyagi I assume you have the luxury of an audience that speaks English as a first language... ?
@matszwecja they have the same type.
I know, but conceptually I think of them differently
10:21
@KarlKnechtel who is your audience?
@Arne it seems their docs are wrong too because it states "local variables"
tsk, and a grammatical error as well.
I'll still include it in the benchmark though, my users might enjoy the fact that they can write plain sql a lot.
Lol, I never got as far as actually writing it out for the benchmark sorry. I started having problems with parameterisation and then got lost in this black hole :P
10:41
@roganjosh The Internet (TM)
10:55
@Arne My grammar skills must suck. What's the error?
FWIW I didn't spot it either
Ehh, "as if they are were regular tables" I guess?
are (---) I hate how SE has 4 billion bespoke Markdown flavours
2
This lawsuit seems weird. It reminds me of "if your friend told you to jump off a cliff, would you?" as a kid. It's also quite worrying since I build routing algorithms that use mapping services and then it's me that conveys that instruction on in the routes I generate. We can forget the trolley problem with self-driving cars if map instructions are legally liable
11:13
@roganjosh The question I'd be asking is why the bridge was not closed off, and so accessible to drive off.
It says later that vandals had removed the barriers
So the article does
But the fact that it could, in any way, be entertained that Google's tech caused the issue is the bit that worries me. It's just a service with no guarantees over your safety; I would expect the case against Google to be thrown out immediately... but it hasn't been, seemingly
@roganjosh google map have and always had these problems, where a specific street was never properly made in the map, and for example, where there is an obstacle/wall that have been there for more than 50 years, is not there, etc
I recall knowing of some jobs where people worked for third party companies that corrected or added additional info on google map and the like, but I forgot the exact name of that job
I'm not questioning the veracity of map data; I use OpenStreetMap all the time and I know it has flaws. I'm questioning the culpability in someone dying
11:22
I have an if condition like this:
if (
    not object.__getattribute__(self, "_state_properties_initialized_") and
    any(prop.name == attr_name for prop in type(self)._state_properties_)
):
And black insists on formatting it like this
if not object.__getattribute__(
    self, "_state_properties_initialized_"
) and any(prop.name == attr_name for prop in type(self)._state_properties_):
Is there any way to make it stop doing that?
You can sue everyone for anything in US, regardless of legal basis
It would be worse, for example, if I take data that I know to be liable to have issues and then tell my employees they have to follow my route. Now I have compelled them to follow that route, which is presumably even worse
The article does say that Google was messaged for nearly 10 years about that missing bridge
right, I guess it's hard to know who is really at fault. Is it the people curating it? the company that made it? I feel like people shouldn't mindlessly trust it in the first place but use it as additional tool. I have seen enough delivery getting lost because someone trusted online maps
@matszwecja But they have no obligation for your safety?
11:25
they don't, but they have a responsibility to at least offer easier/more accessible way to correct things
There's something called "gross negligence" (I'm not saying that this situation counts, but in general)
Gross negligence would surely only apply in the case where you assume responsibility over someone
@roganjosh not really - "thoughtless disregard of the consequences and the failure to use even slight care to avoid harming the life or property of another"
@matszwecja If I assume the first line of the Wiki to be an accurate quote: "Gross negligence is the "lack of slight diligence or care" or "a conscious, voluntary act or omission in reckless disregard of a legal duty and of the consequences to another party." highlight mine
@Aran-Fey mmm "Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting."
@roganjosh There is something known as general duty of care.
If you create a product that is likely to cause serious harm and you fail to provide sufficient warnings, you can be responsible for that harm even if you didn't sign any contract.
11:40
I don't understand why people use this thing willingly
@matszwecja Now I am getting more concerned about this. It literally does boil down to "if you friend tells you to jump off a cliff...". I don't know how I could mitigate it; my situation is "I know this data could have faults, but do what I say anyway"
@matszwecja yeah, expressing something in subjunctive that is not the case uses past tense
@roganjosh It's not hard to figure out what happens if you jump off a cliff. But people usually don't expect to fall off a cliff while following a road.
Sorry, it was a poor example. I meant to say, it does go right back to the fundamentals here. If I made a commercial chatbot saying "do horrible thing X" all the time, I guess that would also be culpability if someone actually did X
Yyyy, yeah? Inciting people to do a crime and or harm theselves is also a crime, afaik
11:47
@KarlKnechtel Native Germans but from the upper end of the education and motivation spectrum.
@matszwecja the point being that I wouldn't immediately know that my product was doing that. A LLM might start doing such things with an edge-case prompt. I wouldn't know that my routing software is telling a driver to go off a broken bridge. But in both cases, the possibility is known. I know that map data is not 100% accurate but my company would enforce drivers following it
@Aran-Fey # fmt: off
Oooh, thanks. Not so uncompromising after all!
@Aran-Fey because then everyone else in the team does too. Black may be weird at times but at least it didn’t hold a grudge.
@roganjosh That's why there are multiple factors that serve to decide if duty of care was breached. In this case, we've got information that Google was repeatedly informed about the problem and didn't do anything to resolve it.
If you got a message that your LLM does something likely to cause harm and did not take any steps to resolve it, I don't see how you being responsible for that harm is a controversial topic
It all has to be analysed cases by case, as obviously there are a lot of separate parties at fault there but I do see a legal basis for the suit.
11:56
It isn't. In the routing case for me, the first I would know about it, though, is when the vehicle either stops short or they plunge. I'm not privy to all the cases submitted to my mapping service. I just know it's "generally not perfect"
Washing your hands and saying "use at your own risk" sometimes isn't enough.
12:09
I need to set static values in a list at given indices, and then populate the list incrementally with whatever values are left
Any ideas?
create new list, iterate over leftover values, append them and if at index that is for static value append it instead.
thanks, let me try
@roganjosh When I raised a formatting issue on GitHub someone unaffiliated with black responded with your quote. Then the issue turned into infinite bike shedding.
I gained a pretty poor opinion of black from then.
Hahaha. This is why I refuse to use black
Sweet, sweet irony, though :)
12:46
@Aran-Fey If it is just a single location, you can wrap in # fmt: off / # fmt: on comments. Or you can move object.__getattribute__(self, "_state_poperties_initialized_") to the previous line and save to a variable with a shorter name.
How is object.__getattribute__(self, "x") different from getattr(self, "x")?
It bypasses my own __getattribute__
It really should be super().__getattribute__('x')
Maps may contain trap streets, "a fictitious entry in the form of a misrepresented street on a map, often outside the area the map nominally covers, for the purpose of "trapping" potential plagiarists of the map"
^^ I remember seeing this as the basis for a sci-fi short story. Where the protagonist was warped to an unknown location, and the auto-navigator was using a galactic map of pulsars to determine where they were. Except the map had a trap pulsar in their vicinity, so it could never get a proper match. I think the story ends badly (protag runs out of air or similar demise).
Oops.
I suppose it wouldn't be too hard to write navigation software that works around trap pulsars... if the devs are aware that the DB might contain them.
en.wikipedia.org/wiki/Agloe,_New_York "When Esso threatened to sue Rand McNally for the assumed copyright infringement which the "trap" had revealed, the latter pointed out that the place had now become real and therefore no infringement could be established."
13:40
@PM2Ring "might contain them". I know my mapping software has issues. I just have no idea what they are because it's a 3rd Party API
Regardless of that, we will enforce routes on drivers (depending on the company I worked at or consulted for). In the gig economy, their pay will also be tied to the mapping service, regardless of the route they actually drove
It feels a bit like an unexploded nuke :/
 
5 hours later…
18:22
is it just me or does chat look different
Not just you, both you and chat look different :P
Jokes aside though, nope, looks the same as always to me
I got my hair done. Thanks for noticing!
Are you on mobile?
 
3 hours later…
21:58
duckdb timings just dropped, apart from abusing scopes they can also do speed
WHat does this mean when I get an exception when reading a tcp socket and
sys.exc_info equals <class 'NameError'>? I see this error inside the exception handler
I am kinda new to Python
22:29
Um, you should see a stacktrace telling you exactly what kind and where the error happened. No need to look at sys.exc_info
22:48
"I see this error inside the exception handler" - well, this is the problem: exception handlers are there specifically to prevent crashing the program and making things awkward for the user, when you already know about the problem and can recover from it. You should not use them when you are still trying to work out the core program logic (unless they are part of the logic that you have already worked out).
If there's a specific sort of exception that you think the code may raise, you should try to catch specifically that type (e.g. except ValueError:), to avoid catching anything else. NameError almost always indicates an actual bug in the code, rather than a problem with the input to the program.
That said: a NameError is exactly what it sounds like - an Error that has to do with a Name that is used in the program code. I.e., a variable name. Most likely you have simply made a typo when trying to refer to a variable that was set up somewhere else.
Python cannot catch this ahead of time, because it is perfectly valid, legal code. Only SyntaxError happens ahead of time.
What do you folks mean by canonical answer?
23:04
The same as the community generally means: meta.stackoverflow.com/questions/291992

« first day (4723 days earlier)      last day (445 days later) »