« first day (3916 days earlier)      last day (1031 days later) » 

1:15 AM
@toonarmycaptain Good stuff. In related news, I saw former Toon Army captain Warren Barton on FoxSports commenting on the Euro finals the other day...
 
Hi I've used pandas dataframe rename() function. For some columns, names are not changed. Can anyone help me ?
 
1:43 AM
@smci :) Cheers. I didn't know he was captain? I was a young 'un then though.
 
2:02 AM
@toonarmycaptain Only occasionally, when Shearer was injured
 
@smci Oh, I see. Those were....frustrating years. Particularly when my best friend and one of my school teachers were diehard Red Devils.
 
2:13 AM
Hi
Does someone here have some experience using Twisted?
 
2:52 AM
cbg
 
 
3 hours later…
6:17 AM
cbg
 
If I am making lets say 100 threaded requests.get call and I am writing the results to a csv file, is it better to write them in batches? say 10 at a time to a file, my concern is to make sure I dont lose any data in case some request in the 100 raises an exception
 
putting the request.get into a try..except is not an option?
 
6:32 AM
Hello
 
so except:pass?
Hello Li Xiu Ying
 
Hello Python_user
 
@python_user yeah or even write the error urls to separate file to keep track of what went wrong
 
How should i interact or chat with peoples here?
Any rules
 
usually i tend to do a recursive loop to call the request once or twice incase it failed for aa simple timeout or something and if it fails again then save to file to check manually
 
6:35 AM
@Kwsswart ok, thanks for the reassurance, this did cross my mind, but "bare exceptions are bad" also crossed my mind
 
@python_user well, I'd rather except: print(f"request to {url} in thread {thread.id} failed", file=sys.stderr)
or even better and as kwsswart said, write errors to a different file
 
Thank you.
 
ok I will use my logger then
 
7:25 AM
I have an algorithm that solves minesweeper, but it's not perfect - in some scenarios, it can't find a solution even though one exists. My question is, what do I do with the unit tests for these scenarios? Do I mark them with xfail? Do I comment them out?
(It's not worth the effort to "fix" the algorithm, so the test cases will only become relevant once I do - or maybe never)
 
@Aran-Fey I would xfail them. It serves as a sort of documentation what these cases are and that the issue is known.
 
Ok. And what do I do about a test that's intentionally "dumbed down" to match the current capacities of the algorithm? I.e. a test that'll fail once the algorithm is improved (because then it'll find more mines than the test expects it to)
Just... write a comment, I guess?
 
doesnt xfail include a comment?
 
this is the case where the test passes
 
whoops, braino
 
7:34 AM
also, cbg!
 
cbg
 
should tests log stuff? if so, maybe a comment + a log/print
 
@Aran-Fey comment in the assert? Does that work in pytest?
there's probably an idiomatic way to label failures
But just code comment can work too
 
Pretty sure pytest can do that, yeah. I think I prefer a comment above the test though
 
@Aran-Fey If you are using pytest (not sure if it works in unittest too), document it in the test's docstring.
 
7:38 AM
Oho, interesting. What does that do?
Is it any different from a comment?
 
It has to be, from context
 
Ha! Luckily for me, there's an easy workaround: Just throw brute force at it
 
I can't find anything in pytest docs. @MisterMiyagi don't leave us hanging.
 
It's just a convenient location to put information since pytest shows the entire function on failure. Docstrings are pretty visible there.
There are some plugins to use the docstring as the test name and such, if you like that kind of thing.
 
7:52 AM
ah okay, so just intended as a comment alternative, not so much for any special behaviour
 
At least in my native IDE and pytest highlighting, docstrings are much more visible than comments.
 
yeah fair enough. out of curiousity, what IDE are you using?
 
I'm pretty happy with PyCharm (and the other Jetbrains IDEs), though I occasionally use Visual Studio Code as well.
 
8:10 AM
Hello everybody !
Until yesterday I was able to do:
python etl.main
But since today it tells me that it is not possible because it tells me that it can't use the same way:
python: can't open file '/home/ac/Documents/Programming/Work/data-tools/etl.main'
 
I don't know why but many python beginner tutorial uses pycharm and later the new user ends up struggling with it, I think they should explain how python script can be written even in notepad and then use something simple like sublime or VSC
 
Sure, but my file is a regular module ...
 
Oh I was not talking about that, I was actually weighing in on Pycharm mentioned above
 
Haa, I am using PyCharm atm so I thought you guessed it wisely ^^
Sorry sorry
 
lol yea that would be cool :P
 
8:17 AM
@RevolucionforMonica Is that the whole error message? Usually it should say something like "[Errno 2] File not found"
 
Yes, as well!
But I found the issue, I forgot to do python -m XXX ... (to be honest I don't get the difference with python XXX
 
TBH I started on VSC and moved to pycharm when started at this company and truely love the ide more just due to the way you can use it integrated with aws... not sure if vsc has that
 
@RevolucionforMonica It's python FILE_PATH vs python -m MODULE_NAME. With -m, python imports that module and executes it.
 
Thanks !!
I have another question if I may
Have you ever had problems with mysqlclient/MySQLdb library?
 
nope, never used it
 
8:29 AM
Ooh, Aall right :)
 
8:42 AM
thats one way to have no problems with a library. ;)
 
Never using it? Hell yea
@RevolucionforMonica What problems do you mean
 
Haha let me sum them up in 3 or 4 lines
My experience with mysqlclient is an absolute nightmare, @CoolCloud
With *Fedora*:
- With mysqlclient==1.3.7 (which we are using on Debian):
`mysql_config` missing, even when reinstalling connectors such as `mariadb-connector-c-devel` or `community-mysql-devel`
- With 2.0.3:
MySQL-python is missing, or not recognized

With *Windows*:
- With `mysqlclient==1.3.7` ,
I can't install it: tells me to use Visual C++ but nothing changes. So I've tried a pre-built version from Christoph Gohlke's site but it's not compatible.
- With >2.* I have:
`OperationalError: (2006, '')` in a large part of SQL queries like the following one:
python
c.execute("""
        UPDATE `{table_name}`
        SET `{column_name}` = CONCAT('hash_', {expression})
        WHERE {pk_name} IN ({ids})
    """.format(
        table_name=table_name, column_name=column_name, expression=expression, pk_name=pk_name,
        ids=','.join(ids)
    ))
My boss is even thinking about firing me
 
9:01 AM
Shouldn't some of those fields be parameters to execute?
 
At the moment I'm retrying on Fedora with mysqlclient2.0.3 but I have a ModuleNotFoundError: No module named 'MySQLdb'
@AndrasDeak, they are. They are stored in the .format() function which passes the parameters
 
Yeah, no
execute gets a single string
That's the opposite of parametrisation
 
worth noting that some of those probably cant be parameterized by the library itself, such as table name and so on.
 
@RevolucionforMonica If the inputs are not sanitized he might :P
 
Andras, I'm probably too stoopid, but don't I have a single string? @AndrasDeak within """ """ ?
Yeah, @Paritosh I set them in the variables before calling execute
 
9:11 AM
@RevolucionforMonica basically, theres a mismatch between what you "think" is a parameterized query, and what's actually a parameterized query.
 
when you take a string, sure even if it's a string that you'll run python's .format on: there's no "safety checking" that's going on. ultimately, it will be parsed and then a single string will be generated from it, yes?
 
so, this does nothing to prevent sql injection and malicious inputs. so, in sql terms, this isnt a parameterized query
the sql library still ultimately gets a single string to run as-is
 
9:13 AM
if you want to actually use a parameterized query, the sql library itself should be getting a string with placeholders, and the args that it needs to sanitize and then use. there's "usually" some kind of .prepare method in sql libraries for this
that, and only that, is a parameterized query in sql terms.
 
@RevolucionforMonica Obligatory Bobby Tables xkcd.com/327
You have to learn this fast
 
Not gonna lie, I did not understand it in the first 10 reads :(
 
basically. "strings from outside sources bad. hurr durr evil, your db is now mine"
5
 
 
2 hours later…
11:33 AM
Once I reach enough rep on physics SE, which I will do at some point, hopefully. It will rain downvotes
 
"On the spatially localised precipitation of subpositive virtual interactions"
 
@MisterMiyagi lovely :D
I hope trough the merger with this homework website they can move traffic from se which is highschool level stuff to the other site
 
Homework dumping ground? Could use that for SO as well.
 
@MisterMiyagi yeah that's what I mean. Somebody here mentioned that the merger with this other company which also owns homework sites could help so, by routing homework traffic to the other company. Don't remember the names
 
Ah. I thought the merger was purely in terms of company/ownership/legalese, not content/strategy.
 
11:45 AM
@MisterMiyagi It is mostly, but who knows
 
recbg, how can I use this when I am not using a session (directly calling request.get or request.post)? stackoverflow.com/a/15431343 I read in an SO answer session is not thread safe
 
Just open a new session per thread?
 
ohh, is that efficient? I mean opening a new session for 100+ requests?
 
12:00 PM
what makes you think it wouldn't be?
 
...is it efficient to have 100+ threads? O.o
 
I am assuming, creating a manual session is an overhead here, I guess I am wrong
@Aran-Fey I use ThreadPoolExecutor(32), I have a list of 100 urls, should have phrased it well, sorry
 
@python_user Most of the time, it is negligible. Your application likely does some actual work that is much more costly.
If it doesn't, then it won't matter either. :P
 
I asked chat.stackoverflow.com/transcript/6?m=52528370#52528370 related to sessions earlier here but no luck, so I just went with some assumptions
if that works, I would just mount it there
 
@python_user what are you actually trying to do anyway... 100 URLs doesn't seem that many and threading seems a bit iffy to me anyway...
 
12:05 PM
@JonClements I make post requests for the urls and then do some modification to the responses and save these as a csv, the sample data I was given is ~100, it could be more, not more than 5k
 
are you confident that threading is actually an advantage rather than a hinderance here?
 
I mean doing 100 requests without thread is slower right? do correct me if I am wrong
 
Pondering point: How much faster is your task when you don't ponder how to do it with threads but just go ahead and do it sequentially?
 
I can mimic this with setting that ThreadPoolExecutor(1) right? if so I will see the rough time it takes
 
insert planning xkcd
 
12:09 PM
for url in urls:
    ...
 
I wonder how hard it would be to train an AI to take these verbal descriptions and actually insert the xkcd. That would be a perfect project for this room :) It's just whacky and hard enough but actually doable :P
 
It's certainly scary how good google is at picking the right xkcd from just some random words.
 
@JonClements this for loop gave me ~71 seconds for 50 urls, ThreadPoolExecutor(32) gave me ~5 sec
 
71seconds stills seems faster than wondering whether you should use threading. :P
 
12:17 PM
Nobody's wondering about that except for you guys :|
 
laurel, maybe I will run this with MM suggestion of creating a session per thread and check, anything less than 71 is a win for me, I get retry behavior after all
 
If your HTTP requests don't have to be sequential, you throw async or threads at it. No-brainer.
 
are github merges also so slow? Bitbucket now took 1min for a 1 line merge, 1 commit merge, that's just crazy slow
 
maybe I will try with aiohttp as well, I have a couple more hours before I call it a day
 
@Hakaishin I've never had a merge that was perceivably slower than "instantaneous".
 
 
2 hours later…
2:17 PM
Is there a version of this question where not every answer proposes to use .read() or .readlines()? Just for line in file: ...?
 
Not that I'm aware of.
 
Oh, I missed that one answer.
 
"(python >= 2.0)". I think that's a record for backwards compatibility disclaimers I have seen.
 
@mkrieger1 I was also a bit unhappy about the answer ranking on that dupe, if that helps. :/
 
@MisterMiyagi :)
 
2:26 PM
Possibly unpopular opinion: I prefer lines = file.read().split("\n") over lines = [line.strip("\n") for line in file]
 
Truth be told, I wouldn't mind unhammering the new question if you want to answer it. It seems to be better than the old one.
 
readlines() is the worst of both worlds IMO
 
There's a 50% chance the New OP actually wanted to feed the lines to int later on, in which case the newline doesn't matter anyway. :/
 
@MisterMiyagi You do mean this one? But then we would have to remove the strange try/except from the question first, it is only distracting.
 
@mkrieger1 That's the one. In most cases, OP's have accepted me cleaning up their questions like that to be more generally useful.
The few cases when they did not ended in tragedy, chaos and doom anyway...
 
2:33 PM
But no, this is really the correct question, just the top answer includes too many non-preferred options IMO.
 
Guess you're right.
 
We would need a recommended answer *cough*
 
Benevolent Collective For Life?
 
@Kevin I find using lines = (line.strip('\n') for line in file) and consuming lines is generally what I end up using (assuming I don't need it in a list after...)
 
Anyway, I've edited the dupe list. Should cover the problem for anyone willing to read.
map(str.strip, file) anyone?
 
2:42 PM
Stripping all the whitespace, eh?
 
3:04 PM
cbg
 
3:19 PM
cbg
 
3:40 PM
Me: "I finished the project that so many others abandoned halfway through. Some of my finest work, if I do say so myself"
Client: "There's a typo here on the landing page"
 
could we change this colour a little, it looks slightly off. oh, and can we shift this bar to the right. and maybe add another filter.
 
And get rid of the duck
 
fun times.
 
To be fair nobody asked me to create a heartbreaking work of staggering genius, so I couldn't reasonably expect the red carpet and flutes of champagne to be ready at short notice
 
3:47 PM
I'm scared to ping them for a routine checkup because they might say "oh, we're still using the Excel sheet that your project is supposed to replace"
"It's just so flexible, watch, I'll change the color of this column a little... Now that's agile"
 
 
1 hour later…
5:04 PM
@roganjosh I missed your last "last thrashing of Duke 3D" - when you next about mate?
 
 
1 hour later…
6:27 PM
Does someone know whether the array backing locals lives on the stack or on the heap? (CPython)
 
does this answer it?
amusing that you have a comment on that question :P
> Note that python makes no guarantees about memory layout. Are you asking about a specific implementation? Are you asking about the name num, the reference num or the value of num? – MisterMiyagi Aug 23 '20 at 5:41
 
I tend to leave that as a cautionary warning whenever someone asks clever questions about memory with unclever words.
But indeed it answers my questions!
Though I give extra points to the source code commentary itself:
 
 
5 hours later…
11:26 PM
Kinda strange how you raise exceptions to develop better code
 

« first day (3916 days earlier)      last day (1031 days later) »