« first day (3992 days earlier)      last day (939 days later) » 

1:44 AM
You're right .. thanks!
 
@Aran-Fey there's a non-null amount of evilness involved in such a person's endeavour
 
 
1 hour later…
3:12 AM
cabbage guys!
 
 
3 hours later…
6:11 AM
@MisterMiyagi @AndrasDeak @roganjosh Thanks a lot!
Fun fact, my car broke down on my way to the graduation exam
 
6:34 AM
@AlexandreMarcq Wow...
At least you graduated :P
 
 
2 hours later…
8:42 AM
Hi all.
I'm sure someone here can help me out. I've got a ton of fun Python questions, which I assume the most of you can answer.
In Python3, I've got this list which consist of more than a trillion lines.
Part of file:

4XOc8tUCQ2sC2oRqd7yUIC0w
3bVEY6LV4tiOJkks3nBbG08J
4GaPG8oDOwxZjg8UXG25sjI8
V1G7VSLQJ7i9Xhz7z9mxbRTk
vhOT537SuiT53nSUFiN5XoEu

I'm reading this file with Python, and placing each line in the file in a list:

with open('big_list.txt') as f:
my_list = [x.rstrip() for x in f]

However, I'd like python to create this list, and place a certain string in front of it.
 
@KevinC Like: my_list = ['certain string' + x.rstrip() for x in f] ?
 
As in my_list.insert(0, 'a certain string')?
 
Since we're covering more than a trillion lines, I'm not sure how to do this efficient and in a pythonic manner.
Example outcome:

print(my_list)
something_before_eachline-4XOc8tUCQ2sC2oRqd7yUIC0w
something_before_eachline-3bVEY6LV4tiOJkks3nBbG08J
something_before_eachline-4GaPG8oDOwxZjg8UXG25sjI8
something_before_eachline-V1G7VSLQJ7i9Xhz7z9mxbRTk
something_before_eachline-vhOT537SuiT53nSUFiN5XoEu
 
@KevinC Ah that's what I suggested
 
is Kevin C affiliated with Kevin? Did Kevin change his name?
 
8:48 AM
@u1
 
@Hakaishin Kevin is still in this room :)
 
@U12-Forward right :)
 
@U12-Forward Ah, I tried it the wrong way. It seems that is possible and convienant
 
@KevinC So it works?
 
ye
noice
 
8:49 AM
@Hakaishin Yeap!
@KevinC Cool!
 
Another question. I'm using this python module: https://github.com/spyoungtech/grequests

Now, when printing the result of the request, I don't know which url was part of the request. Does anyone know how I could print the outcome e.g:

<Response [200]> 'http://python-requests.org',
None 'http://some-example.com'
 
if you're the one sending the request, shouldnt you always already know what request you sent?
maybe i didn't understand the question
 
@Aran-Fey Reminder: yield is an expression whose value can be set by the generator's send method. Technically, next(g) is equivalent to g.send(None).
 
Looking at the example of the python module, he places 6 url's in the urls list.
The result is the following:
[<Response [200]>, <Response [200]>, <Response [200]>, <Response [200]>, None, <Response [200]>]

Now, this is fine when working with a small amount of requests. However, what if there are 1000 requests to be sent. I won't be able to 'map' the request to the response value.
 
why not
 
9:01 AM
@KevinC I don't think there is any way to work with a trillion lines efficiently. Even if by trillion you mean Tera, not Exa, that's way more than a list can hold or a stream process in a reasonable time.
 
@ParitoshSingh because I don't see it in the results
 
but you have it as a list already, no?
like in the example, the post has url list that has the urls
 
@MisterMiyagi it's an example, but the idea still stands. I'm looking for a file which eventually will have that amount of lines
 
just reuse whatever list you used to make the actual requests, with the outputs.
 
@ParitoshSingh I have a list, but I need to match the response to the value in the list.

E.g. stackoverflow.com -> response 200.
Currently, this is not shown
 
9:09 AM
yes...so is your question how to match it?
if im assuming correctly, map already keeps it in the same order as the input list, so just use a zip
zip(url_list, responses) or whatever
 
The question could be extended to; what is the best method to print the match between the response and the url in the list.
using zip seems a bit counterintuitive. I assume the module itself has a method of returning the url per response, right?
 
> Note: You should probably use requests-threads or requests-futures instead.
 
@KevinC i dont see why. it's like.. "heres an input. here, give me an output but also give me the input again for some reason"
 
@ParitoshSingh well, for convenience
 
you literally already have it
 
9:16 AM
I had a look at the source code and it turns out that the None you get in the output is actually the return value of the exception_handler. So there you go
 
@KevinC I am just warning that whatever approach you design for less content just won't work for that much content.
Been there myself…
 
haha, yeah the project reminds me of when I tried to find my pw in a pw hash dump file. In the end I just let it run over night. It aint pretty but it worked :D
 
At Tera-counts, we're most likely not talking about a single night anymore. :P
 
Yeah, I agree my design could be improved.
 
9:41 AM
Then, my final question:

Taking a big list as the source file. The grequests module takes simply a list, and sends all the items in the list as a requests at once.
So, having this big list of URL's would make no sense, as it would overload my memory and most likely the server(s) as well. What would be a pythonic approach to solve this?
 
Just throw a ThreadPoolExecutor at it
Hmm, is there no async executor in the stdlib anywhere?
 
Nope.
There's only asyncio and despair...
@KevinC I would recommend to already plan for having multiple lists/files. File I/O is very difficult to parallelise otherwise.
Ideally, you want to have multiple threads/processes and just tell each one which file to process.
Pushing all through a main process/thread will be a significant bottleneck.
 
9:56 AM
Exactly my thoughts
Would the threadpoolexecutor suffice
 
Why would file IO be a significant bottleneck in a program that sends HTTP requests? O.o
 
Wouldn't sending websocket requests be more sufficient?
 
Well now we're comparing apples to oranges. All this time you've been sending HTTP requests, so why are you suddenly talking about websockets out of the blue?
 
@KevinC sed -e 's/^/something_before_eachline/' big_list.txt and if this works, slap a -i in-place flag on it
 
Andras, that one has been solved, hehe.
@Aran-Fey I thought communicating with websockets is way more efficient.
I just thought about it
 
10:11 AM
@KevinC with python?
 
ye
 
OK, perhaps you're on windows... otherwise there's absolutely no reason to do this with python
(even on windows there's git-bash and wsl...)
 
I have a macbook
It works fine with python
 
Sorry, I thought you said something about efficiency. I'll let you back to your things.
 
I tested this, it doesn't take that much time to place the string in front of it. In the end, I could indeed always use something like sed to prepare the file before processing it with python
 
10:19 AM
@Aran-Fey Iff we are aiming for reading several TB of data from a single file, it will be a significant bottleneck. As in, doesn't-matter-what-else-you-are-working-with bottleneck. If reading and splitting the data takes several weeks, then whether the requests also take weeks or seconds isn't the only thing to worry about.
The general problem is just that when you're dealing with Tera-counts, you need to be able to (trivially) parallelise every step.
 
Well, but there's nothing you can do to optimize the actual reading. Reading from 2 files isn't faster than reading from 1 file, is it? The only thing you can optimize is how the data is distributed to the workers. Is that really worth worrying about?
 
Well, if a request is 0.1s, or 0.01s, it would make a huge difference.
Well, I'm just looking the most efficient, most effective, and most pythonic setup and writte python code to achieve my goal. As I'm working on this project, I believe I'll re-use the code in the future as well. Meaning, yes, somethings are valid to worry about.
Again, i'm not an expert in the field of python programming. That's why I'm asking it here
 
stackify.com/premature-optimization-evil generally python people will respond with following link.
 
@Aran-Fey If we're talking single HDD, then yes we're yammed – probably not just because of the HDD, though. SSD scale pretty well for concurrent accesses, and multiple HDD are decent depending on the filesystem.
Either way, I'd hope we're not just talking about doing this on a single macbook…
We aren't, right?
 
10:36 AM
SSD
 
Oh right, SSDs are a thing nowadays
 
Well, I tested already parts. It seems that the reading/writing text isn't the problem. The problem lies most in the single request which is send/received everytime from the servers
My macbook/SSD can handle everything without issue.
It's that the reply from the server isn't that fast, it's about 5 requests per second.
 
Is whatever server you are sending the requests to actually prepared to handle them that quickly?
 
Yes, easily
When using requests, it handles about 5 requests per second. Using the other module, about 80 per second
 
11:03 AM
By "requests", do you mean the regular requests module or grequests?
 
regular requests
 
Unless you add your own concurrency like threads on top, requests will handle one request after the other.
That's inefficient for network related tasks.
 
Indeed, that's the reason. But aren't there better written modules for this use case?
 
Depends on how you want to use them. Concurrency depends on your code as much as it depends on the libraries.
The simplest approach is just to add threads on top. grequests/gevent should feel similar to threads, but shifts slightly towards efficiency vs simplicity. If you want to go all in, you'll need some async library.
 
11:27 AM
All-in? Sounds legit.
async... haven't used it before.. I'll find some examples
 
 
2 hours later…
12:59 PM
Are there any guidelines on how descriptors are expected to cooperate? I've written a custom descriptor for wrapping e.g. classmethod and expected that I have to call __set_name__ of the wrapped descriptor. Apparently, that's not the case. Many dutiful stacks lost their livetime that day…
 
I don't recall seeing guidelines, having spent perhaps an hour reading about descriptors over the course of my entire life
 
Is that you, KevinClone20092021? *does the barcode squint*
 
How I love Laptops with Password stickers on them :D
 
I don't know my own Kevin identification number. You'd have to ask the technician in charge of my nutrient tank.
 
1:14 PM
For real now, here I thought the unchartered territory of descriptors would be one of your preferred hunting grounds for chaotic weirdness.
 
Reading docs.python.org/3/reference/datamodel.html#object.__set_name__, it sounds like your descriptor should define its __set_name__ so that it calls __set_name__ on the descriptors that it wraps. But I'm guessing you tried that and the outcome was unexpected.
 
Let's say there wasn't a general aura of rejoicing when I tried.
 
i can be here for moral support if you like. i'm great at rejoicing especially at inopportune times.
 
I wonder if the source code for classmethod has any interesting insights, since it too can wrap other descriptors. As of 3.9.
 
That's an interesting idea… *heads to the science cave*
 
1:21 PM
I don't think I've ever seen a descriptor that did something meaningful in its __set_name__
 
Swapping classmethod and my descriptor leads to less rejoicing. p=0.05
 
Here is classmethod's implementation. I don't understand any of it.
 
1:41 PM
docs.python.org/3/howto/descriptor.html#customized-names gives a practical example of __set_name__, but doesn't talk about nested descriptors or best practices or anything
I get the impression that __set_name__ is kind of like __matmul__ in the sense that most stdlib types don't implement it, and the language doesn't have a strong opinion about what it should do
Perhaps standards will crystallize somewhat in the near future, since evidently some man-hours are being put towards support for nested descriptors. Granted, The commit that makes nested classmethods work changes only a single variable, so the number of man hours might be 0.01.
 
2:08 PM
python.org/dev/peps/pep-0487 has some interesting examples of practical __set_name__ usage. Namely WeakAttribute and Trait
 
2:42 PM
man 2 weeks of holidays and already I'm out of xkcd, smbc and dilberts :(
anybody knows any other good comics?
or 10-30s pastimes?
 
user16657590
should a mathematician use python or other language?
 
Yes, both
Ideally, everybody would know at least five programming languages by the time they turn 18
 
user16657590
Ohh Thanks!!
 
@Hakaishin I have about 80 in my RSS feed. I wrote up descriptions of many of them a couple years ago, I'll see if i can find it.
 
@Kevin uh exciting :)
also, rss feeds are still a thing?
 
2:56 PM
They're still a thing, despite a precipitous drop in popularity after the death of Google Reader. Myself, I migrated to Feedly.
Zero hits for "comic" or "webcomic" in the chat room transcript and in my gists... Where the heck did I put this thing
 
@Kevin Looks like selection bias got me there. The only stdlib descriptor I looked at lately was functools.cached_propery which does use __set_name__. :/
It seems as if only descriptors that actually need the name use have the method.
 
The nice thing about open source is getting back from holidays to find the awesome new features you wanted have been implemented, praise the people volunteering :)
 
 
3 hours later…
5:44 PM
recbg
@MisterMiyagi cached_property is b0rken
:(
 
Define "broken"
 
6:12 PM
"The logic is hairy, so I've added Serhiy and Tim to the nosy list to help think it through." I'm sure this hairy nose will get a good picking
 
@Kevin yes that broken.
 
I accept "slower than it needs to be, for no good reason" as a valid definition of "broken"
 
@Kevin it is more like: "I accept that an interstate road wherein just only one car can move at any instance is a valid definition of broken"
 
When I am fabulously wealthy, I will construct an interstate that nobody else is allowed to use while I am driving on it
 
6:30 PM
PYTHON!
 
Yes
 
@AnttiHaapala Wow. How the yam didn't they notice that before it was released...
 
They created 300 interstate roads and put one car on each, and they all got to their destinations in a reasonable time
On my interstate, even if those 300 cars aren't anywhere near me, they will still be gently nudged off the road using powerful fans installed in the median
 
For those of y'all who don't hang out on reddit, check out this masterclass of trolling. The OP has better python skills than everyone in the comment section combined
7
 
6:49 PM
@Aran-Fey brilliant
 
@PM2Ring because it was too complicated
same thing all over: async is too complicated. pydantic is too complicated etc...
 
Only the guy at the bottom of the comments section I'm convinced is 100% clueless. The ones that ask "are you joking?" are at least suspicious, and I half-suspect the long effortpost guy is knowingly playing the "book-smart yet oblivious" role
 
this would make a nice interview question
 
I'm not well-acquainted with type annotation syntax, so I couldn't deduce much about the program's behavior until I ran it and experimented a bit. Negative one quatloos to the one guy who dismisses it as having syntax errors without running it.
 
it doesn't use the type annotations for anything. Of course it doesn't.
 
7:10 PM
Yeah, it's pretty hard to make annotations do something
 
there is the walrus in the annotations
wat, did general variable annotations only become a thing in 3.8? :D
 
I don't see walrus in annotations
 
puts: ((str) := void) = (print), and its cohort I believe
 
@Kevin especially since the TIO link contains runtime results as well
@Kevin aaah, missed that, thanks
not that it does anything
 
I also missed that the first couple of look throughs, but I always pentuple-check before publicly disagreeing with Antti
 
7:21 PM
we're practically cousins so it's OK
 
Valid
 
TIL thr position of the *args parameter can make a significant difference.
def f(a, b, c=23, d=42, *x):
    print(a, b, c, d, x)
f(1,2,3,4,5,6)  # prints (1, 2, 3, 4, (5, 6))
def f(a, b, *x, c=23, d=42):
    print(a, b, c, d, x)
f(1,2,3,4,5,6)  # prints 1 2 23 42 (3, 4, 5, 6)
How come I've only just noticed this?
 
Sounds like it's more about the position of the kwargs. Subtle difference in my head. They are the ones who are optional positional vs keyword-only.
 
I've also had a related epiphany recently. One day I thought, "wait, parameters before a *args can still function as keyword arguments?" and the answer is yes, yes they can
 
that's why we needed def f(/, a) to create positional-only kwargs in native python
umm, not exactly that
 
7:36 PM
The defaulted positionals are weird.
def f(a=42, /, **kwargs): print(a, kwargs)
f(a=1)            # prints 42 {'a': 1}
 
weird that you can't define a single positional-only arg, only two or more
 
Our enemies must never know of our weird and hairy parameter syntax, for they will point and laugh at us
 
def f(a, /, b): pass is OK
 
Is it, though? </meme>
 
Where's Miyagi's magic 8-ball when you really need it...
 
7:38 PM
@Kevin Having to document it is quite an exacting task.
 
@holdenweb just leave it as an exercise to the reader
 
@AndrasDeak Hmm? def f(a, /): works just fine
 
The specification is "whatever the parser accepts"
 
@Aran-Fey oh poop I kept thinking about it backwards, thanks
 
Whoops, that's the KevinScript specification... Move along, nothing to see here
 
7:40 PM
@Kevin "I am altering the specification. Pray I do not alter it any further."
 
"Your prayers are ineffective because KS' code is housed on unhallowed ground, but it's still a reaffirming therapeutic exercise, so go ahead and pray anyway"
 
7:53 PM
> convert-im6.q16: attempt to perform an operation not allowed by the security policy `PDF' @ error/constitute.c/IsCoderAuthorized/421.
welp
(imagemagick gradually refuses to parse PDFs due to inherent vulnerabilities bugs.archlinux.org/task/60580)
 
"You may enable it locally by removing 'PDF' from below line:" -- oh good, they didn't weld the footgun's safety closed
 
fortunately all attacks in the pdf were put in by yours truly
Ken Thompson's pdfLaTeX compiler not withstanding...
 
My sophisticated Thompson-Kevinson hack works without a compiler, and it's so undetectable, it has no observable impact on physical reality whatsoever
A bit of googling reveals a github repo with a proof of concept for this pdf vulnerability, designed specifically to target ImageMagick. So much for my usual strategy of shrugging and saying "maybe nobody will bother to exploit it?"
Ugh it's written in Python 2.7
 
that's how you know they mean it
 
8:09 PM
He's a madman with nothing to lose!
 
Hi everyone.
 
Hello.
 
Today I woke up hoping to figure out how relativistic physics works. Current progress: I don't know how pulleys work.
 
Relativistic or non-relativistic pulleys? I suspect relativity will mess with the concept of an ideal rope.
I can help with non-relativistic ones ;)
 
8:24 PM
I wish I had learned about tension before I transferred out of mechanical engineering. Or if the horror stories I've heard about statics are true, then maybe it's for the best.
 
elasticity was invented by the devil
 
8:48 PM
Just model each atom as a rigid sphere, easy
 
 
2 hours later…
10:49 PM
Is there any practical difference when saying "the scope of the function" VS "the namespace of the function"
 
Both are equally vague IMO
 
11:37 PM
cabbage
 
11:59 PM
Hello I have a problem trying to run a local python server from pycahrm CE with FastApi. It says typeError: post() missing 1 required positional argument: 'path' at the route operation although I have set a path to it (that would be called from a html form).
 

« first day (3992 days earlier)      last day (939 days later) »