« first day (2997 days earlier)      last day (1952 days later) » 

5:09 AM
For your feedback/comments: Type error with classmethod constructors is a good question where the OP was trying to call __init__() from a factory-function classmethod. But see my comments on whether their use-case should have been better solved by overloading __init__(self, x=None, y=None) to handle multiple optional args in the first place, not having the classmethod perpetrate LOD violations...
... (Should I post an answer saying so?)
Obviously there is a use-case for factory-function classmethods, but the OP hasn't shown they need that.
and Happy New Year cbg to all
 
 
1 hour later…
6:33 AM
Hi guys. Are there any recommendations to free proxies that I can use that does not keep timing out? I need them for webscraping. If none please suggest a good site that offer premium/paid proxies if possible :D
 
6:46 AM
@Pherdindy If you are going to use premium, might as well try scrapinghub.com which is listed on the "commercial support" tab at Scrapy - scrapy.org/companies
 
7:28 AM
@shad0w_wa1k3r thanks! I'll look into it
 
 
4 hours later…
11:08 AM
anyone got a clue where peer_id from here comes from? github.com/centricular/gstwebrtc-demos/blob/master/signalling/…
 
11:45 AM
@towc odds are it should be uid or something else...
 
right?
 
That name should not exist in that scope
 
so you can confirm there's something wrong there?
 
unless weird black magic, but I'm not familiar with that
I'd first ask if it's a bug :)
 
12:10 PM
@d_kennetz Thanks for reading!
Added the section "Useful Canonicals" to the pandas tag wiki. Perhaps we can add this list to the SOPython canonical list as well, although I doubt it will get much visibility there.
 
every once in a while smci goes on a dupe frenzy, occasionally leading to entries in the canon list
 
12:26 PM
gotta appreciate their enthusiasm, though! If it was up to me, I would just focus on cleaning up whatever I've answered, and nothing more (which I do). Doing any more would lead to a burnout very fast.
 
yup
 
1:27 PM
@piRSquared I think I will have time for one last post before school begins. There is a widely believed perception that for loops are blanket "bad" and "taboo" when working with pandas. I am going to try and convince people otherwise, providing specific examples and use cases where list comprehensions would outperform equivalent pandas code. Mostly with string and regex operations.
Thoughts?
 
FWIW list comprehensions are not just for loops
 
2:02 PM
quite so, they are more like for loops++. I have seen instances where vanilla loops still outperform equivalent pandas functions, although I would not suggest that unless error handling is required.
 
whatever you do though, you have to need a crapton of disclaimers because there's a reason that taboo came along
or put it in a different way, would you disagree or agree to the following? : "for loops with pandas are blanket good"
Because for people who have a sense of why loops are bad, your post would prove useful, but if not handled carefully, it could be very misleading to someone whos new to pandas.
 
@ParitoshSingh There is no doubt that loops or comprehensions are not as appropriate for numerical computation (because cython functions are already loopy implementations on steroids; vanilla cpython loops will still not compare), but the thing not many understand is that string functions are not "vectorised" in the true sense of "vectorisation" because of the inherent difficulty in doing so.
 
Indeed, myself included! Im fairly new to programming and python, and i love reading about these common gotchas and things.
 
IS THERE A DIFFERENCE IN USING MULTIPLE RQ WORKERS VS MULTIPROCESSING
 
(and side note, maybe i'll revisit pandas someday, but for now, ive just learnt to hate/avoid it. everything is just faster if i don't use pandas. Oh well, one day i'll run into a problem that pandas should be the best approach for. but i haven't seen it yet)
 
2:08 PM
@ParitoshSingh Here is a post I wrote a while back, and is a good example where string functions are outclassed by for loops. The caveat here is that they do not handle exceptions, mixed dtypes, or NaNs as well, those require some loop massaging.
 
@Prometheus multiprocessing would be easier to setup but limited to one machine in layman terms. RQ would require more abstraction and setting up, but can be scaled to different machines. The two are inherently different though share similarities as well.
 
LETS say I have a code that scrapes data. say for 1000 URL's. Which option would be faster? Multiple rq workers vs Python multiprocessing
 
ooh, iterating through a list is faster than iterating through a pandas series then? you mentioned using df['text'].tolist() for performance.
 
@Prometheus please don't shout
@coldspeed so the point you should try to make is not "loops can be fast", it's "some builtins are as slow as loops"
 
oops
 
2:14 PM
@AndrasDeak you're right, that is definitely a more appropriate way to say it
 
3:09 PM
cbg
So I have around 6 million HTML files and now want to parse them. What would be the best way to do this?
 
I'll go out on a limb and say "with an HTML parser"
 
BeautifulSoup with mutithreading was slow
 
no, 6 million is just a lot to ask from anyone.
if you expect it to be fast, you might want to invest in a few supercomputers or something.
 
'supercomputers'
 
@MayurBhangale what size are they? Similar format? What are the representing?
 
3:18 PM
@towc entire dataset is around 4.5 GB. All of them are HTML files. I can parse them with selectolax. Was getting hints from others if I can make the process much better.
 
if they're completely arbitrary, probably best thing you can do is parse them once, then write to other files in a format that only contains what you care about and is much faster to parse, and then perform the next searches on that
@MayurBhangale not file format, but format inside the files. Are they all the exact same tags with different data, or random webpage samples, for example
 
Random webpage samples
 
and question is, what are you using this for? It might be ok for it to take a long time once
 
I'm using it for an experiment which sort of does analysis on sublinks within pages. If it succeeds (which it did on smaller dataset), I will have to run the script every month or so
 
run once, keep what you want to know and a checksum. Next time, compare checksums to detect change, if so, delete the previous entry and run the parser there
 
3:26 PM
Thanks
 
and yes, maybe there are faster ways to parse it, but I doubt it will ever finish in less than 5 minutes
how long is it taking now?
 
It's taking around 40 mins right now
 
that's not too bad
can you multithread with what you have?
are you doing it?
 
I'm already. Using 80 threads.
 
oh, ouch
how many cores do you have?
having more threads than cores might actually slow it down
and the parser might be using threads itself
try a single thread, run htop, and take a look at core usage variations
 
3:30 PM
Yes, thats the problem.
I have a quad core CPU right now.
 
right, don't use more than 4 threads
also, I might be wrong, but this is my current understanding
 
Yup
If you're CPU bound there's no point in more processes than cores
 
Thanks, let me try with 4
 
again, try 1 with htop
there are good chances the parser is using multiple threads itself
 
3:40 PM
and then 6M fs requests are expensive too. Maybe try to find a way to chain the files beforehand
and split them in memory
 
and huh, are your files about 800B long on average?
becuase that's weird for html files
 
Some of them were empty
 
so, my .profile is ~800B long. Running this script gives me how many ms it takes for the file to be fetched and read: t = time.time() * 1000; f = open("/home/user/.profile", "r").read(); (time.time() * 1000) - t
that happens to be this value: 0.365966796875
now, this gives me how many minutes it takes to even just do the fs stuff: 0.3659667968 * 6e6 / 1000 / 60
and guess what: 36.596679679999994
if the time it currently takes is 40 mins, 91% of the time is spent doing fs stuff
or maybe the library you're using does its own fs optimizations, and doesn't just use open().read()
 
Maybe.
 
4:05 PM
you have to read one way or another and IO will be slowest
Threading might actually help
 
4:20 PM
Or buy an SSD :P
 
4:51 PM
I understand the HNQ, but why the other one?
Actually, don't answer that, it doesn't matter.
 
I presume that's aimed at me because I interacted with the HNQ you answered. I didn't downvote that answer of yours nor "the other one", whichever that is
 
No Andras, not you :)
 
Of course :D
Nice save
 
funny the kind of sleuthing you can do by playing with the delete button
;)
 
Very smart, I give you that
 
4:58 PM
And while we are on the topic, I have made it clear I would not close it myself, but I would not protest if someone else were to, because I get they're not identical but related enough to justify. So, have at it
 
Actually, it doesn't matter
 
6:01 PM
Cabbage
So I'm trying to use the win32api for Python to implement an independent study project
Essentially creating an extension enabling accent long press in Windows
However, I'm just wondering if anyone has any ideas where to start or which functions I need
It appears to me that I will have to do the following:
- determine the active application with focus
- determine in the input field in that active application
- block input/access to all other applications momentarily
- create a secondary window to which I will switch focus
- that window will receive an input as to which accented version of the character is desired
- there will be a "context switch" back to the first active application
 
Hi Malik. I don't think anyone here will design a full app for you
 
- the stored selection will be returned to the input in there
I understand it from a design perspective
I just don't know where to start in terms of code
 
What do you need help with. This is a big spec
 
I need to find a set of tools to achieve my goal
Basically, I don't know which functions facilitate these sub tasks
 
So in terms of code you mean you're on windows and looking for infrastructure help, or that you don't know the Python libraries
 
6:07 PM
I mean pywin32 is based on the standard win32 libs so it's not really a matter of Python
I'm pretty fluent with Python
Yes infrastructure
I just need help understanding what features are accessible via which APIs
 
Isn't that all in docs
 
I don't know about you but low level OS docs make no sense to me
 
lol
 
And documentation is kinda sparse to be honest
 
Well then proceed. I was just confused about the list you were writing. Out. I won't interrupt if you write it out again. GLHF
 
6:12 PM
They don't teach anything like this in school, would be nice to be shown the ropes
 
6:36 PM
expecting school (or even uni) to teach you relevant life skillz is generally a bad idea, as far as I can tell
 
unless your life is in academia :P
 
can't argue with that :p
 
 
2 hours later…
8:28 PM
Of course lol
I'm well aware
 

« first day (2997 days earlier)      last day (1952 days later) »