« first day (3155 days earlier)      last day (1794 days later) » 

9:04 PM
its useful in the sense that you have to click one less time to see the answer, thus saving you bandwidth
:p
 
wim
@cs95 they meant late answers with info which is just duplicating content in other answer on same Q.
 
ah
by all means dv away
 
wim
which does the opposite of save people's bandwidth / page load time
 
actually, don't downvote those. flag for removal with a link to the original answer.
 
wim
pet peeve: people that insist they want to make the float have 2 decimal digits when they really just want to represent it with 2 decimal places
 
9:09 PM
save your votes
 
I think I've only run out of vote votes once or twice
 
@cs95 wth. I've just gone back and seen that select_dtypes has always had include/exclude and not just exclusively include. What was going through my head when I put that together?
 
fix it please :D
 
I am doing now
I think it's because the OP started with "drop" columns, and that set my mind off
 
much better
although this led me to an interesting discovery: df.dtypes == object works but df.dtypes.isin([object]) does not
technically they should be doing the same thing. But it appears it has to do with the fact that df.dtypes returns a series of numpy dtype objects. == is doing something that isin isn't.
 
9:18 PM
wat
 
wim
 
Good afternoon...
 
@cs95 how about np.isin(df.dtypes, [object])?
I hope at least that's sane
 
I have a problem, I would like a little of help if somebody can.
I have a crawler... But I want to make it scalable.
 
@AndrasDeak yup! that works... now that you mention it, df.dtypes.isin([np.dtype('O')]) works when df.dtypes.isin([object]) did not
 
9:19 PM
Jun 3 at 18:58, by Victor Martinez
I'm going to read more about that.
 
hmmmmmm
 
@VictorMartinez did the first round work out? ^
 
My idea is, if I have 20 sites to crawl, I want to launch 20 Dockers. But I don't know yet how to connect that to docker. I mean, some kind of queues.
 
MWE:
In [189]: df = pd.DataFrame({'x': ['a', 'b', 'c'], 'y': [1, 2, 3], 'z': ['d', 'e', 'f']})

In [190]:  df.dtypes.isin([object])
Out[190]:
x    False
y    False
z    False
dtype: bool

In [191]: df.dtypes.isin([np.dtype('O')])
Out[191]:
x     True
y    False
z     True
dtype: bool
 
@AndrasDeak What do you mean with the first round?
 
9:21 PM
You asked about the same general problem on Monday, also about docker or kubernetes for scalability (???), then you were given suggestions such as multiprocessing
 
@cs95 df.dtypes.isin(['object'])
It needs to be a string
 
I just don't want others to replay the same discussion if there's something you've learned on/since Monday
 
Jaaja oh great, you remember : )
 
Which is consistent with the API for select_dtypes etc
 
I tried Multiproccessing, but I had some problems with it. I wanted to give a try to Docker
 
9:22 PM
@roganjosh lol what version does this work for you?
 
I don't know docker. People who know docker: does it make any sense to try to solve "scalability" of a crawler with docker?
 
In [196]: df.dtypes.isin(['object'])
Out[196]:
x    False
y    False
z    False
dtype: bool

In [197]: pd.__version__
Out[197]: '0.24.2'
 
Just to see how it works.
 
... No way
0.20.3
 
@AndrasDeak Do you think that the best solution to scalability is Multiprocessing?
 
9:23 PM
@AndrasDeak I don't know docker but I think the only reason to use docker is to avoid giving other peeps the burden of installing hundreds of dependencies manually to run your project
OK, sounds like a bug, think I'll open a question...
 
@VictorMartinez I don't want to answer that question because I know next to nothing about web stuff
 
df.dtypes.isin(['object'])
Out[85]:
x     True
y    False
z     True
dtype: bool
 
but from my also-limited impression of docker I don't think that's the solution
but this is just a guess
wow, pandas docs calls llamas lamas pandas.pydata.org/pandas-docs/stable/reference/api/…
 
wim
@AndrasDeak yes
 
OK, thanks :)
 
wim
9:26 PM
but docker would be one piece of a much bigger picture, not a magic bullet
 
    result = algorithms.isin(self, values)
    return self._constructor(result, index=self.index).__finalize__(self)
pandas source code <3
 
@cs95 what's more confusing is that they said they wanted to move more away from numpy a-la pandas 2.0 so I don't know why they'd switch from python object to np.dtype('O')
 
I think I find some throubles with Multiprocessing, mostly because is not so good with handling multiple network connections, the crawler is builded with Requests library.
 
> # GH16012
# Ensure np.in1d doesn't get object types or it *may* throw an exception
 
wim
if a crawler is not scaling I'll bet you 10 bob that you are IO-bound and multiprocessing won't help you
 
9:29 PM
seems that object might be special-cased ^^ though I haven't checked which side of the operator is checked for being object
 
pandas is the one library that I just refuse to update in my base environment because of these things. I guess it's very quickly going to result in me not being able to reproduce a huge range of issues or try answer questions, but I need stability for my sanity
 
Hahaha, very wrong country
(read in a laughing-with-you kind of way)
 
We have a strong and stable govt, thank you very much
 
@AndrasDeak if you can shed some light, I'd welcome an answer
 
<screaming internally>
 
9:34 PM
@cs95 nah, thanks, I don't have another hour and a half to write a pandas answer ;D
 
wim
it's a shame ignored tags don't work in chat
 
well you can ignore cs and piR and most of the pandas here will be covered :D
 
is there some kind of ongoing feud between pandas and badgers?
 
No, I don't think so
 
then it's settled
 
wim
9:38 PM
badgers are pretty good at killing pythons
 
International politics has a lot to learn from this discussion
 
wim
I don't think they kill pandas, but those guys are almost extinct anyway
 
"Are we at war?" Someone: "Nope". "Ok then, good day, Sir"
 
wim
whoosh
 
9:41 PM
@AndrasDeak yeah. "pooping your pants" doesn't quite cover his situation
 
gold badgers are a special subcategory of animal that wield hammers. There's an ongoing battle between gold badgers and pandas who enjoy farming.
 
wim
heh
found badger and red fox webcam ... getting closer
does china even have badgers?!
 
pictured: badger not giving a yam
 
Literally just looking up where they live en.wikipedia.org/wiki/Badger#Distribution
 
wim
I guess most animals know not to mess with badgers
 
9:46 PM
I still wonder what happened here
 
When I was staying at my dad's in the countryside, I heard badger encounter stories all the time. People being chased down lanes screaming, mostly.
 
probably humans
 
cbg
 
cbg
 
If I am using the length of a list as part of an [INFO] print statement during a loop that is doing some processing, should I find the length of the list and store it in a variable or is it OK to call len(my_list) in the string formatting with each print statement? Is there a big performance hit or other reason to store the length in a variable is basically what I am asking.
 
wim
9:47 PM
nice gif, hadn't seen it
 
colorised depiction of gold badgers running to close dupe
 
wim
IIRC one of the US presidents had a pet badger. I forget which
 
@wim It's one of the classic nope gifs. Just like this one
 
@Dodge you can time it. But len() is pretty cheap. Still, I don't know what you gain by not calling it once instead of every single time
 
length of a list is O(1)
 
wim
9:48 PM
thanks, will use it in next code review at WimCorp
 
@roganjosh I was thinking that finding length on list might be super cheap
 
the list knows its length
 
wim
call len every time
 
It is cheap
 
cool cool, thanks
 
@wim why?
 
wim
@roganjosh see here
 
Yeah, we posted at the same time
But it still has a function call overhead to call it every time. I don't see what you gain from calling it repeatedly
 
clarity perhaps, not having to have length_of_list = len(lst)
 
@wim that was useful, gracias
 
wim
9:53 PM
@roganjosh they said part of an [INFO] print statement
sounds like stdlib logging, which means you already have a billion function calls in your stack anyway
 
I'd rather have a little bit of overhead than an additional mostly useless variable in my code
 
wim
for every person that needs to worry about function call overhead there are a thousand people that worry about function call overhead
 
@wim No, something much more simple. Image pre-processing for training an image classifier and need to know how things are doing when loading thousands of images.
I initially wanted the variable because I knew it must be less work for the computer but then I thought that finding length on list is something that has been probably made as efficient as possible at this point in Python development, hence the question. I'll just call len() until I need to squeeze every drop of performance out of my computer.
 
if you're doing image classification: this will never be your bottleneck
profile first, optimize after
 
True, good advice
 
That's a youtube video
The voice narrating it is annoying, but I do enjoy the content
 
Wow, Hungarian version is creepy. Reading about programming in Hungarian is, I mean.
 
@cs95 look into htable.ismember_object. That comparison is where the equality breaks down on my machine at least
It's the fallback for inclusion checking for Series of dtype object
 
10:22 PM
@user3483203 I must be slow today, but why would that make a difference?
 
@AndrasDeak The speed at which you process English and type it back out; I'm curious whether you still have a preference for reading Hungarian (as in, you'll look for the translation button in general rather than curiosity)?
 
I read everything in English if it's originally in English
I only checked the translation out of curiosity
I've also been reading every book in English for a long while
 
does that include books that were written in hungarian originally?
 
ha, no, though those don't often get translated into English I think (and they rarely interest me anyway :P)
 
My dad's wife is Polish, has lived here for nearly 10 years, but all her books are in Polish and she prefers Polish subtitles on films. I don't speak another language (I can botch French if I have to) so I've always found it curious how the mind works with multiple languages
 
10:37 PM
I learned that, given the choice, you should always pick the original over a translation. Other than that, I don't really think about it.
 
@roganjosh I started learning English at age 8, but watched cartoon network even before that. That's the main reason I think why my English is good, and when I speak English I also think in English. I've even had dreams where I spoke English. Russian, on the other hand, was very different, even though I learned it well enough by the end of high school. I would still find myself thinking about what I want to say in Hungarian and trying to translate that into Russian.
There are a lot of studies that languages encountered young enough get handled by different parts of the brain (same parts as the native language), while later languages have higher cognitive overhead
 
I learned internet english on a pokemon forum ... good times
 
@Aran-Fey i agree. I'm sorry I don't speak German, so Neverending Story was one of the books which I read in Hungarian even though I'd love to read it in the original.
I started reading some Dostoyevsky in Russian but it was too advanced for me to appreciate it
 
If it's any consolation, I've read the Neverending Story and I found it meh
 
:) Do you otherwise like fantasy?
 
10:40 PM
fun gimmick but otherwise lacking, IMO
Yeah, fantasy is pretty much the only thing I read
 
well, yeah, I liked it, but I wouldn't think to mention Michael Ende as a favourite
 
I never realized, but that's pretty ironic how the author of the never ending story is called Ende
 
hehe
 
wim
still reading it .. no spoilers please
 
Has anyone here read DJ MacHale's "Pendragon" series? Wondering what other people think of it
 
10:50 PM
@cs95 the comment trail on your question is very confusing. Does it invalidate my other answer for you or does it go down a different code path?
 
@wim I see what you did there...I think
 
I wanna know whether to brace for "this doesn't work" and try to understand what Pandas has done in later versions
 
@roganjosh by "your other answer" do you mean the fact that it worked on older versions but not now?
the comment trail is mainly people arguing about what version the bug appears on
 
I mean this
 
nah they're not related
 
10:53 PM
Ok, that's good :P
 
select_dtypes does its job. I just tried to emulate select_dtypes with these methods and found the bug
 
I would have thought that select_dtypes would have been a convenience function around isin() tbh
I'm too tired tonight to look through pandas source properly, but your approach seems like a first-principles shot at the expected result, but it fails, and select_dtypes doesn't suffer for it.
 
yes, you've put it nicely.
 

« first day (3155 days earlier)      last day (1794 days later) »