« first day (4271 days earlier)      last day (660 days later) » 

12:02 AM
stackoverflow.com/questions/72757932/… is this the best duplicate here? Do we actually have something for "when I write for x in y:, what does the x actually mean, and what values will it have?"?
 
 
8 hours later…
7:59 AM
Umm folks... has anyone heard from PM2 since end of May?
 
 
1 hour later…
I meant more why he hasn't popped his head in here... I know I've been distracted and not about much myself - just concerned I might have missed something is all
 
How dare a user post somewhere other than Stack Overflow!
 
@JonClements he'd been away for long stretches before
Half a year easily
You can both directed reply and superping him :P
 
10:16 AM
@AndrasDeak--СлаваУкраїні I'll guess I'll have to make notes of some music he might like... I think @Cody might prefer he was around though so I wasn't picking on him in the mod room with "favourite" tracks :p
 
Hey, I learned something new. I'm not complaining.
 
/me slips Cody the $20 for saying that as agreed...
Is it just me - or does it feel like every other (okay - exaggeration) Python question now seems to be about scraping
(or as most spell it "scrapping")
 
Python users are scrappy?
 
10:33 AM
Mostly seems to be people with no idea what they're doing scraping products (or similar) off websites
(and the myriad of libraries either builtin or otherwise work with Python that can do that is quite large - so it's a natural go to...)
 
11:09 AM
@JonClements There's currently a sizeable chunk of "how do I install library X" and "I want to solve <waves hand> with ML" as well, as far as I can tell.
If you just mean that there are a lot of scraping questions... yeah, it's been quite some lately.
 
and some stuff like that one...
it's getting harder to be generous these days towards blatantly bad mentors just leaving their mentees in the lurch... or just that people aren't even bothering to do such simple things
I mean I do a little bit of work for a guy that spends an hour/two with me a week while he's doing a course to go through some of it - after he's either done well or not
or he comes to me and says... "this is something I'm likely to come up against but what I don't get it is... can you help my thinking on it..."
(and I enjoy that)
 
@AndrasDeak--СлаваУкраїні meh... he's all right when you get to know him :p
 
Hi
 
hi there
 
I have a problem formatting my graph. Can I ask my question here?
 
if you're not referring to a question you've recently asked on the main site - then yeah... when you say graph... do you mean graph/chart?
 
11:41 AM
Yep, a matplotlib graph
 
I've no objections... (unlikely myself to be able to help much though... but...)
 
Thank you
I get the following error:
 
May 19 at 14:20, by Andras Deak -- Слава Україні
@rb3652 please post text as text, not images
 
97832 pixels does seem a tiny bit excessive...
 
OK
Also, the graph itself is very simple so I'm not sure why it's says that many pixels
 
11:44 AM
That's, like, 150 monkey islands.
 
@rb3652 any chance we can see your actual code?
 
'
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from matplotlib.pyplot import figure

data = {'2:31': -6.78787878,
'2:32': -6.87878787,
'2:33': -6.987987987,
'3:31': -6.78787878,
'3:32': -6.87878787,
'3:33': -6.987987987,
'6:31': -6.78787878,
'6:32': -6.87878787,
'6:33': -6.987987987,
'9:31': -6.78787878,
'9:32': -6.87878787,
'9:33': -6.987987987
}
names = list(data.keys())
values = list(data.values())

fig, ax = plt.subplots()
ax.plot(names, values)
plt.gcf().set_size_inches(8, 6)
That's my code (slightly modified)
When you run it, you should see the following plot:
 
why import seaborn and pandas? the code doesn't seem to use it.
 
@MisterMiyagi I know, I was planning on trying different styles for the future, so I thought I'd import seaborn for now
As for pandas, I used it for old code (because I originally put the data in a df) and it's left over.
 
@MisterMiyagi I once inadvertently brought a cluster down trying to generate a 300billion by 100billion pixel - so this one I'd shrug off :p
 
11:47 AM
As you can see, there's only 12 points and the code is very simple
 
It also doesn't cause the error.
 
@MisterMiyagi I just changed the data-values slightly (i.e., instead of 6.78124315512, i just put 6.78787878)
That stops the error, but I still get the error for my original data, even though it's basically the same (except for a few decimal digits)
 
@JonClements There's a huge line between madness and eccentricity. I am happy to congratulate you on successfully crossing it!
@rb3652 Well, giving us the code including the data that triggers the error sure would be helpful.
FWIW, replacing 6.78787878 with 6.78124315512 in the above code also doesn't trigger an error for me.
 
@MisterMiyagi open mouth emoji
 
OK, let me see, please wait
 
11:51 AM
Any chance it has something to do with screen resolution?
 
I'm also looking forward to the future question of "why are my points positioned weird along the x axis" and "why does the same code plot differently on this old system with python 3.5"
@Aran-Fey naaaah, wait for it
 
@MisterMiyagi both accepted with thanks? :p
 
Ok, I'll shut up and defer to the experts
 
Who?
 
I mean it could be anything, but based on past history I expect user error
we'll never know until we see the actual code
 
11:53 AM
@MisterMiyagi It may be an empty set, but that's not my problem!
 
@MisterMiyagi cthulu?
 
Hmmmm, squid...
 
ohhhhhh I figured it out! I'm a goddamn genius!
 
it also seems to be inline plotting in a jupyter notebook, so resolution seems less likely perhaps
@rb3652 of course you are!
 
I had text on the image that was too high up, causing matplotlib to exaggerate the image size.
 
11:54 AM
no worries... we'll all send you an invoice soon
£3000 split errr... 6 ways... everyone else happy with that?
@rb3652 I take it that you're cool with that?
 
I'd prefer per-pixel billing
 
Wait, you guys are getting paid?
 
@AndrasDeak--СлаваУкраїні yeah... but that'd just be unfair... plus annoying for the dark council to breakdown the RGBA aspects of it as well...
 
12:54 PM
I'll just move 'em there just in case
 
 
2 hours later…
3:23 PM
Am I the only one seeing ludicrous upvotes for common dupes? There was a +9 answer for "memoizing fibonacci" yesterday and just now a +3 answer for "sort by length and value".
 
3:56 PM
hello guys, I have python script running in vscode
but after 2-3 days of continuously running, it gets editor crashed or something
any ideas what I can do for this?
 
I'd start by not running scripts through editors
 
Any hints that you are running out of memory?
 
 
2 hours later…
6:20 PM
that's a strong hint
 
 
1 hour later…
7:37 PM
Hello, I do have a case which i looking for some advise with it. Actually i do have around 520 CSV files within S3 Bucket. each file size is around 1 GB so a total of 520 GIB. What am looking to do is to combine all those files into a single CSV file in order to run Pandas operation over it.
I tried different ways such as read in chunks but that's still count as heavy operation. I'm open to do this operation on cloud computing, How about Google Colab + Google Deep learning VM? is there someone tried that ?
 
Doesn't pandas require all that data to be loaded into RAM? I don't think it supports any kind of streaming data processing?
 
also i read some article about vaex
 
"But first, it’s worth considering not using pandas. " thats relevant advice from that post there.
 
 
2 hours later…
9:20 PM
@αԋɱҽԃαмєяιcαη is there a reason this can't be pushed into SQL?
S3 is Amazon so why are you jumping to Google in the pipeline?
 
google deep learning, mind you :P
 
@roganjosh Am not sure about SQL, but i will discuss that with the client.
 
@AndrasDeak--СлаваУкраїні I imagine that there is an equivalent. They are direct competitors with comparable ecosystems
 
I didn't realise google used aws internally
 
@roganjosh I'm open to any solution, either within Amazon or Google.
 
9:25 PM
There is just no reasonable way to get 520GB of data into memory in a pandas dataframe. That's giving me palpitations
Because that's the disk space, so before you even load it into RAM
 
technically "use the cloud" is "download more RAM"
 
To that level would be eye-watering in bills, I'm not sure there are standard instances for anything like that scale in either platform
The database type instances have a hell of a lot of RAM, it seems. But I'm less sure about compute ones that you'd need for pandas
Hmm, perhaps it's me behind the times on what people are processing in memory these days
 
9:53 PM
you get whatever you pay for :P
we've got instances with about 40gb of ram just so we can churn through some ridiculous amounts of data without having to worry about memory
im sure we'll end up exhausting even this soon though
 
10:32 PM
I have one at 128GB of RAM and I'm getting complaints about that, but for a 500GB data source in pandas... that's surely in the terabytes
 
10:45 PM
i see.
so i think we all agree in order to Analyze such big files into Pandas, then it's really need a serious large RAM
i seen an article using vaex which convert the csv file into hdf5 and then analyze it. vaex.io/blog/…
 
11:24 PM
I was suggesting pushing it into SQL e.g. redshift since it's already data in AWS
What is it that you need from pandas, specifically?
 

« first day (4271 days earlier)      last day (660 days later) »