« first day (3980 days earlier)      last day (958 days later) » 

3:26 AM
cbg
 
 
1 hour later…
4:27 AM
finding it surprisingly hard to search for how to ignore abc.ABC complaining about a derived class not implementing a specific function. I don't even care to do it, just curious about how you even would do it.
I can always just read the abc source to see what the metaclass is doing to enforce it (presumably the decorator just adds it to a list and it checks those methods exist on the subclass?)
 
 
2 hours later…
6:52 AM
@alkasm The easiest approach is to add a stub function. That'll satisfy the ABC check.
 
7:21 AM
The equivalent of ramen noodle fixups. I love it.
 
8:07 AM
psa in case it helps anyone: pandas to_json function apparently has a memory leak. i don't know a lot of details im afraid
 
8:26 AM
wat
 
it's a small leak, but it's taken out our 60 gig vm. apparently if you run to_json enough times the memory usage slowly keeps creeping up.
found a bit more info here
so uh yeah, be aware i suppose.
 
8:46 AM
@ParitoshSingh seems it was fixed for 0.25.0. So if there's a regression you should let them know
 
good shout. it exists on 1.2.4 still.
though we aren't on the latest pandas version, looks like 1.3 is out
 
 
1 hour later…
10:18 AM
cbg guys, cant figure out how to get T1 = 95 and T2 = 35 from the following
import pandas as pd

seq = [("id", "item_id", "quantity", "volume", "division"), ("T1", "A1", 5, 10, 72), ("T1", "A1", 10, 15, 72), ("T1", "A2", 20, 30, 74), ("T1", "A3", 5, 40, 72), ("T2", "A1", 35, 35, 72)]

df = pd.DataFrame(seq[1:], columns=seq[0])
print(df.groupby(['id', 'division']).sum())
that is what I have tried
I need to add the volumes again, but I cant figure it out
 
Well, for starters, you should probably sum up volume rather than division
 
ok, will try that way then
 
I think df.groupby('id')['volume'].sum() does what you want?
 
I guess it does, didnt know you could index a group by that way
thanks!
 
 
1 hour later…
11:50 AM
cbg
 
 
1 hour later…
1:03 PM
could any one solve please this thank in advance
 
@JackHammer Hello. Please see the room rules particularly in regards to waiting 48 hours before bringing questions from main here
 
@JackHammer Please don't advertise your question here until it's 60 times as old, as per our rules
 
okay
thank you for informing me @roganjosh
 
1:32 PM
Content of version.json file is :
{"version": 18.6}
with open(software_ver_json_file) as f:
    data = json.load(f)
The above python code results in
 JSONDecodeError("Expecting value", s, err.value) from None
intermittently!
What's wrong here?
 
Can you please provide the formatted text of the error, not the code of the error?
 
Sure
 
There's likely something else in the file – the string itself is fine: json.loads('{"version": 18.6}')
 
  File "/opt/psa_integration/config.py", line 127, in <module>
    data = json.load(f)
  File "/usr/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
@MisterMiyagi That file has only so much!
 
Show us the output of a print(repr(f.read())) please (do that before the json.load)
 
1:43 PM
o/p:
'{"version": 18.6}'
 
Well, now I'm stumped.
 
Hey, if multiple workers try to read the same file simultaniously, can we expect this situation?
 
If they all try to read from the same file object, then yes. But if each worker has its own open(), then no
 
I am not sure they doing it. Let me check through it.
@Aran-Fey Thanks :)
 
Can there be race conditions causing the file to be intermittently empty?
Is the file "just there"?
 
2:18 PM
A bit of a vague question, but I'm working with plots for the first time (plotting the price of an item over time) and while it's easy enough to create a basic plot, it feels like there are a bajillion small improvements you can make to make the thing more useful. For example, letting you hover over the graph to see the exact price for a specific day, or drawing increasingly thick marker lines every week/month/year, or zooming/panning, etc.
It feels like I could invest weeks into playing around with all this stuff, so... does anyone have any recommendations what I should focus on? Are there any features every plot should have?
Turns out that just having a squiggly blue line on white background isn't particularly useful and/or user friendly
 
It depends on a lot of things I guess; who is the plot for, and how are you presenting it? (i.e. how are you going to pass on that interactivity?)
The stuff that I would want to see in a graph is often very different to the things I'd create for customers because I care about wildly different things (usually)
The one thing that you've mentioned that would probably confuse me, though, is an increasing thickness of the plot line based on time. I would probably use some vertical lines to demarcate the different time periods or (probably easier to implement) a change of colour.
 
It's a personal tool only for me. The plot is embedded in a Gtk gui, if that's what you're asking. If I had to say what I'm using the plot for, it's essentially supposed to let me figure out 1) the item's current (say, within the last week) price, 2) the item's minimum and maximum price within a longer time period, say the last year or so
@roganjosh Right, I did mean the vertical lines, not the plot line itself
The goal is essentially to buy cheap, hold on to it for a while, and sell high. So I need to identify items with a large difference between min and max, as well as the current trend
 
If this was market analysis you might look for Bollinger bands but I think you need something a bit different. The second example here might be something to repurpose for your visualisation
 
2:39 PM
I'm still processing the wikipedia article for bollinger bands, but regarding the visualization, that's a good idea. I was planning to plot the average price, but with that I can also visualize the minimum and maximum for each day
I wasn't even thinking of high-level stuff like analysis or prediction though. Right now, when I look at my squiggly blue line on white background, I have a hard time even figuring out what the current price is because that data point is on the right side and the Y axis labels are 800 pixels to the left. So I need to draw some horizontal lines at, say, 50$, 100$, etc. That's the kind of problem I'm having
 
The link should cover most of that, I would think? "tick" is a common term between different plotting libraries (I'm not sure which you're using. I don't know gtk so I don't know whether "embed" extends to supporting other plotting libraries), so you'd want to set the ytick to whatever unit helps you understand the data
 
2:55 PM
I'm using matplotlib, but only because it was the first thing that came up when I looked for Gtk integration. I don't mind switching to another framework if there are better alternatives
 
I generally use plotlyjs because I'm usually putting plots into online dashboards, but plotly itself seems easier to use out-of-the-box by putting in some default interactivity and a cleaner formatting style by default. As you're seeing, matplotlib can be pretty involved to configure from the outset (so perhaps better-suited when you know exactly what you want). seaborn is another option.
 
Alright, I'll check those out
Never would've thought the amount of basic customization you can (or rather, should) do would be so overwhelming
 
You can agonise over plots, especially ones for scientific publications. You get (kinda) one shot to get your point across, so there's a lot of minute details you might need to tweak
 
3:26 PM
hello guys
so I am going to build a rest api using DRF, what installed_apps am I allowed to remove?
 
 
2 hours later…
5:23 PM
hey
 
5:44 PM
@Aran-Fey ax.grid(True) is something I have in muscle memory
that goes a long way in making your data clear
Seaborn is for the most part a thin wrapper over matplotlib that makes your plots look fancier by default. Not sure about the data visualization boost it gives.
But I'm not a seaborn user
 
6:16 PM
Hmm, looks like I missed out on a lot of discussion on that Keras/tensorflow problem that came up the other day
 
That's a good thing, right? Right?
 
very
 
I haven't read the whole transcript, so maybe I'm retreading established ground, but I was able to make the 406 error go away on my machine by providing a user agent string identical to firefox's
 
Hint: it wasn't a Keras/tensorflow problem.
 
As I suspected, since I was able to replicate it with just urllib
 
6:17 PM
@Kevin they said "OK let me try" and they were never heard from again
But something tells me they will be back. [Ominous music starts playing]
 
Hehe. He did that a couple times too while I was here. "Try checking the response body" -- "ok, I'll go do that now" -- [some time later] "so uh, how do I check a response body?"
 
Weird that curl works without a user agent, though. But this would only concern me if I cared about the whole issue.
 
Wild guess: ModSecurity accepts requests from human-looking user agents, and curl-looking user agents, and nothing else
The more often a tool is used for legitimate purposes by system administrators, the more likely that tool will be quietly white listed so it isn't as aggravating to use
 
Only 5 more minutes until my program finishes its 2000 HTTP requests and I can try Andras's grid() recommendation *continues to twiddle thumbs*
I really need to offload these into a background task
 
patience-driven development
 
6:30 PM
...seems I also need to fix a bug or two
 
Matplotlib usually has pretty good tick placement by default. If you want something hands-on, you can use Locators to define where ticks are put automatically, or even define them manually. And Formatters can let you customize how the tick labels (500, 1000 etc.) are actually generated. But I'm almost always happy with out-of-the-box ticks and labels.
 
It's troublesome when my application has to do a ton of I/O before I can test out some experimental UI feature. Half the time, it doesn't occur to me that I could generate my own test data instead of fetching it over the network. When it does occur to me, it's not worth the effort because making a test data generator takes twice as long as doing the regular I/O
 
Fetch the real data once and use that as mock input?
 
I passed in a dict of {datetime: float} and it does nicely display the dates as text... except they all overlap and you can't read anything
 
Passed in a dict? To what? Sounds dubious.
 
6:33 PM
@AndrasDeak Yeah, I've done that a couple times when my data is under 100MB. Most common failure mode: my real data saving code is buggy because I forgot the correct order of arguments for json.dump
 
(but yeah, it's datetime-aware, and datetime ticks have some silly defaults though... like uneven tick placement due to ticks falling on the 1st of each month or whatever)
 
@AndrasDeak ah, you're right, I store the data as a dict, but then pass keys() and values() to matplotlib
 
phew
I've had to do datetimes exactly once so far, and in 100% of those cases I had to use a WeekdayLocator because the default was crap.
 
Dang, I was hoping I wouldn't have to deal with those. Had a short look at the documentation and had a hard time wrapping my head around it
 
import matplotlib.dates as dates

ax.xaxis.set_major_locator(dates.WeekdayLocator(byweekday=dates.SU))
that places a tick on every SU(nday)
 
6:41 PM
I wonder whether Matplotlib's crap default for datetime ticks is because of:
- not enough demand/interested volunteers
- choosing sensible default behavior is actually really hard, because all things involving time are hard
- their administration rejects all PRs about the topic because they like it the way it is
 
@AndrasDeak Oh, that's all it does? I feared it would have some effect on how the data is sampled/rendered (like, it would only render one data point per week)
 
@Kevin Probably because numpy/pandas were all over the place. I've linked this before but - this
 
Just looking at this diagram tells me there's something fundamentally wrong with all this time stuff. — demented hedgehog Oct 12 '16 at 23:57
True for both computational representations of time, and just the concept of time in general
 
@Aran-Fey nope, locators only locate ticks.
@Kevin middle one
 
"This graph covers a time range that includes a leap day, so for proper visualization you'll need to print it out onto a mobius strip"
 
6:47 PM
It really is about unequal tick distances due to month starts. And I guess there might be similar issues on an intraday scale.
*inter-day
this question I read a while back has a nice example of uneven default ticks
every week... but it restarts on the 1st
 
7:10 PM
How do you mess up so bad? By the way, the Y axis is upside down :|
 
@Aran-Fey they are strings, thus plotted as categories, aren't they?
 
Oh, you're right. I'm loading numbers from a csv file without converting them. This is strange, where are all these bugs coming from all of a sudden?
 
ax.tick_params('x', labelrotation=45)  # to help with your xticklabels
the rotated labels will be "below" the ticks, so if you want them to join the tick at their end you'll also have to set the alignment...
 
Do you know if this gets reset by an axes.clear(), like the grid() does?
 
nope, never used those
I'd expect them to be, for what it's worth
But you can play with these on a small interactive example without the 2000 requests and GUI integration :P
 
7:23 PM
Well, I guess it doesn't really matter. I would've preferred to the configuration only once, but it's not like re-doing it every time is a big deal
 
>>> import matplotlib.pyplot as plt
>>> plt.ion()
<matplotlib.pyplot._IonContext object at 0x7fd49ef36970>
>>> fig, ax = plt.subplots()
now you have auto-opening interactive plots in your REPL
 
Oh it's fine, the 2000 requests only happen once per... uh... week, I think
 
@Aran-Fey is this an XY problem? If you only want to delete the actual plots, do that instead.
 
Yup, definitely an XY problem if that's possible
 
lines = ax.plot(...)
for line in lines:
    line.remove()
 
jrh
7:28 PM
Does anybody know offhand if Spark renamed "master" as well as "slave"? I've got a book that seems refer to both "Cluster Manager" and "master" as the thing that handles scheduling jobs on the worker (AKA slave) nodes and I'm starting to think not all the figures in the book got updated. Or is it that Cluster Manager is a process running on the master node?
 
@AndrasDeak Neat, thanks. Am I stupid or is the remove method not documented in the Line2D docs?
 
Going by the Wikipedia description of "cluster manager", it sounds a bit orthogonal to the concept of master/slave architecture. My guess is that they're different things.
 
@Aran-Fey probably inherited from Artist
 
Ah, right
 
> In some cases the cluster manager is mostly used to dispatch work for the cluster (or cloud) to perform.
This case sounds pretty much like master/slave to me. But there must be other cases, and those cases might have a more distributed hierarchy
 
jrh
7:38 PM
Reading this it looks like they are two distinct processes with different jobs

https://www.informit.com/articles/article.aspx?p=2928186
I think it'd probably qualify as "master/slave" too, but I think they changed some terminology

Also (chat markdown won't render this quote properly?)

> As discussed earlier, the Cluster Manager can be separate from the Master process. This is the case when running Spark on Mesos or YARN. In the case of Spark running in Standalone mode, the Master process also performs the functions of the Cluster Manager. Effectively, it acts as its own Cluster Manager.
 
I can never get quotes to look right in chat unless I quote the whole message. One of the more long-lived quirks of the system.
 
jrh
@Kevin To the sandbox!
 
[batman-themed scene transition]
 
Yep, looks about as confusing as it usually does
 
jrh
7:45 PM
Seems like it only likes > on the first line, and if it's there, the whole thing is a quote. Looks like maybe Jeff baked this parser at home or something.
 
Plausible
 
@jrh I think it was balpha, not Jeff, but it's very homebrew, yes
 
jrh
Apparently, once it starts quoting, it can't be persuaded to stop
 
that is my experience, yes
but multiline code is also all or nothing, so this is hardly surprising
Just wait until you see how directed replies work with quotes
 
jrh
I like the time limit on edits too, it's like a fast paced fun quiz "you have 10 seconds to try every single parser wart workaround you can think of... go!"
 
7:51 PM
well you have more shots in the sandbox
Or you could run for moderator and have ultimate editing privileges.
 
New moderator elected in landslide victory, based on platform of "I just want power, honestly"
"Oh, you thought I was going to take matters into my own hands and clean up the dirty streets? No, I just wanted an unlimited editing window for chat. Sorry about the confusion"
 
Cabbage
It looks like the accept unpinning experiment was a huge success. meta.stackoverflow.com/q/411352/4014959
 
8:07 PM
wooot
 
And now they're asking us how it should be rolled out to the rest of the network. meta.stackexchange.com/q/369568/334566
@Kevin My guess is it's the 2nd option: choosing good defaults is hard when you have so many choices, and so many different ways that people will want to format / lay out time & date stuff.
 
@jrh I don't know about Spark specifically, but I have seen other posts about getting rid of the "master"/"slave" paradigm due to the connotations of the words
 
jrh
@roganjosh yeah, that's part of what confused me, I didn't know if they went all the way and binned both master and slave; it looks like they just got rid of slave
This book is not helping with the terminology problems either, sometimes a figure says "master", sometimes it says "manager", sometimes it shows "master" in a "manager". It sees like they are just two processes though, with two different responsibilities.
 
Like I said, I don't know for Spark specifically (it's been a few years since I was using that) but I wouldn't be surprised to see this terminology being phased out and becoming obsolete in textbooks. Then again they could be different concepts
 
jrh
Yeah, the book is new enough to know about that (2020), and it talks about the renaming, I picked it hoping it would fix the confusion
 
8:19 PM
For things like multiprocessing, for example, the Manager will deal with IPC but I'm not sure whether it would have been considered a Master in any terminology
 
jrh
> Spark Master
> The Spark Master is the process that requests resources in the cluster and makes them available to the Spark Driver. In all deployment modes, the Master negotiates resources or containers with Worker nodes or slave nodes and tracks their status and monitors their progress.
>
> Cluster Manager
> The Cluster Manager is the process responsible for monitoring the Worker nodes and reserving resources on these nodes upon request by the Master. The Master then makes these cluster resources available to the Driver in the form of Executors.
 
Yep, that makes sense to me. Thanks for the clarification
Out of curiosity, what do they call "slave" now? Is it just going to be "node"?
 
jrh
I think the master is what decides which worker node runs the code, and the manager is what spins up the container on the worker node to actually run the code.
@roganjosh slave nodes have been renamed to worker nodes, from what I understand
Not to be confused with executors, which is a process that runs on the worker AKA slave node
 
I have to say that seeing YARN has made me somewhat nostalgic, even though I don't have a hope in hell answering any questions about it now :P Interesting stuff, though
 
jrh
It still strikes me as weird that yarnpkg.com apparently also picked such a non-sequitur name for their project; I don't think that's an acronym though
 
9:17 PM
Hello everyone;
@AndrasDeak And I'm back
 
@rb3652 re-asking your original problem as if we hadn't spent hours helping you is a terrible waste of our time. If necessary, reread where we left off.
 
9:36 PM
Eureka! I've done it!
Thank you, @AndrasDeak and @Kevin -- you both spent a lot of time helping and I truly appreciate it. Thank you for helping me debug the code.
I'm going to answer my own question now. If you're interested, you can take a look.
 
I'll pass, thanks
 
10:30 PM
@rb3652 os.path.join(os.path.dirname('FunctionIdentifier.zip'), 'FunctionIdentifier') sure is a roundabout way of writing 'FunctionIdentifier'
 
10:49 PM
@Aran-Fey Genius, I know.
 

« first day (3980 days earlier)      last day (958 days later) »