« first day (4973 days earlier)      last day (237 days later) » 

04:30
@PM2Ring Lots of concerning discussion going on in there
 
8 hours later…
12:38
@roganjosh Yes, it doesn't fill me with confidence for OpenAI. OTOH, Scott Aaronson doesn't seem too concerned. He posted that link on his blog, but he said in the blog comment thread scottaaronson.blog/?p=8005#comment-1977479
> I think the explanation overwhelmingly favored by Occam’s Razor is that pretty much everyone at OpenAI really believes in the mission of “building AGI that’s beneficial to humanity”—just to different extents, and sometimes with wildly different ways of operationalizing what it means.
He's still working as a contractor at OpenAI for a few more months. He was hired to do stuff related to AI safety. That materialised as a fairly robust method of watermarking ChatGPT text output. OpenAI haven't yet deployed Scott's watermarking protocol, but they haven't outright rejected it either. ;)
13:38
@PM2Ring On the theme of search results that are already bad but likely to get enshittified by AI, look at the diversity of opinion in Google top-10 hits for "Is python OrderedDict obsolete?" (in all Pytyon, not just cPython), a question that was answered in the affirmative by Raymond Hettinger back in 2016. (Other than code reliant on the OrderedDict.popitem, move_to_end methods)
...although I guess popular Reddit posts illustrating some user confusion are legitimately relevant. But in future it may also get added by machine.
13:55
@PM2Ring Wouldn't be surprised if this work as well as those AI detection tools. hit or miss. Namely, "existing text might correlate with their watermarking text", unless of course, it is actually adding invisible bytes, but that can easily be stripped off, so probably not that
Does anyone have a suggestion for an error metric when evaluating forecast accuracy in aggregate? I have 1700 forecasts (combinations of stores and departments) and I can see that MAPE (on a first pass) is garbage. The aggregate error is 5% but each individual forecast is swinging +/-50%
There's a few suggestions in that article and I already had an idea that MAPE wouldn't work, but I'm time-limited so just curious if anyone has a favourite
14:11
@smci It's a bit like our HNQ. Stuff that's popular gets its popularity exaggerated. So it gets harder & harder to find stuff that isn't close to the centre of the bell curve.
Jun 4, 2023 at 11:21, by PM 2Ring
@NordineLotfi The ChatGPT watermark isn't like a visible watermark on an image. It's incorporated into the pseudo-random sequence of tokens.
Scott vaguely describes his watermarking in scottaaronson.blog/?p=6823
So there are no invisible bytes. The watermarking is created by using a secret key with a crypto-grade random number generator to choose tokens. Forging the watermark is equivalent to guessing the crypto key.
The watermarked text is robust to minor editing, but it is disrupted by edits that have a big impact on the token sequences.
@roganjosh Well what's the relative penalty for underforecasting vs overforecasting, and is it linear/exponential/logarithmic/what? That depends on the warehouse's operations and logistics (are unfilled/late orders less bad than a glut? Does that penalty it depend on teh unit volume or nature of the item? (frozen/perishable/seasonal/etc.)). And as to being sensitive to spiky vs reguar sales volumes, hard to comment, maybe it's allowable to use annualized/seasonal averages for some products...
All excellent questions, some of which I asked them myself. The consequence is that they will use each individual forecast to set a shrink target on a per-store-per-department basis and then start hassling the individual managers if they exceed their target
...as was said before, sounds like your customer is using you as an operations consultant not just a data scientist. How you play that is up to you... Do they want to pay you to go research the relative penalties for underforecasting vs overforecasting across a range of product? Perhaps you could make some premium vodka go missing... as a test of course...
On aggregate I can give them a decent global figure of shrink just through the law of large numbers, but the idea they can use any method on a granular level...
@smci unfortunately, the first day I came back from holiday, my boss has gone on holiday for the week and there's a guy assigned to bridge the gap between us so I have to keep going for now. I felt sorry for him this morning while I was explaining how this thing "works". I still have to wait a week before I can tell them I'm done with this
In any case, MAPE has bugged me here. The more I look at it, the more I feel like there should be some dimensionless (?) evaluation of forecasts in aggregate. Everything I can find seems to apply to how a single forecast matches reality over a time period. Maybe it's just stdev on the output accuracy in some form but it doesn't feel satisfying
14:35
Oh that work situation sounds even weirder (this is why I was suggesting not discussing somewhere that AI will index it to you, but anyway). If it was an ongoing project, I would have suggested trying to add penalty coefficients for the things I mentioned (think of how ARIMA generalizes timeseries). But you only have a week so, don't innovate. Next time, negotiate your compensation in vodka, then tweak your model to misestimate vodka demand...
I'll just write a library for it. Input: [forecast_1_error, forecast_2_error, forecast_3_error, forecast_4_error, forecast_5_error]. Output: ['crap', 'crap', 'semi-crap', 'passable', 'crap'].sum()
May 15 at 11:46, by roganjosh
@smci FWIW, all of my messages are posted knowing full-well that they are indexed and open access. Furthermore, I even invite colleagues to join the chat at times but they just never come back. If they chose to, they could go back through my history. So far, it's never happened. In any case, I'm aware of the possibility
@roganjosh No, because some items are heavy, bulky, wet/frozen/perishable and time-critical (like Christmas turkeys or Valentine's Day flowers), whereas household and consumer cyclical stuff isn't, and the penalty for underforecasting those might conceivably be less than overforecasting. I think you're making a mistake trying to overgeneralize wildly different product ranges to one metric, that doesn't exist. But this is surely their next operations research consultant's job, not yours.
@roganjosh I guess you could have sold them on SO for Teams so you could have had a company-private discussion... has anyone here ever done that?
@roganjosh That's the spirit. Don't forget the vodka metric.
And once the managers figure out their penalty depends on how head office estimates sales, they may start gaming sales estimates... are you allowed tell us which products have the worst shrinkage? Small concealable nonperishable stuff with high street resale value per unit volume, like batteries, razors, toiletries, electronics, alcohol?
14:54
@smci as I keep saying, you're way over-thinking this. I have literally just a cash value lost
That's it. Dept 3, probably stocking 500 products, lost £250 today. That's my data
@roganjosh So how does the penalty vary for underforecasting vs overforecasting (shrink?)? Here you're only talking about estimating shrink, not sales demand? If you overestimate shrink on something that didn't get sold, you count it the same as if it was sold.
Because someone goes around and scans items they think are lost. But one day they might catch 30%, the next day they catch 50%
In any case. I don't want to keep taking up more room time with this. I do really appreciate your feedback but if I had any of these other indicators, I'd be using them
@MisterMiyagi I take it you're merely talking about not adding gratuitous levels of design-pattern cruft, not about Jack Diederich-style minimalism (even eliminate methods and data members that a typical SWE would use to implement the thing, trade off runtime performance vs possible reduced readability)?
15:10
@smci I'm all for doing things that have a good reason to be done.
@MisterMiyagi I meant was your comment intended to be general, or specific to whatever usecase inspired your discussion with metatoaster or NordineLotfi? (it wasn't clear and I didn't have time to read further back)
It's a general comment, but should not be read to imply one should "eliminate methods and data members that a typical SWE would use to implement the thing".
Well, depends on typical SWE of course...
If you can't decide whether to create a class, or just use a dict, then use a dict. But if your code would benefit from instance methods, and be more readable as a class, then use a class.
Of course, readability is a bit subjective. To someone coming from Java land, Python code can look a bit ramshackle because everything isn't neatly boxed up into classes.
Similarly, JSON is usually easier to produce & consume than XML. But sometimes the extra structure that XML provides is beneficial, and worth the extra effort. Eg, SVG is a nice XML format. It would be painful as JSON.
@MisterMiyagi because that was my take on Diederich's "Stop Writing Classes", or at least how it could be overzealously applied. And people here keep quoting it.
@PM2Ring equivalently, Python doesn't have to plod through twenty pages of boilerplate class definition to get something done. Python's functions are first-class objects.
15:28
IMHO, Diedrich's point is that using classes doesn't automatically make the code better, for various meanings of "better", eg readable, maintainable, efficient. Don't use explicit OOP just because OOP is supposed to be a good thing. OTOH, do use OOP when it's appropriate to the task.
15:48
@PM2Ring Yes, but he finds "appropriate" hard to define-by-example, and the title is more extreme than the message (it omits the "Stop automatically..."). Conversely, automatically using dicts tends to be a tradeoff of less readability and thus maintainability - except for toy usecases, in which case it's irrelevant.
16:00
Why less readable? If you just need a structure with some named attributes, with a function or two to operate on your structures, it doesn't make much difference in the readability to use a dict vs a class, although admittedly accessing class attributes is slightly less verbose.
But if you need a whole suite of methods to operate on those records, it's silly to avoid using a class, since you're going to end up creating the same thing anyway, with the dict & a bunch of functions.
 
1 hour later…
17:29
@smci Meh, people have a tendency to take any advice too far.
^ agreed
 
2 hours later…
19:43
@PM2Ring right, I remember when we talked about this :) so basically I still think the same thing as back then hmm
uh, today learned you can get some speed up when using numpy by using from keras import ops as np instead of import numpy as np and then have the code using numpy run on a gpu or be compiled to XLA :D
@MisterMiyagi Especially when it's from someone well regarded or popular. See, if I made that talk, not that many would quote it /s

« first day (4973 days earlier)      last day (237 days later) »