« first day (3982 days earlier)      last day (981 days later) » 

1:08 AM
@roganjosh I guess you could also get an invite if you search something like "array index java" or related, I did manage to get an invite that way around 2 years back
 
@AndrasDeak Yeap! that's what I did
 
 
4 hours later…
5:26 AM
Is there a python autogenerater code like copilot?
 
7:11 AM
Hello, I am looking for download plenty of .csv files in a website as there is no "download all" option and i'm looking to have a script download all of these automatically without having to download each file 1 by 1. I only use google chrome and hope for simple methods. My friend suggested selenium or puppeteer but i'm not sure which I should use
 
Try requests first and only consider selenium if you really need it
 
I was actually looking at chrome dev tools to use the requests module initially but was suggested by my friend for my use it was better to look into selenium/puppeteer
My issue is mainly that I don't know what parameters to send to t he request URL to get the report I need since it doesn't look so clear
 
Then selenium is definitely easier, yeah
 
When I click the "download" button I get a response with the ID number of the report but I can't seem to see how to make a specific request. The request URL is the same for all reports, but have very long query strings with random numbers and letters
Yeah will probably take a look at selenium first then since it seems more organized than puppeteer to begin with
 
7:57 AM
Don't forget your proxy and captcha solvers
 
No need for those as there is a nice blue download button at the web page
 
Have you considered contacting the website owners?
 
The request i'm making now is my financial reports there is an actual API but the values there are not aligned to the actual data for some reason
So I am downloading the .csv files which is more accurate normally i'd download them manually but it's a drag
 
8:12 AM
@Pherdindy There's a good chance the service provider isn't aware of that discrepancy. Please consider to let them know, especially if it's about direct monetary value.
 
Yeah i'll actually check back the API again it's been 2 years since i've tried it
Maybe they fixed it already if not i'll send an email
 
Does that not feel great? Not having to fight the service provider?
 
Definitely preferred just hope there was more options
 
 
1 hour later…
 
1 hour later…
10:45 AM
Howdy... just wanted to vent my anger at pandas. Horrible library. Thanks :P
 
Suppose I have two objects a b, where b is a modified version of a. I'd like to transform b into a, such that Python thinks b is simply a mutated reference to a. How can I achieve this?
 
@duhaime you don't.
 
What is a "mutated reference"?
 
Does python use the id attribute to assess a is b?
 
Roughly speaking, yes.
 
10:57 AM
like a = {}; a[1] = True
Can I change that id attr?
So more concretely I'm going through leetcode problems and want to solve this puzzle with a numpy one liner, but the one liner returns a new object. I want to trick leetcode into thinking my new object is a mutated copy of the old object: leetcode.com/problems/rotate-image
 
There must be a way!
 
In CPython, id is the memory location.
 
can one spoof the memory location on the object?
 
That is exactly the kind of question that should let you hurry back to the drawing board.
For example, just change the original object to have the content of the second.
For lists, that would be original[:] = new_object.
Decently sure numpy arrays also allow copying content.
 
11:12 AM
just tried [:] in my one liner and it worked
 
11:48 AM
I know what they want me to type--I'm just trying to get there faster.

One can't use a new object for this problem. It's all good I'll just type out what they expect.
 
12:27 PM
@MisterMiyagi yes
@duhaime if you showed your code it would be easier to help
 
12:45 PM
Debating whether I should mention setattr as a way of mutating an object without having to use an assignment statement...
 
this is about containers, so not setattr, but yeah
There's probably an answer to the X part
 
Yes I agree with both your sentences
 
My hunch is that there would even be dunderfree answers, should we know the question
 
I considered writing "Yes I agree. Yes I agree." since that's fewer characters than "Yes I agree with both your sentences" but I'm not sure it would be clear that I was employing a queue-based conversational model where the latest and second latest messages were the target of my agreements. More likely the reader would assume that I was agreeing twice to one statement.
 
Add "Respectively." Might be more bytes though.
 
12:52 PM
I've already ruined my efficiency for the day by spending a paragraph explaining something that didn't need explaining. It's all a wash, I'll start fresh tomorrow
 
Might be a good day to write some documentation?
 
@Kevin Honorary mention to ctypes?
 
Certainly. I salute any soul brave enough to use ctypes for any purpose
One time I tried using ctypes to mutate a tuple. Python didn't like that.
 
raise WhatDidYouDoError()
 
Specifically I wanted to create a self-referential tuple, so the_tuple is the_tuple[0]. I think I blew the stack trying to print the result, because the usual "check for reference loops and gracefully output a '...'" code path didn't run
 
1:03 PM
The question is linked above--one just needs to solve this problem: https://leetcode.com/problems/rotate-image/

I wanted to use a numpy one liner but it returned a new object. Anyway there are a million ways to solve the problem and I gave them the vanilla they wanted
 
Of course, ordinary tuples can have "..." somewhere in their string representation, and they don't crash. For example tup = ([],); tup[0].append(tup); print(tup). But perhaps Python had cached some fact about the tuple, like "this doesn't contain any mutable objects", that became invalid when I magically turned the first element into something else
My other theory is that it was related to reference counts
 
Hi there, does anyone here have experience with 1d CNNs?
I'm trying to understand how to use a method in a ML library on my own data.
 
My experience begins and ends at "knows what CNN stands for", alas
 
1d is the simplest case, what are you trying to do?
 
@duhaime Classify a time series
I'm using the tsai library
 
1:07 PM
Can you show your code and a little snippet of your data/inputs?
 
Sure
 
Convolution is when you take two lines, and employ Dark Mathematics to multiply them together. You will need 7 to 13 white candles and 1 ml of blood. Vegans may use soy sauce as substitute.
It can tell if you're not a True Vegan, so don't try to get tricky
 
I have a working ANN based on tsai: colab.research.google.com/drive/…
However, I would like to use a 1D CNN such as FCN, as explained in the tsai documentation: timeseriesai.github.io/tsai/models.FCN.html
@duhaime The documentation states one must write FCN(c_in, c_out, layers=[128, 256, 128], kss=[7, 5, 3]) to employ a FCN, but I have no idea how to apply this to my own data.
 
from tsai.all import * is a really unfortunate decision they made
 
@Kevin By the way, thank you for helping me debug that 406 Error Code -- I figured it out in the end (you can check my solution under my question)
 
1:11 PM
I'm guessing FCN == "fully connected network" and c_in, c_out are meant to be your X, y?
what is kss?
 
@duhaime Indeed, FCN is a fully connected network. I'm pretty new to this field, so I'm also learning. kss is not explained in the documentation.
 
@rb3652 I'm glad to hear you got it sorted out :-) I do wish I could have given a practical solution involving my user agent string idea, but ultimately I was flummoxed by tensorflow's interface
 
I'm sure kss is explained in one of the cited papers raw.githubusercontent.com/timeseriesAI/tsai/…
 
@Kevin No problem, I appreciate you taking the time to help me out (credited you in my answer)
 
👍
 
1:14 PM
@duhaime OK, let me check it out.
 
So before you train the model, is it the case that the X, y you get from your df2xy call represent some time series (X) and some labels for that time series (y)?
 
Yes, so a bit of context to clarify:
Just for testing purposes, I have a small training dataset.
It has 60 instances (20 for each of the 3 classes) and 40 data values per instance
@duhaime So for example,
I take 40 y-values from the above graph (for t = 1,2, 3 ...)
And I feed the data points for 60 such graphs (20 Linear, 20 Quadratic, 20 Cubic)
 
Hmm I don't follow
 
@duhaime I'm new to the library, but it states that df2xy simply turns a pandas dataset into a numpy array.
 
You have 60 curves / different plots or lines?
yes that's straightforward
 
1:19 PM
@duhaime Yes, exactly, maybe a picture will help:
 
Oh, you have 20 linear plots, 20 quadratic, 20 cubic
I see
You could solve those without deep learning but it's all good
so you want to predict which line/curve is which type?
 
@duhaime Yes, exactly. It's a multi-class classification problem.
I've already solved it using tsai's default ANN (75% accuracy after 10 epochs):
 
I hear you, so yes, your X, y should be your curves (X) and their labels (y)
and those are what you'll feed to that FCN model
I'd read the cited literature to learn what those hyperparams are, and what makes sense for model architecture (in terms of number of layers and sizes)
 
Just a note that quantizing your curves to just 60 observations will be losing lots of info--if you were to sample say 1000 points from each curve I bet your models would do better instantly
 
1:23 PM
@duhaime alright then, keep your secrets
 
@duhaime Ah yes! I actually had 500 sampling points per graph initially, but I just wanted to test out this library using a smaller data set
In fact, I had 30,000 graphs previously, but now I made it 60 just for testing purposes!
Now I will try implementing the FCN on my data using the method listed on the documentation.
Let me see if it works.
 
If you are trying to predict when to buy dogecoin, the time is now
 
@Kevin The recursion escape is something that must be explicitly used in a type's repr. I doubt anyone did that for tuple.
 
I vaguely remember looking it up to see whether anyone did, and the answer was, "it's complicated". But that's true of anything written in C, dohohoho
github.com/python/cpython/blob/main/Objects/tupleobject.c#L308 calls Py_ReprEnter, which is effectively the watchdog for infinite repr recursion
I'm not sure what it means by "this should only be possible within a type". My example from earlier, tup = ([],); tup[0].append(tup); print(tup), doesn't call type(), or define any custom classes. Maybe type() is getting called behind the scenes, but it's too many abstraction layers deep for me to draw any conclusions about its behavior
Perhaps my example isn't relevant here because it's not triggering exactly the code path on line 314. An ellipsis string is still getting allocated, but it's probably coming from listobject.c.
 
This tsai library has an unusual architecture, but after your df2xy call you can use:
tfms = [None, TSClassification()]
batch_tfms = TSStandardize()
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=accuracy, cbs=ShowGraph())
learn.fit_one_cycle(100, 3e-4)
One wishes they didn't import everything with that asterisk--it makes it harder to figure out where in the source code to look to find things
 
1:39 PM
@Kevin No idea what the "only possible within a type" means. But it's indeed the tuple that adds the … if it's the loop's root.
 
@duhaime Interesting. Let me take a look.
Let me try that in Colab.
 
I think it's saying that type() can create a self-referential tuple during the ordinary process of constructing a valid user-defined class. Maybe if you do something silly with the MRO or suchlike.
I would be very interested in triggering this behavior.
Darn, typeobject.c never calls PyTuple_SetItem. No easy win for me.
 
@duhaime Wow, this is fascinating
How did you figure that out?
 
"it's indeed the tuple that adds the … if it's the loop's root." Oh, you're right. I misread my own output and thought it was printing ([...],).
 
Same here.
 
1:48 PM
Half-joking justification: ([(...)],) and ([...],) are semantically identical because the inner paren isn't a tuple unless it has a trailing comma :-P
 
I just took a look at the docs! :)
If you like parentheses boy you'll love this one: leetcode.com/problems/longest-valid-parentheses
 
Apparently, tuple expects self recursion after the Great SVN Merge of 2007.
> tuple.__repr__ did not consider a reference loop as it is not possible from
Python code; but it is possible from C. object.__str__ had the issue of not
expecting a type to doing something within it's tp_str implementation that
could trigger an infinite recursion, but it could in C code.. Both found
thanks to BaseException and how it handles its repr.
FWIW, I still have no idea what that means exactly.
 
Well, I'm happy to get some official word on the subject, even if neither of us know what it means exactly :-)
"A blessing from the lord!"
"Yes, but what is it?"
"Hush, do you want to get our blessing privileges revoked?"
BaseException's repr looks pretty unexciting to me, don't know why it got singled out in the quote github.com/python/cpython/blob/main/Objects/exceptions.c#L127
Let's see if the issue is still in the bug tracker...
Python SEGFAULT on tuple.__repr__ and str(). Sounds relevant to my interests.
class X(BaseException):
    def __init__(self):
        super(X, self).__init__(self)
print(X())
#RecursionError: maximum recursion depth exceeded while getting the str of an object
Better than a seg fault I suppose.
 
2:11 PM
Ok, I basically get the Exception issue. If an exception's init is called, and self gets passed as the first non-self argument, then the_exception is the_exception.args[0]. This is bad, because exceptions create their string representation by getting the string representation of their first arg and adding "Exception()" around it.
There are no self-referential tuples in this scenario, but presumably the mechanism is similar
 
I guess the point is "C code can have self-referential repr" rather than "tuples can have self-referential repr".
 
Yeah
 
2:37 PM
@duhaime I keep thinking I have an O(N) solution and then I find yet another corner case that needs special handling
I've put duct tape over all the cracks in my dam, but I am ill at ease
 
3:18 PM
Hi everyone, can someone help me in optimizing this piece of code?
def convert_to_json(dataframe):
    '''
    Converts a dataframe object to a json object
    appropriate for analysis
    '''
    json_obj = dataframe.to_json(orient='records')
    processed_json = json.loads(json_obj)
    print(len(processed_json))
    return processed_json
This is just to convert a dataframe to a json structure needed for analysis. However with this I am running across MemoryError even with 0.4 M rows. Is there a better way I can do this?
 
I don't have any concrete advice, but keep in mind that a "json structure" is only an informal name for the kinds of objects that json can easily represent. If all of your google searches include "json", then you may be missing out on more general solutions.
While I continue to not answer the actual question, I suggest examining the code that wants a json structure as input. Perhaps you can modify it so it also accepts a regular dataframe. Then no conversion will be necessary.
 
I don't quite get what this is supposed to do. JSON is a notation – i.e. it's a way to write out data as text. If you just convert a dataframe to JSON and then immediately load it again, you could have skipped the step.
Most likely, you just need something like pandas.DataFrame.to_dict
 
"How do I convert this into a data structure composed from native Python types?" just doesn't roll off the tongue :-P
 
That probably means it's the right question to ask. :P
Let's be honest here: JSON is just quicker, easier, more seductive.
 
3:35 PM
Developers can fall to the dark side a little, as a treat
 
My dam is made entirely out of duct tape :/
 
I'm 97% sure a bulletproof O(N) solution is possible, and 80% sure that the solution can be written in a non-ugly way
(this is in regards to the balanced parentheses substring leetcode problem from earlier, if that's not clear)
 
3:54 PM
I'm currently using classes derived from tuple to make immutable instances, but the indexing/iteration/sequence leftovers are a bit distracting. Is there some builtin immutable type that can contain an arbitrary value but has less surface area than tuple?
 
Hmm, maybe... Darn, Exception args are mutable.
 
df.to_json will account for things like multi-indexed rows etc. I can also say that after a full day of fighting with from_json maybe 2 years back that I won't be touching either method again. Unless there's cake involved. I might consider it then.
 
4:32 PM
@MisterMiyagi where's the difficulty in making your own class from scratch? Making it truly and utterly immutable?
 
4:45 PM
@Kevin "this should only be possible within a type" I didn't look at the source, but my guess is that's a comment from Ancient Times, and it refers to types defined in C (like the built-in types), rather than classes defined in Python.
 
cbg
@RaphX what are you even trying to do there?
 
@PM2Ring git blame points to github.com/python/cpython/commit/…, 2010, predating 3.2
 
@JonClements yeah I guess the loading was unnecessary, well I am mainly trying to get the data in the format of embedded dicts within a list. I need this format to later flatten it for analysis.
@MisterMiyagi yes , I agree that the loading part was unnecessary. Maybe removing that will cause no memory error issue at all.
 
I daren't ask why a dict is better for "analysis" than a dataframe
 
ditto... but I'm a glutton for punishment so @RaphX - what's your original format? Have you ended up with a dataframe that contains dicts/lists or something and that's why you need to expand it out?
 
4:58 PM
yes its a nested dict data format that I get from BigQuery @JonClements
 
@AndrasDeak Thanks. I was expecting it to be much earlier, like from before new-style classes. OTOH, I guess there's no major reason for devs to stop using that terminology.
 
@RaphX so... seems like you should be making a BigQuery differently then
 
@PM2Ring I'm on mobile so didn't look further back. That may have just been a cosmetic change.
 
There is an UNNEST function available within BigQuery but it doesn't allow the results to be downloaded in that format @JonClements
I mean like I can only see the flattened results in BigQuery itself using that
 
what are you trying to download/process that blows up at ~400k rows?
(on any recently modern system that is - I mean - one would really really struggle to get a computer with less than 8gb ram these days)
 
5:05 PM
@JonClements yeah its a 2 GB linux machine
 
okay... so the 400k is probably pushing it in the first place then (memory wise at least)
how much disk does it have?
from you saying 2GB - I 'm guessing you mean 2Gb and you're actually on some kinda cloud server or something?
 
@JonClements yeah on AWS, its a t2.small linux instance
 
these details matter - you should really mention them - without having them teased out of you :p
 
@JonClements sorry, I will definitely keep it in mind for next time, this was not the typical MCVE I have been sharing for my past questions
 
@RaphX no worries... MCVE's always help regardless...
I'm trying to remember if the t2.small has some access to ES or of it's just all in memory period server thingy
(AWS sometimes has rather dubious setting to default disk and network settings)
yeah... just reading up a bit... I 'd forgotten what AWS names all their stuff now... but yeah, a t2.small is something you're not going to have fun with doing what it sounds like you're trying to do
 
5:22 PM
@JonClements so do I have to upgrade then?
 
you're going to have to some point for something
if you login to your server and do df -h? what does it say?
 
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  121M  1.8G   7% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/xvda1       30G   26G  4.6G  85% /
tmpfs           379M  260K  379M   1% /run/user/1000
 
5:48 PM
@RaphX yeah... either 1) you need to optimise or 2) just shell out some $'s on a medium
(1) is ideal as you can carry on doing what you're doing but need to work out the better way first... (2) works because you can get the stuff done, see the results, and get back to (1) if you need to
 
@AndrasDeak The truly immutable part. But I'm probably overdesigning…
 
@RaphX anyway... what does it say if you put free -m in ?
gotta run for a bit foodies! :p
 
6:48 PM
@MisterMiyagi do you have to plan for maliciously ignorant users? Scratch that, you always do.
 
7:03 PM
@AndrasDeak Overloading __setitem__ with rm -rf /* sure could be educational…
 
Can't one seal __set*__ at the end of __init__?
 
it's still possible to use object.__setattr__ or super(…).__setattr__.
 
ah... silly python
so you just have to build a C extension with a type system that doesn't inherit from object
(*probably not possible with a C extension)
(**previous message mostly not there to nerd snipe someone)
And there's also the fact that dunders are looked up on the class, so I don't even find the straightforward part of the problem straightforward.
Yeah, I can only pull that off with object.__setattr__ trickery in __init__
 
I currently tend towards just not having any sympathy for people that nuke their own foot.
 
yup
just put a post-it note on the class that says "this is immutable, m'kay?"
 
7:17 PM
A hand-drawn class diagram annotating __setitem__ as "here be nasal demons" would be overdoing it, I take it?
Hm. PEP 666: The -> demon annotation for undefined behaviour.
 
 
2 hours later…
9:26 PM
Cbg. How can I check the shape of a MapDataSet (tensorflow.org/jvm/api_docs/java/org/tensorflow/framework/data/…)?
 
9:39 PM
I find the TF code so difficult to read :/ I set my cursor where I would expect a new class definition to start, but after 500 lines of code scrolling, I can't convince myself that I've not missed a new class definition
I'm tempted to see what ._get_flat_shapes() returns. You're obviously not supposed to do that but I'm not straining my brain on this one
 
9:52 PM
I wonder how easily their developers can switch between 2 and 4 spaces of indentation? Maybe I just need to subject myself to it for a few hours and I could switch pretty easily. In theory it sounds easy, but in practice I don't find it that way at all
 
10:09 PM
Yes, it's a bit hard
I've tried that function (._get_flat_shapes()).
Result:
AttributeError: 'MapDataset' object has no attribute '_get_flat_shapes'
I've figured it out
At least in my case, just printing the MapDataset itself returns the shape(s)
Thanks anyway, @roganjosh
 
11:13 PM
@Marco I can't find any python on that link, only java. Where's the python docs?
 
I realized this a short time ago, I was going to comment about it here, I didn't even realize it, sorry. I tried to find the Python documentation but I couldn't find it. But I've already solved the problem, thanks.
 
11:41 PM
Hmm. Now I'm confused, because I was looking through this in the source
Then started tracing the inheritance back until I got bored of my mental parser trying to read the code. The "JVM" in the original link is suspicious, but it does appear to exist in TF in Python form
 
@roganjosh Whoa
Yes, it has a Python form and its Docs is somewhere
 
That's where my ._get_flat_shapes() came from... by tracing it back and then getting bored before I could trace anything down. They literally crushed all the whitespace out they could find so it's so dense to read
 
:(
The Google's men
Thanks for the efforts
 

« first day (3982 days earlier)      last day (981 days later) »