« first day (4909 days earlier)      last day (44 days later) » 

12:33 AM
I've ploted that histogram via matplotlib (np.hist()) and instead of the bars all coming in the standard color (blue), they came in color, does anyone know why?
Note: I didn't set any color in np.hist().
They became colorful*
Simple code:
plt.hist(data, bins=10)
axes = plt.gca()
axes.set_xlabel('PERÍODOS')
axes.set_ylabel('CONTAGEM')
axes.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%m'))
plt.gcf().autofmt_xdate()
plt.show()
 
12:51 AM
Hmm, I've discovered the problem.
 
1:10 AM
I had forgotten to flatten the data. The data shape was 2D.
I believe that the histogram ended up considering the data as having multiple distributions at each record on axis 0.
 
 
5 hours later…
6:19 AM
@Marco Sorry, what do you even mean by strings in that comparison? Some specific (ISO) format of dates/times?
 
6:29 AM
The C datetime is certainly more efficient than strings at just 10 bytes plus 2 bytes metadata.
The Python datetime is a real Python type with 10 fields, so we're talking at least 80 bytes in.
A date string like 2024-03-25T07:33:13 ships in at 19 ASCII characters, so at least 19 bytes for the data.
Not sure how much overhead there is for the type metadata, but the native types (C datetime and string) should win here.
Anyways, proper C datetime should win in every regard. I wouldn't be surprised if numpy has extra representation for datetimes without tzinfo as well.
 
Apparently pandas doesn't care
import pandas as pd

df = pd.DataFrame({'a': ['2024-01-01', '2024-01-02', '2024-01-03']})
# print(df.dtypes)
print(df.memory_usage())

df['a'] = pd.to_datetime(df['a'], format='%Y-%m-%d')
# print(df.dtypes)
# print(df.head())
print(df.memory_usage())
 
6:49 AM
I have no idea what that output means. ^^
 
To the best of my knowledge, the strings and the datetimes both take the same number of bytes
Which I find super-suspicious hence the commented out bits where I wanted to assure myself it had actually cast the data from string
 
24 is suspiciously close to three pointers...
 
Yeah. "Lol, you wanted to see the heap size? Tough doo doo"
Probably a better test would be to generate a big df of strings and save that to a parquet file, then do the same with it as a datetime type, then look at the file size. I'll leave that as an exercise for Marco, then there need not be any more speculation
 
Well, at least you've given them some... puts on glasses ... pointers what to look for. YEAAAAAAHHHH
 
Why parquet? Because polars is backed by Arrow and pandas is moving that way too. To the best of my knowledge, the parquet format should be as close to its in-memory representation
@MisterMiyagi 😎
Then again, you can have different compression regimes even on parquet files. Why is nothing easy to figure out? :'(
 
7:06 AM
Memory is hard stuff.
 
For all practical intents and purposes, I think I'd be satisfied with a parquet file test, though, since it's relatively practical in workflows
 
At the very least that parquet would be a well-defined scenario.
 
7:21 AM
This got old quickly:
2 days ago, by roganjosh
The code block is beautifully book-ended by a VSCode bug that currently shows all the remaining body of the function as unreachable... for even more noob-looking impact :D
I don't want to hack things here, I hope VSCode just gets it fixed. It's borking quite a bit of code now for me. It seems to mess with auto-completion amongst other things
 
7:38 AM
Oh wow, that linked issue is… interesting.
 
8:02 AM
cbg
 
That this actually fixes the issue is just bizarre. Thanks, typing.
 
9:00 AM
Well, it does actually make sense in some sense. But the usability really is bad like that.
FWIW, it's a good thing you shared this. The type hints for my async library have the exact same problem.
 
Ah, well I'm glad that ranty roganjosh had some utility :)
 
9:16 AM
What's the issue, exactly? Why does that -> Never overload exist?
 
The -> Never overload is basically just a special case to type hint incorrect calls.
The problem is that when a type checker sees Any as an input, and has to infer the return type, it matches Any -> ??? against overloads. This will always match the first overload, since Any matches everything.
This is a problem in practice when the first overload is a special case.
starts writing type tests
 
Oh, that's nasty
 
9:50 AM
What I don't get is when exactly this triggers. For a simple toy with just overloaded (a: int | None) -> int | Never the Any path picks the int -> int variant. :/
bets two quatloo on "variance" being the buttler
 
10:20 AM
Offtop, but what should I search for if I want to have links from my site display some embedded info on my discord server?
 
10:36 AM
Yam. I'm in the middle of installing an 11 year old package on Py2.7 on our RHEL7 machines from source and I'm so glad we have wheels today.
 
 
3 hours later…
1:30 PM
@MisterMiyagi Today on "You wouldn't believe the typing horrors I have seen, deary!": MyPy only infers Any when resolving overloads with Any, PyRight picks the first overload but supports and prefers overloads with Any.
Imagine an immovable problem vs an unstoppable specification.
 
1:42 PM
On reading through the 'net a bit, this is apparently a big 'ol can o' worms. Cthulhu worms. And they are all gazing back into you.
 
What's worse - a Cthulhu worm staring at you or a roganjosh with a broken environment staring at you?
Only kidding. Cthulhu is so misunderstood by the mainstream media. We met for scones for breakfast only last weekend
 
2:09 PM
Cthulhujosh D:
Cthulhujosh :)
There, that's better.
 
2:22 PM
@MisterMiyagi Python datetime vs Python string
@MisterMiyagi Type metadata?
 
Yes, like timezone. I think the parquet test I mentioned would be most useful for you
 
Well, we can't forget that we are dealing with numpy arrays also
 
Except you might not be. Again, go with parquet
 
@roganjosh But I am not dealing with dataframes...
@roganjosh Great
 
@Marco Python string containing what?
 
2:27 PM
@MisterMiyagi 'YYYY-MM-DD HH:MM' format
Numbers :P
 
sigh
 
No, characters
 
Yeah, sorry
Characters representing numbers
@roganjosh Ok
 
This is why I suggest dumping it to a file. It doesn't matter whether you're really dealing with dataframes or not, the parquet file will give you some idea on how the values are held in memory. There is no other way to do it, really - you can't write datetimes to a text file without it being a string literal or being in a format that something else (e.g. Excel) understands.
 
Ok
Thanks
 
2:31 PM
And, we all know that writing datetime formats for Excel is hopeless anyway because it'll straight-up trash it where it can
 
Hmmm, yes
What is a parquet file?
 
I'm pretty sure Google has answers for that to save my fingers
 
Ok
Apache Parquet?
 
Yes
 
Ok, I didn't know this
 
2:38 PM
The point being that parquet is an ultra-fast format for getting things in and out of dataframes. What it dumps to disk, regardless of any fancy compression, should be pretty close to how datetimes are held in memory. It won't apply directly to a Python datetime but, then, when do we need to even care about having millions of them around?
 
Nice
Maybe I'll do this test within the next few days, but a simpler and more efficient test I thought of is to do a slice, for example, of 10 records in the 2 arrays and save them in an .npy, to see the difference in size between them.
 
Ok
 
Roganjosh, who here in the chat would you say is the most knowledgeable about Python data science/machine learning?
 
2:56 PM
FWIW, I just built your test case I suggested:
import pandas as pd

dates = []
for x in range(10000):
    dates.extend([
        '2024-01-01 00:01:00',
        '2024-01-02 00:02:00',
        '2024-01-03 00:03:00'
    ])

df = pd.DataFrame({'a': dates})
df.to_parquet("PRE-CONVERSION.parquet", index=False)
df['a'] = pd.to_datetime(df['a'], format="%Y-%m-%d %H:%M:%S")
df.to_parquet("POST-CONVERSION.parquet", index=False)
 
Oh, thank you very much
Nice
 
@Marco You mean, other than obviously me? But, joking aside, that question doesn't make sense. I have decent exposure in the field but there are a number of people that I just naturally defer to to answer questions. For example; does DS use async? Yes. Can I answer anything about it? No. Do people here know this better than me? Absolutely yes. Do some people have a better grasp on linear algebra? Absolutely yes.
It doesn't make sense to rank the room. Like, at all. I'll be honest - I've actually raised my RO status here a couple of times in interviews, not to say "well I'm a room owner here" but actually "I have a team of consultants if I get really stuck"
 
Okay, I believe what you mean is that this is a very broad question. And I could include you too, as far as I remember I asked about you and you said that you are more of a specialist in the area of operational research.
What would you do if you were in a situation where you are really struggling with a machine learning problem, you are not getting results, you don't know whether your problem is with the data or the model, or both, and you need to for help to save you?
@roganjosh Ok, I understand, maybe I could aks which ONES understand the most in general.
and you needed help from someone to save you?*
 
If I told you it's Bob, what difference would that make?
 
3:11 PM
@roganjosh It would be important information for me.
 
It certainly shouldn't be. Now you're taking my opinion on who is an expert? What makes me the arbiter of who is best-placed to answer?
 
Okay, I think you know everyone relatively well.
But that's okay, I won't ask about it anymore if you mind.
 
18 hours ago, by MisterMiyagi
@roganjosh I think datetime can express some things that timestamps can't. Like leap seconds and all that fun stuff.
@MisterMiyagi No, datetime is totally oblivious to leap seconds. But time has some very limited (and OS-dependent) support for them. docs.python.org/3/library/datetime.html docs.python.org/3/library/time.html#module-time
 
@Marco I don't mind but I think your idea of a hierarchy in this room might be skewed. There are plenty of discussions that I actually don't follow because they're way too technical on something I don't understand. I pull rank as a Room Owner when there's a problem - it doesn't mean I understand everything being said. I do try to get to know people, but regulars also know other regulars just as much as me
@PM2Ring have a go with this. I refrained from my "WTF" at the result :P
 
@roganjosh Ok
@roganjosh I just ran the test on a Google Colab, the file size before conversion is 1.6 KB, and after conversion, 1.59 KB. So datetime is a little lighter.
kB*
 
3:28 PM
@roganjosh Sorry, I don't know Pandas. I tried running that code on SageMathCell. It has Pandas, but it doesn't have parquet.
 
Basically, the files are the same size
 
Yes
 
It's not to do with pandas per sey, but that the datetime seems to be basically the same size as the string version
 
@PM2Ring Yam, I got tripped up by fold. Thanks for the correction!
 
@roganjosh if and when you can, could you please answer? chat.stackoverflow.com/transcript/message/57216585#57216585. I would greatly appreciate it.
 
3:38 PM
@MisterMiyagi No worries. In theory, leap seconds probably could be handled by the same machinery that handles daylight saving. But that would be too easy...
Fortunately, we haven't needed a leap second for several years. And we probably won't need another one for a while. There was even talk of having a negative leap second. That's in the standard, but we've never had one, so it would probably crash lots of stuff. ;)
We use leapseconds to ensure -0.9 seconds < DUT1 < 0.9 seconds, where DUT1 = UT1 - UTC. This discrepancy arises because UTC is based on atomic time (TAI) and UT1 is based on the rotation of the Earth.
 
@PM2Ring Hi, how are you? Are you an astronomer?
 
@PM2Ring I think I said this before, but keep 'em coming! I find this topic both fascinating and scary. ^^
It's a good thing I have no usecase that depends on getting this 100% right.
 
@Marco Hi Marco. I'm just an amateur astronomer, but I know a bit about time keeping as it relates to astronomy.
 
Okay, nice! It's because I also saw that you are very popular in the Astronomy Stack Exchange.
I mean, has big reputation.
I did a master's degree in Physics and Astronomy.
But I don't consider myself even close to having a good knowledge of the area despite my master's degree hahaha.
I believe that master's and PhD degrees serve more to show you how much you don't know about the area hahahaha.
 
3:55 PM
The IERS publish current DUT1 (and Earth rotation parameters) every week. datacenter.iers.org/data/6/bulletina-xxxvii-012.txt The data is available in various formats. datacenter.iers.org/availableVersions.php?id=6
 
@PM2Ring Weirddd and scaryyy
 
It's kinda scary that the fine details of Earth's rotation are so irregular that we can only predict them for ~2 weeks in advance. OTOH, for most purposes we don't need such high precision.
 
Yeah
 
We have excellent models of the larger factors involved in the rotation rate, and the orientation of the rotation vector. But predicting the small scale effects is tricky because it involves the effects due to circulation in the atmosphere, ocean, and mantle.
 
Wow
 
4:03 PM
It's not easy predicting the weather in the atmosphere & ocean. It's even harder predicting the "weather" in the mantle, since it's so hard to observe what's going on down there.
On the very long term, Earth's rotation is slowing down because tidal interaction transfers angular momentum from the Earth's spin to the Moon's orbit.
 
@PM2Ring Of courseee
@PM2Ring Ohhhh
 
On a somewhat shorter timescale, the continental plates in the northern hemisphere are still rebounding from the last ice age. As the plates rise, the rotation rate slows, like a spinning skater raising their arms.
It's pretty awesome that we can "see" that rise as a component of the change in Earth's angular momentum vector.
 
Roughly speaking planet Earth is just a "dust" of space, perhaps even our galaxy is also a dust for the entire universe.
 
Sure. In cosmology, when they model the expansion of space, they treat the universe as a homogeneous cloud of particles, with each particle being at least the mass of a galaxy, although the modelling is easier with heavier particles.
 
@PM2Ring Wonderful
Changing the subject a little, what is your main area of work using Python?
 
4:18 PM
We had a question about that kind of modelling a few months ago. astronomy.stackexchange.com/q/55010/16685 There are some excellent links on that page. Here's a very crude (but pretty) plot that I did:
 
@PM2Ring Great!!!! Thanks for sharing it.
 
@Marco These days, I do a lot of stuff related to astronomy, often using Solar System data from JPL NASA's Horizons system.
 
Hmm, nice. Do you primarily do data analysis?
 
Eg, here's a 3D plot of the Sun, Earth, and Halley's comet in 1910.
@Marco Not really. I don't do much analysis. I just extract the data and present it in plots and graphs.
 
Great
@PM2Ring Nice anyway
 
4:28 PM
I occasionally need to calculate mean values, though. My favourite technique for that is to construct a cubic Bézier spline passing through the points, and then calculating the exact integral under the Bézier curves.
 
Nice, nice
I'm more in the area of data science and machine learning, I'm looking for someone (preferably free of charge) to help me find out if my problem is in the data, in my model, or even in both.
:(
 
I used that technique to get the means shown on the graphs in this answer: astronomy.stackexchange.com/a/55112/16685
 
I'm doing my PhD and my data is related to space weather.
@PM2Ring Nice!
 
Sorry, I don't know much about machine learning. I've done a little bit of stuff with multi-variable optimisation, though.
 
No problem, thanks :)
 
4:36 PM
One nice thing about working with Horizons data is that I know it's (mostly) quite trustworthy. They integrate the equations of motion for all the major Solar System bodies and then fit the results to ground-based and space-based observations. If they make a mistake, spacecraft don't go where they're supposed to...
 
Hi my application is leaking open files, I'm not sure how I can go about debugging that lsof my process shows me just
python  201105  username  194u  a_inode               0,14        0   12497 [eventfd]
python  201105  username  195u  a_inode               0,14        0   12497 [eventfd]
python  201105  username  196u  a_inode               0,14        0   12497 [eventpoll]
and 100s of those
any ideas how I can get more info where exactly all these files are being opened? Ideally I could connect the to a specific line in my code
 
Those aren't real files. Something in your application is opening pseudo-files to receive event notifications.
Are you using something like inotify?
 
@MisterMiyagi nope, we are using protobuf for communications
Chatgpt tells me to try "strace -f -e trace=file python your_script.py" which I will tomorrow, too tired now
 
I'd probably rummage around in proc and look up each fd's metadata directly.
 
@MisterMiyagi how would you do that?
 
4:52 PM
ls -l /proc/201105/fd for example.
 
but that's the same as lsof or am I missing something?
Anyways thanks for the help, but I'm too scatterbrain right now to start this, shouldn't have started it even. See you guys tomorrow :)
 
You can navigate proc a bit better, but yeah, at the end of a day lsof has a flag to drag each and every bit from there to you.
Searching the source code for eventfd usage would also be a cheap stab in the dark with good chance to turn something up.
 
5:35 PM
@roganjosh that' a clever way of putting it. I wouldn't do that myself though since I don't feel as confident hmm
@Hakaishin you could log for anything in your application that open files (since it's Python, just log open() usage or such)
@PM2Ring that looks cool! reminds me of manifolds plotting or latent space plotting
 
6:01 PM
@NordineLotfi Thanks. It looks even better in 3D. ;)
 
 
4 hours later…
10:07 PM
SE still hasn't learned "If it ain't broke, don't fix it" meta.stackexchange.com/q/398680/334566
 
To err is human...
 
 
1 hour later…
11:32 PM
@Marco Go ahead and state your problem here and let's see. What's the exact issue? selecting a classifier/regressor? training not converging? feature engineering? choosing the objective function? something else? etc. When you say "you don't know whether your problem is with the data or the model, or both", that's too vague, give us more specifics. (Try a simpler model on your data, or the model you chose on a well-known dataset; or use crossvalidation).
 
Hi @smci, thanks for your answer, I'll explain more now
Well, in general, in my best results, I cannot obtain better convergences, the training losses end up becoming stagnant and consequently I end up not being able to improve the validation and/or testing losses.
Today I was thinking more about the problem and in the last few minutes I found something that can help me check how predictable or not my time series is, using Shannon entropy, I found this answer: [datascience.stackexchange.com a/95232/141037] (datascience.stackexchange.com a/95232/141037).
In fact, I just asked a question on CrossValidated to clarify whether I can apply it to my time series that is windowed (I applied a sliding window): stats.stackexchange.com/questions/643523/check-if-my-time -series-is-forecastable-using-shannon-entropy. This would help me know if I have checked the limit of what I can predict given the time series I am using.
 
@Marco Sorry this is still way too vague. What model are you trying to use, on what data? ARIMA on tidal heights? RNN on email marketing response? etc. (You'll have to say much more than "a time series"). What is the variable you are trying to predict? What is your objective function?
 
Oh God, I can't adjust the links.
 
Yes but what type of time series? That could easily be a billion different things.
(Don't sweat it about your link formatting)
 
@smci lately I'm using ConvLSTM. I'm trying to predict videos from data relating to space weather.
I know, I'm available to give all the details, it's just that at the time I believe I could speak in general, as I was just trying to find something that could help me.
 
11:46 PM
Ok, I don't know much about predicting spatial data. Did you first try a tutorial or standard dataset they use with ConvLSTM?
 
Of course, I'm not having difficulty using ConvLSTM, just having better results, in fact recently I want to know if my data is predictable, since I've already done extensive testing.
 
Is this work? research? bluesky experimentation? What simpler baseline approaches do people use in that area? Start with something simple and build out.
 
In the last few days I was considering that my data was unbalanced somehow (stats.stackexchange.com/questions/642885/… ) and in the last few days I confirmed that the beginning of my data was really unbalanced
but if you go to see the chat related to this question that I linked above -
https://chat.stackexchange.com/rooms/152163/discussion-between- marco-and-dikran-marsupial - the friend commented that considering that my data is related to the solar cycle I should use much more data than I am using to reasonably predict.
 
@Marco I don't know how to calculate Shannon entropy on a spatial timeseries. But a more empirical question is: how does your training converge? Don't say "the training losses end up becoming stagnant(?)", show us the graph of whatever your objective function is versus number of epochs. (Actually post that in your question on CrossValidated)
 
So only is possible to calculate Shannon entropy
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.entropy.html#scipy.stats.entropy using a flattened array?
Ok, I'll do these things
That's why I've posted that question: stats.stackexchange.com/questions/643523/…
:P
 

« first day (4909 days earlier)      last day (44 days later) »