« first day (4988 days earlier)      last day (93 days later) » 

3:12 AM
@Qualcuno2 A Python float uses the IEEE-754 binary64 format, which has 53 bits of precision. If you double the scale each time you zoom into the Mandelbrot, you lose (at least) 1 bit, so after ~45 zooms you only have about 8 valid bits remaining. The only way around that is to use a higher precision format to hold your point coordinates. Eg, in plain Python you could use fractions, but I guess Numba doesn't support them.
Another issue that arises with floating-point numbers is catastrophic cancellation
3:26 AM
Numpy provides longdouble and clongdouble, numpy.org/doc/stable/reference/… but (I think) that docs page says their existence depends on your platform.
@AndrasDeak--СлаваУкраїні OK, so that's very close to what I had come to understand, thanks for the clarification
3:40 AM
Going in the opposite direction, the latest GPUs provide 8 bit minifloats, which are popular in LLMs. en.wikipedia.org/wiki/Minifloat
@PM2Ring Hmm, I feel like I heard something about GPUs splitting 32 or 64 bit instructions into parallel 4x8 or 8x8 instructions. Regardless of if I'm remembering things correctly. Do you know if 8bit numbers help with speed?
@Peilonrayz Well, yes, they are doing arithmetic in parallel, so using 8 bit vs 16 bit doubles the amount of data you can process at each step. But more importantly, it doubles the amount of data you can store per GB of RAM and HD space, which is important for these large models.
A LLM token is represented as a vector in a space with a lot of dimensions. You don't need a lot of precision for each component, but you want a lot of dimensions.
An old mathematical joke says "If you peel a 1000 dimensional apple, use a very sharp knife, or you'll throw most of the apple away".
In other words, when n is huge, most of the points in a n-dimensional ball are very close to the surface.
3:56 AM
Thanks, forgot about large magnitudes of dimensions.
@PM2Ring Ah yeah, an apple of size 100 in 1000 dimensions makes a lot of sense.
4:08 AM
GPT-2 used 1600-dimensional vectors for its word embedding, GPT-3 uses 12,888 dimensions. IIRC, GPT-3.5 dropped that slightly, but increased other stuff. Details for GPT-4 are not easy to find...
4:53 AM
@Peilonrayz CPUs do this too with SIMD. The smaller the values the more that you can do in parallel
@roganjosh Nice, reading through the page I noticed SSM... which I have seen for a while now, but never actually looked into.
/SSM/SSE/s
That said, I think the mini floats were created by NVIDIA so I'm not sure the intrinsics exist to make use of them on a CPU, but numpy etc. can do it with the traditional data types
5:22 AM
Hmm, seems it should be possible
Aaand stuff like this is where I realise I'm completely out of my depth and should probably go wash the pots or some other mundane task I can actually understand
 
3 hours later…
8:24 AM
duplicate; see discussion (you may need to re-find a proper target) stackoverflow.com/questions/38189660
 
5 hours later…
1:38 PM
morning cabbages, folks!
 
5 hours later…
7:07 PM
Hello, can anyone please help me understand how to use this SSIM function (tensorflow.org/api_docs/python/tf/image/ssim)? The filter_size parameter has a default value of 11, according to the documentation information, "the image sizes must be at least 11x11 because of the filter size." If I then use an image of size 8x11, how should I adjust this, and if applicable, other function parameters? Thanks.

« first day (4988 days earlier)      last day (93 days later) »