last day (15 days later) » 

6:29 AM
7
Q: Why is np.dot imprecise? (n-dim arrays)

OverLordGoldDragonSuppose we take np.dot of two 'float32' 2D arrays: res = np.dot(a, b) # see CASE 1 print(list(res[0])) # list shows more digits [-0.90448684, -1.1708503, 0.907136, 3.5594249, 1.1374011, -1.3826287] Numbers. Except, they can change: CASE 1: slice a np.random.seed(1) a = np.random.ran...

 
CASE 1 is curious. It seems that the same multiplications should be reported in each row because of the selection [0] outside the dot product. But more and more irrelevant work is being done as i increases.
 
0 0
For what it's worth, I can't reproduce this. I wonder if it has something to do with underlying (Fortran) libraries, and then possibly with CPU optimizations.
For the record, my output is a consistent repeated set of lines of [-0.9044868, -1.1708502, 0.90713596, 3.5594249, 1.1374012, -1.3826287]. So, essentially your first line for case 1.
 
@00 Updated; for context, this discrepancy was spotted in my investigation of TensorFlow model prediction inconsistencies, where outputs depended on number of samples fed (dimension 0, i.e. a[0] is the prediction for first sample). Unfortunately np.dot is encapsulated in a compiled C implementation, and I'm not digging that far
 
Two things to keep in mind: 1. Floating point arithmetic is inherently imprecise. Read stackoverflow.com/questions/588004/… for a discussion of this. 2. Output is a string representation of the floating point number. It is not the actual value stored in memory. For example, it might be truncated or rounded.
 
@Code-Apprentice This fails to explain the consistent inconsistencies; two same numbers multiplied should not yield a different output depending on what other numbers from the same array are involved. Further, it's not an output representation problem, the values actually differ, and differ notably in source application. This SO is more relevant, but incomplete w.r.t. to my question.
 
6:29 AM
Seems likely that the implementation takes a different code path for different input shapes and strides. This may involve performing operations in a different data type, or with SIMD instructions (affecting precision and operation order). You shouldn't expect consistency in the first place, anyway.
 
I'm not seeing any discrepancies in CASE 1 or CASE 2 as shown on an Intel i5-3320M, running Ubuntu 18.04.3 LTS, ipython 5.50, python 2.7.15+.
Also no CASE 1/2 discrepancies on AMD FX-8150, Ubuntu 18.04.2 LTS, ipython, python 2.7.15rc1
The "stress-test" example also isn't showing any differences for me.
CASE 1 has discrepancies in 2 cells from the first line on Google Colab (Jupyter-as-a-service). Python version is 3.6.8. Google Colab's environment has a /proc/cpuiinfo entry for a 2 core Xeon @ 2.3 Ghz (but its probably a VM). You can access my shared Google Colab notebook here while it lasts.
 
If you want higher precision, using float64 (aka double precision) is probably wise. For double the ram usage, you'll get something like millions of times more precision, which might knock down the inherent floating point errors down to the insignificant level again, even if they accumulate over many calculations. It's worth noting that Python's native float type is double precision, only numpy lets you use single precision floats, which are pretty notorious for their errors.
 
@Blckknght You are right; float64 does solve this, but it's hardly a 'solution' for an application where speed is critical: training deep neural networks. Thus, a workaround is highly desirable.
 
CASE 1 has discrepancies in 2 cells at top row (vs all other rows) on Jupyter run on docker desktop for Win 10 Pro on an Intel i7-8700K. Docker command line is docker run -d -p 127.0.0.1:8888:8888 jupyter/scipy-notebook followed by docker logs <containerid> to get the login URL. Python is 3.7.3.
 
@Paul Added your results to the question - thank you for your work, and for the question bounty
 
6:29 AM
@OverLordGoldDragon I had to edit the cases a bit. In the examples I've seen, only the first line is different. You might try announcing this on jupyter gitter or as an issue against jupyter on github.
@OverLordGoldDragon Interesting that it doesn't happen on the older hardware.
 
@Paul Quite unfortunate this problem's so environment-dependent - and it appears imprecisions diminish w/ CPU quality, which spells bad news for typical users. As Numpy's built on C, I now seek a C-level explanation, which may shed light on a workaround aside swapping to float64. And yes, older hardware winning here is odd - perhaps it's per the older times' prioritizing of reliability over performance, whereas today it's the FLOPS that sell. I just hope the answer isn't buried in assembly
 
@OverLordGoldDragon The other thing is that the jupyter environments saw the issue but not the ipythons. I have docker on those platforms. When I have time I may check it.
 
Don't have time to dig deeper, but I strongly believe this links whether, and to which, BLAS libraries numpy is leveraging under the hood. The 1d and 2d cases map to different BLAS functions, so no big surprise there. Try recreating w & w/o MKL and so on.
 
@BiRico Finally something specific - thanks for noting, added your info to the question
 
Just a note on pairwise summation. It is not as you assume faster but less precise. On the contrary it is a bit slower but typically much more precise. See eg here for a rather impressive characterization.
 
6:29 AM
@PaulPanzer Thanks, fixed - and nice link
Some C implementations I found for reference: sdot.c; arraytypes.c
@Paul Improved stress test; discrepancies exacerbate, and become more frequent, when increasing the inner dimension - and 9999 is hardly atypical in my (and many) applications; would be good to know if your envs continue to perform consistently for that and higher dims
@Paul Doubt it; from what answers I've read on SO, methods may compete via mere approach to determining row/col dimensions - so sorting by numeric size would involve, well, 'sorting', which should be ridiculously expensive - also confirmed via np.random.uniform(-1, 1, ...). Order of ops for size would also reflect in varying the random seed - but doesn't
 
@OverLordGoldDragon Strangely, on the same machine, using 9999 as the inner dimension creates issues for the dockerized jupyter/scipy-notebook environment (e.g. 10.002979 vs. 10.003017; -21.336536 vs. -21.336555) in all cells of the first two rows. Subsequent rows match the 2nd row. The iPython and iPython3 environments on the same machine. are OK and all rows match.
Possibly, you might want to unpack/examine the ipython vs. jupyter environments. Alternatively, you could probably get the build scripts from github and look for stuff thats different. The numpy for ipython is separate. I'm using the ubuntu apt-get package manager for the ipython and numpy.
apt-get package versions. ipython 5.5.0-1, ipython3, 5.5.0-1, python-numpy 1:1.13.3-2ubuntu1, python3-numpy 1:1.13.3-2ubuntu1
Notice that there were ubuntu-specific patches made to the numpy library.
OK. When I run dpkg-query in the dockerized jupyter environment; I can see that numpy is not installed via the apt-get package manager. It is an ubuntu-like environment.
The numpy versions from np.version.version are 1.9.2 and 1.13.3 in the ipython(2) and ipython3 environment on the i7-8700K but 1.17.3 int the dockerized jupyter enviroinment.
 
@Paul I'm surprised the extended discussion bot is only now kicking in; consider posting more text per comment. Also consider using the following to get env info: np.show_config(); I also tested against multithreading, but does nothing, at least per these settings. Doubt I'll be digging into Jupyter's source and the like, that'd be overkill for me
 
Jupyter should be just plumbing, much as ipython is. The differences might be in a specific version or compile of numpy. That might be something that can be changed in a build script. Alternatively, it might be possible to "re-install" numpy to an older/newer version from the docker terminal.
I think you should "move to chat" because otherwise some moderator or bot will do it.
 
@Paul "something that can be changed in the build script" - now that sounds like a workaround, though I wouldn't know where to begin this "plumbing"; which version of what am I looking for? Numpy? BLAS? Whatever it is, it can't afford to conflict with TensorFlow, the source application. As for 'chat', it broke my record for the longest comment chain - so I'll let robos handle it
 
I suppose dependency hell, testing, etc., is indeed an issue if you want to do anything important with it....
 

last day (15 days later) »