last day (15 days later) » 

3:32 AM
2
Q: AVX 256-bit vectors slightly slower than scalar (~10%) for STREAM-like double add loop on huge arrays, on Xeon Gold

SeanI am new to AVX512 instruction set and I write the following code as demo. #include <iostream> #include <array> #include <chrono> #include <vector> #include <cstring> #include <omp.h> #include <immintrin.h> #include <cstdlib> int main() { unsigned long m, n, k; m = n = k = 1 << 30; auto...

 
@StaceyGirl Of course, you can see that in my code.
 
What hardware / microarchitecture, and compiler version? Using "just" 256-bit AVX like you're doing in the question shouldn't lower the clock speed very much; the bigger penalty is with AVX512 which you enabled at compile-time but didn't un-comment. (Memory bandwidth may depend on uncore clock speed. I think modern Intel keeps uncore clock speed fixed even if none of the cores happen to be running at max turbo, though.) I'd recommend compiling with -O2 -march=native to enable and tune for everything your CPU has.
@StaceyGirl: GCC compiles _mm512_store_pd to vmovapd or ps which would fault on unaligned; we can rule out unaligned loads/store that overlap and do every element twice. But yeah, I wondered the same thing at first. Compiling with MSVC or ICC would use vmovupd even for alignment-required intrinsics, letting you silently make that mistake.
For the record: a double with bit-pattern 0x101010101010101 (produced by memset(1)) represents a value of ~7.75E-304 (binaryconvert.com/…). This is tiny but not subnormal: it has a non-zero exponent. So there won't be any subnormal inputs or outputs to the vaddpd / addsd. I'm surprised that scalar -O2 is faster than 256-bit vectorized; it should be at best equal, or more likely worse with out-of-order exec not seeing as far ahead to trigger TLB misses earlier and so on.
Amusingly, g++-9.2 -O3 does loop-inversion: repeats each add 30 times before moving on. (Or do a 30-iteration empty loop if you don't use the c[] result) inside an outer loop that increments pointers). godbolt.org/z/YmGDjs. So you can only use -O2 unless you take extra steps to avoid -O3 full optimization defeating your benchmark.
 
@PeterCordes Yes, I only use the -O2 optimization.
 
I know you do, I was just saying that you can't use -O3 to see if it helps the compiler make better or different asm (e.g. pointer increments instead of indexed addressing modes).
 
@PeterCordes I have attached my cpu information above. I also use the random number to fill in my array and perform the add operation. The AVX version is slower than the original version.
 
3:32 AM
Does the performance ratio match the CPU frequency ratio between runs? (Lower max turbo with AVX than scalar?) But yeah, Skylake Xeon has quite low single-core memory bandwidth (worse even than previous generations of many-core Xeons) so it's no surprise that a single thread can pretty much saturate it with 2 read + 1 write stream of 8-byte loads/stores, even without GCC doing loop unrolling. Do you really want to get to the bottom of why AVX slowed it down the fully memory-bound case, or do you just want to look at a more passes over smaller arrays where AVX gives a big speed?
And BTW, you generally don't need to manually vectorize loops this simple. Just use -O3 -march=native to get good code (for the non-benchmark case where there's no repeat loop to worry about). The default -mprefer-vector-width=256 is often good, but also try 512.
 
@PeterCordes Actually, at first I just want to see the big performance from AVX. But now, I also want to know why AVX slowed it down. Thanks
 
at first I just want to see the big performance from AVX As Mysticial explains in the Q&A Daniel linked above, you can't do that with arrays this big. A speedup could only come from using multiple cores to use more total memory bandwidth. It's interesting and unexpected that AVX causes an actual slowdown, though. Can you consistently reproduce that? What gcc version (gcc -v)? Can you use perf stat on your runs to record average CPU frequency during each run?
 
@PeterCordes The g++ version is gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) . But I am sorry that we do not install the linux-tools-common on server and so far I do not have the permission to install it.
 
edit that into the question. For CPU frequency, grep MHz /proc/cpuinfo while the test is running. For anything more in-depth to test any other hypothesis, you're going to need perf. You can probably just extract the perf binary from the relevant Ubuntu linux-tools-x.y....deb and run it from your ~/bin, I think without even even needing a library + LD_LIBRARY_PATH. You should be able to profile user-space code without ever needing root or modifying any sysctl settings. (The default for kernel.perf_event_paranoid doesn't allow profiling kernel code but that's fine for this)
 
I have edited my question. I just use the command grep MHz /proc/cpuinfo to see my cpu state when I am running the both version. I found that one of cpu ratio is much higher than others. Is it overclocking?
@PeterCordes Actually, I have tried to use sse2 and it seems that there is also no improvement at all.
 
3:39 AM
Turbo is sort of like overclocking when power / thermal budget allows, but it's really just DVFS / the opposite of downclocking on idle working as intended within design limits. So in that sense not overclocking beyond factory rating. ark.intel.com/content/www/us/en/ark/products/120491/… has "base" 2.1GHz, max turbo 3.7 GHz.
@Sean Not sure if Intel publishes the L1 / L2 turbo "license" table for every CPU for light/heavy AVX / AVX512. stackoverflow.com/questions/56852812/… says wikichip usually has them.
@Sean You mean it was exactly the same speed as scalar? That hopefully rules out some kind of effect from manual vectorization / intrinsics or not. And confirms that scalar was saturating per-core memory bandwidth. What about AVX with 128-bit vectors? Machine-code instruction sizes should be identical to AVX with __m256d there, unlike with the legacy non-VEX encoding for actual SSE if you left out any -m options. (compile a __m128d version with -march=native or at least -mavx)
 
 
1 hour later…
4:50 AM
@PeterCordes I use the sse2 instruction with 128-bit vectors and finally, the result is that the sse2 is a little bit slower than the naive version.
As you said, the max turbo 3.7 GHz. I check the cpu information during executing my program and I find that the cpu rating is lower than that.
@PeterCordes By the way, in the past, I used the sse2 instruction to boost my program. It is much easier to see the improvement. you can see my Github project as this link. github.com/crownk1997/Mixed-Model-Net/blob/master/src/…. line 249
 
@Sean How much is "a little bit", and how much is "lower than that". The specific numbers might not help you come up with an explanation, but that's why you're asking experts...
 
@PeterCordes One of cpus achieve 3363 MHz.
The sse2 version is slower 10% than the naive one.
 
@Sean Sure, if you're not memory-bound, SSE2 can give the "expected" factor of 2 speedup for simpler vertical loops. With such a low computational intensity (ALU ops per load/store of the data), that'll only happen when your data fits in L1d cache, or probably also L2.
@Sean Oh, so it's the same speed as AVX?
 
Yeah.
@PeterCordes From my perspective, the cache line should be able to contain 2 double precision floating points. When I used sse2 instruction, there might be some improvement.
 
@Sean That's obviously important and checking for that was the whole purpose of the experiment. Why didn't you say that in the first place? Of course it's not going to be faster than AVX.
 
5:00 AM
@PeterCordes Because I just did the experiment.
@PeterCordes Well, let's come to the simplest case. I just want to see the improvement from sse2 instruction. I think the simple two loads, one add and one store should give me that. Is it right?
 
@Sean I mean your first report of it was that it was "a little bit slower than scalar"
 
@PeterCordes Sorry for my vague destriptionn.
describe
 
@Sean Did you mean SIMD vector, not cache line? Have you still not read and understood Mysticial's answer about memory bandwidth? Nothing is going to be significantly from scalar when you're using huge arrays that don't fit in cache so the CPU core bottlenecks on memory bandwidth: prefetch can't keep up with demand loads so you get loads that miss in L2 and even L3 cache, stalling the pipeline while they wait for data from memory.)
Using wider vectors just means it takes fewer cycles to handle a cache line when it does arrive from memory, but the loads for the next cache line are already in flight so this faster computation happens in the shadow of a cache miss.
@Sean Like I've told you several times now, if you want to see SSE2, AVX, and AVX512 be faster than scalar, make your arrays much smaller. Specifically, a total footprint under 1MiB = L2 size, and crank up the repeat count so you can still measure it. Cache bandwidth is much higher than memory bandwidth; that (and latency) are why cache exists.
 
5:23 AM
@PeterCordes Thank you for your patient explain first. I exactly understand about memory bandwidth. But I think I am confused about the prefect. In the past, I always believed that although my array is large, it can be perfected into cache for the further computation.
One more question please. If my array is very large, for example 1GB, the OS will help me first load the former part into cache which allows me to use SSE2, AVX to improve the performance. Why not this case?
@PeterCordes Thank you
@PeterCordes Actually, I write another demo.
#include <emmintrin.h>
#include <cmath>
#include <iostream>
#include <chrono>

void normal(float* a, int N)
{
for (int i = 0; i < N; ++i)
a[i] = sqrt(a[i]);
}

void sse(float* a, int N)
{
// We assume N % 4 == 0.
int nb_iters = N / 4;
__m128* ptr = (__m128*)a;

for (int i = 0; i < nb_iters; ++i, ++ptr, a += 4)
_mm_store_ps(a, _mm_sqrt_ps(*ptr));
}

int main(int argc, char** argv)
{

int N = 64000000;
auto *a = static_cast<float*>(aligned_alloc(128, N*sizeof(float)));


std::chrono::time_point<std::chrono::system_clock> start, end;
In this case, my array is still large, but I can still get a lot improvement.
 
5:46 AM
@Sean Are you confusing disk cache (using RAM to avoid disk IO, OS managed) with CPU cache (using internal fast SRAM to avoid access to DRAM, hardware managed)? When you loop over the arrays the 2nd time, since the last time you touched the beginning of the arrays, you touched every other part. So every part you touch has long since been evicted. Hardware doesn't cache the "first part"; it doesn't even know that it's all one array, just trying to cache each cache line separately.
Although blog.stuffedcow.net/2013/01/ivb-cache-replacement does show that Intel CPUs are more sophisticated than just LRU replacement in their L3 cache sometimes. But apparently not for sequential access; that does still tend to blow away everything else.
 
6:09 AM
@PeterCordes But what is difference between my original demo code and the one I just show you? It seems that the new demo code involves more computation.
 
 
11 hours later…
5:02 PM
@Sean sqrt is slow enough that ALU becomes the bottleneck, not memory, at least for scalar. See agner.org/optimize for instruction timings. Also, you're updating one array in-place, so you only need about half the memory traffic of your previous case (pure store of c[] isn't much cheaper than load + store of a[], so your c[] = a[] + b[] was basically 4 cache-line transfers vs. 2 for your sqrt version).
Other things like computing a 6th order polynomial or something with FMA could also be slow enough that memory wasn't the bottleneck for scalar, and could scale all the way to AVX512. (But remember clock-speed reductions). The hardware SIMD divider is "only" about 256 bits wide, and not fully pipelined. 128-bit sqrt instructions are IIRC enough to maximize sqrt throughput, if the front-end isn't a bottleneck.
 

last day (15 days later) »