From owner-svn-src-all@freebsd.org Fri Dec 18 19:15:26 2015 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CB6CA4B1B6; Fri, 18 Dec 2015 19:15:26 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id 4325A178A; Fri, 18 Dec 2015 19:15:26 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 5AB8E56489; Fri, 18 Dec 2015 13:15:25 -0600 (CST) From: Eric van Gyzen Subject: Re: svn commit: r290014 - in stable/10: lib/libthr/arch/amd64 lib/libthr/arch/i386 libexec/rtld-elf/amd64 libexec/rtld-elf/i386 share/mk References: <201510261621.t9QGLuL2028872@repo.freebsd.org> <71109998-711D-4ECA-9B44-5A7B1F8705F3@FreeBSD.org> X-Enigmail-Draft-Status: N1110 To: David Chisnall Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Message-ID: <56745B49.6050903@FreeBSD.org> Date: Fri, 18 Dec 2015 13:15:21 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <71109998-711D-4ECA-9B44-5A7B1F8705F3@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Dec 2015 19:15:26 -0000 David, I apologize for the slow reply. Your message went to my "stable" box, which I read less often. On 11/14/2015 12:30, David Chisnall wrote: > On 26 Oct 2015, at 16:21, Eric van Gyzen > wrote: >> >> One counter-argument to this change is that most applications >> already use SIMD, and the number of applications and amount of SIMD >> usage are only increasing. > > Note that SSE and SIMD are not the same thing. The x86-64 ABI uses > SSE registers for floating point arguments, so even a purely scalar > application that uses floating point will end up faulting in the SSE > state. I'm aware. Using the term "SIMD" was an admittedly weak attempt to be platform agnostic. > I believe that the no-sse option for clang is ABI-preserving, so will > not actually disable all SSE unless you also specify -msoft-float. I'm afraid that's not the case: $ cat square.c double square(double x) { return (x*x); } $ clang -mno-sse -c square.c fatal error: error in backend: SSE register return with SSE disabled clang: error: clang frontend command failed with exit code 70 (use -v to see invocation) FreeBSD clang version 3.7.0 (tags/RELEASE_370/final 246257) 20150906 Target: x86_64-unknown-freebsd11.0 [snip] Shall I file the bug report, as it suggests? > I don’t think that libthr uses floating point anywhere, but libc does > and you only need to call one function that takes a floating point > argument in between context switches to lose this gain on x86-64. > With this change, we’re making the compiler emit less efficient code, > on the assumption that nothing will touch the fpu in the quantum > before the next context switch. I’d really like to see the set of > applications that you benchmarked the change with on x86-64 to reach > the conclusion that this is a net win overall. > > Or, to put it another way: How many applications are multithreaded > but don’t use any floating point code? If I showed you the applications that I care about the most, I would risk losing my job. When we updated from FreeBSD 9 to 10, we measured a significant loss in performance. This was due to multiple factors, one of which was that clang started using SSE widely. We were not yet using that version of clang for our own code, so most of the performance loss was due to the usage of SSE in libthr. Using -mno-sse restored the lost performance. It's possible that we lost performance due to SSE in other libraries; I haven't pursued this. These applications only use floating-point in some rare corners of management code, not in any performance-sensitive paths. They also don't use libc very much. On a recent head, I used this script https://people.freebsd.org/~vangyzen/thr_sse/thr_sse_file_line.sh to generate this list https://people.freebsd.org/~vangyzen/thr_sse/thr_sse_file_line.txt of line numbers in libthr that use SSE. I manually reviewed those to write this list: https://people.freebsd.org/~vangyzen/thr_sse/thr_sse_uses.txt The vast majority of these simply aren't interesting, because they would not be called in a performance-sensitive code path, or the code that uses SSE pales in comparison to the weight of the surrounding code. The only one that I find truly interesting is mutex_unlock_common(), which uses SSE to NULL two pointers in the "fast path", which is rather lightweight. So, I wrote this nanobenchmark https://people.freebsd.org/~vangyzen/thr_sse/movups/ to measure the effect of using SSE in such a way. I ran it on five machines and got these results: https://people.freebsd.org/~vangyzen/thr_sse/movups/summary.txt As you can see, most of them show no significant difference. One machine, however, showed a 16.7% improvement with SSE. I find this fascinating, and I honestly can't explain it. As always, I welcome feedback. I then wrote this /slightly/ more realistic microbenchmark https://people.freebsd.org/~vangyzen/thr_sse/mutex_bench/ which uses pthread_mutex_unlock and therefore mutex_unlock_common. I ran it on /that/ machine. I got these results: https://people.freebsd.org/~vangyzen/thr_sse/mutex_bench/summary.txt When libthr was compiled without SSE, the throughput was improved by 7.25%. Performance of a real-world application improved 3-5%. I honestly don't like the change any more than you do. I committed it just because it helped us measurably, it might help others, and I doubt it hurts anybody. If that last point is disproven, I'll be happy to revert it. Now, I look forward to a lively discussion. :) Eric