From owner-freebsd-current@freebsd.org Fri Dec 16 19:31:50 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D95A2C83E9E for ; Fri, 16 Dec 2016 19:31:50 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C7DC0819 for ; Fri, 16 Dec 2016 19:31:50 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: by mailman.ysv.freebsd.org (Postfix) id C73E4C83E9D; Fri, 16 Dec 2016 19:31:50 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6EA3C83E9C for ; Fri, 16 Dec 2016 19:31:50 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238]) by mx1.freebsd.org (Postfix) with ESMTP id 92BB4815; Fri, 16 Dec 2016 19:31:50 +0000 (UTC) (envelope-from luigi@onelab2.iet.unipi.it) Received: by onelab2.iet.unipi.it (Postfix, from userid 275) id B553C7300A; Fri, 16 Dec 2016 20:45:19 +0100 (CET) Date: Fri, 16 Dec 2016 20:45:19 +0100 From: Luigi Rizzo To: David Chisnall Cc: Alan Somers , "current@freebsd.org" Subject: Re: best approximation of getcpu() ? Message-ID: <20161216194519.GA71398@onelab2.iet.unipi.it> References: <20161216021719.GA63374@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2016 19:31:50 -0000 On Fri, Dec 16, 2016 at 09:29:15AM +0000, David Chisnall wrote: > On 16 Dec 2016, at 03:10, Alan Somers wrote: > > > > What about pthread_setaffinity(3) and friends? You can use it to pin > > a thread to a single CPU, and know that it will never migrate. > > This is not a useable solution for anything that needs to live in a library and also doesn???t solve the problem. > > The Linux get_cpu call() is used for caches that are somewhere between global and thread-local. Accessing them still requires a lock, but it???s very likely to be uncontended (contention only happens when you???re context switched at exactly the wrong time, or if a thread is migrated between cores in between the get_cpu() call and usage) and so you can use the userspace fast path for the lock and not suffer from cache contention effects. > > One x86, you can use cpuid from userspace and get the current core ID. I have some code that does this and re-checks every few hundred accesses, storing the current CPU ID in a thread-local variable. Using the per-CPU caches is a lot faster than using the global cache (and reduces contention on the global cache). It would be great if we could have a syscall to do this on FreeBSD (it would be even better if we could have specify a TLS variable that the kernel automatically updates for the userspace thread when the scheduler migrates the thread between cores). indeed the following line seems to do the job for x86 asm volatile("cpuid" : "=d"(curcpu), "=a"(tmp), "=b"(tmp), "=c"(tmp) : "a"(0xb) ); (there must be a better way to tell the compiler that eax, ebx, ecx, edx are all clobbered). 0xb is the CPUID function that returns the current APIC id for the core (not necessarily matching the OS core-id) The only problem is that this instruction is serialising and slow, seems to take some 70-100ns on several of my machines so you cannot afford to call it at all times but need the value cached somewhere. Exposing it as thread local storage, or a VDSO syscall, would be nicer because the kernel knows when it is actually changing value. cheers luigi