From owner-svn-src-all@freebsd.org Wed Feb 20 02:40:39 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7ED5414E2A51; Wed, 20 Feb 2019 02:40:39 +0000 (UTC) (envelope-from bde@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2104487FB7; Wed, 20 Feb 2019 02:40:39 +0000 (UTC) (envelope-from bde@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0920E26E6F; Wed, 20 Feb 2019 02:40:39 +0000 (UTC) (envelope-from bde@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x1K2ecLk075498; Wed, 20 Feb 2019 02:40:38 GMT (envelope-from bde@FreeBSD.org) Received: (from bde@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x1K2ec5b075465; Wed, 20 Feb 2019 02:40:38 GMT (envelope-from bde@FreeBSD.org) Message-Id: <201902200240.x1K2ec5b075465@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bde set sender to bde@FreeBSD.org using -f From: Bruce Evans Date: Wed, 20 Feb 2019 02:40:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r344332 - head/sys/x86/x86 X-SVN-Group: head X-SVN-Commit-Author: bde X-SVN-Commit-Paths: head/sys/x86/x86 X-SVN-Commit-Revision: 344332 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 2104487FB7 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.955,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Feb 2019 02:40:39 -0000 Author: bde Date: Wed Feb 20 02:40:38 2019 New Revision: 344332 URL: https://svnweb.freebsd.org/changeset/base/344332 Log: Fix hangs in r341810 waiting for AP startup. idle_td is dereferenced without thread-locking it to make its contents is invariant, and was accessed without telling the compiler that its contents is invariant. Some compilers optimized accesses to the supposedly invariant contents by moving the critical checks for changes outside of the loop that waits for changes. Fix this using atomic ops. This bug only showed up for the following configuration: a Turion2 system, amd64 kernels, compiled by gcc, and SCHED_4BSD. clang fails to do the optimization with all CFLAGS that I tried, because it doesn't fully optimize the '__asm __volatile' for cpu_spinwait() although this asm has no memory clobber. gcc only does the optimization with most CFLAGS. I mostly used -Os with all compilers. i386 works because gcc -m32 -Os only moves 1 or the 2 accesses outside of the loop. Non-Turion2 systems and SCHED_ULE worked due to different timing (when all APs start before the BP checks them outside of the loop). Reviewed by: kib Modified: head/sys/x86/x86/mp_x86.c Modified: head/sys/x86/x86/mp_x86.c ============================================================================== --- head/sys/x86/x86/mp_x86.c Wed Feb 20 02:14:41 2019 (r344331) +++ head/sys/x86/x86/mp_x86.c Wed Feb 20 02:40:38 2019 (r344332) @@ -1088,8 +1088,8 @@ smp_after_idle_runnable(void *arg __unused) for (cpu = 1; cpu < mp_ncpus; cpu++) { idle_td = pcpu_find(cpu)->pc_idlethread; - while (idle_td->td_lastcpu == NOCPU && - idle_td->td_oncpu == NOCPU) + while (atomic_load_int(&idle_td->td_lastcpu) == NOCPU && + atomic_load_int(&idle_td->td_oncpu) == NOCPU) cpu_spinwait(); kmem_free((vm_offset_t)bootstacks[cpu], kstack_pages * PAGE_SIZE);