From owner-freebsd-arch@FreeBSD.ORG Tue May 10 12:46:20 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96915106566C for ; Tue, 10 May 2011 12:46:20 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E21348FC1D for ; Tue, 10 May 2011 12:46:19 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA13985 for ; Tue, 10 May 2011 15:46:18 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4DC93399.8070706@FreeBSD.org> Date: Tue, 10 May 2011 15:46:17 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110504 Lightning/1.0b2 Thunderbird/3.1.10 MIME-Version: 1.0 To: FreeBSD Arch X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Subject: some inter-cpu interaction changes for review X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2011 12:46:20 -0000 I've created a new branch user/avg/xcpu where I committed a number of changes that I would like to propose for the main tree. The changes affect CPU stopping, both hard (for panic and kdb) and soft (for suspend, reboot, etc); CPU rendezvous/cross-calling; panic; kdb; system reset; TLB shootdowns for x86. MD changes are provided only for amd64 at the moment (i386 should be trivial to add). Some highlights of the changes: o hard stopping is completely separated from soft stopping Hard stopping is performed in a limited number of very special cases. It also should have an ability to "penetrate" (almost) any contexts on CPUs being stopped. o new approach to avoid inter-cpu deadlocks The idea is that if a CPU waits either (a) on a lock protecting some inter-CPU actions or (b) for other CPU to perform some action request by the master CPU, then the waiting CPU should also check for incoming events while spinning. For example, if two CPUs try to hardstop all other CPUs (and each other), then the CPU that failed to acquire a hardstop lock should check if it's being hardstopped while spinning on the lock. Similarly, when a CPU requests an action to be performed on a set of CPUs and it waits for the action to be completed by all the CPUs in the set via the rendezvous mechanism, then that CPU should check for incoming rendezvous requests while spin-waiting. o rendezvous actions can be done in parallel Multiple rendezvous requests from different CPUs can be piggy-backed onto the same IPI to a target CPU. o x86 TLB shootdowns are re-implemented via the rendezvous mechanism Please see the branch for all the changes and details. Please review and/or test (on amd64 only) these changes. I will appreciate any comments, suggestions, critique and help. Especially the help with MD bits for other architectures iff you find the suggested approach to be correct and useful. Thank you! P.S. I see the largeSMP work by Attilio and I think that those changes should go in first and then I will adapt my changes for cpumask_t -> cpuset_t transition. That is, I do not want to cause any delays for Attilio's work. -- Andriy Gapon