From owner-freebsd-hackers@FreeBSD.ORG Fri Nov 16 13:18:24 2012 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AB6E820D; Fri, 16 Nov 2012 13:18:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id ADBB28FC12; Fri, 16 Nov 2012 13:18:23 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA16173; Fri, 16 Nov 2012 15:18:21 +0200 (EET) (envelope-from avg@FreeBSD.org) Message-ID: <50A63D1D.9090500@FreeBSD.org> Date: Fri, 16 Nov 2012 15:18:21 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121029 Thunderbird/16.0.2 MIME-Version: 1.0 To: attilio@FreeBSD.org Subject: Re: stop_cpus_hard when multiple CPUs are panicking from an NMI References: <50A5F12C.1050902@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@FreeBSD.org, Ryan Stone X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Nov 2012 13:18:24 -0000 on 16/11/2012 14:30 Attilio Rao said the following: > On Fri, Nov 16, 2012 at 7:54 AM, Andriy Gapon wrote: >> on 16/11/2012 00:58 Ryan Stone said the following: >>> At work we have some custom watchdog hardware that sends an NMI upon >>> expiry. We've modified the kernel to panic when it receives the watchdog >>> NMI. I've been trying the "stop_scheduler_on_panic" mode, and I've >>> discovered that when my watchdog expires, the system gets completely >>> wedged. After some digging, I've discovered is that I have multiple CPUs >>> getting the watchdog NMI and trying to panic concurrently. One of the CPUs >>> wins, and the rest spin forever in this code: >>> >>> /* >>> * We don't want multiple CPU's to panic at the same time, so we >>> * use panic_cpu as a simple spinlock. We have to keep checking >>> * panic_cpu if we are spinning in case the panic on the first >>> * CPU is canceled. >>> */ >>> if (panic_cpu != PCPU_GET(cpuid)) >>> while (atomic_cmpset_int(&panic_cpu, NOCPU, >>> PCPU_GET(cpuid)) == 0) >>> while (panic_cpu != NOCPU) >>> ; /* nothing */ >>> >>> The system wedges when stop_cpus_hard() is called, which sends NMIs to all >>> of the other CPUs and waits for them to acknowledge that they are stopped >>> before returning. However the CPU will not deliver an NMI to a CPU that is >>> already handling an NMI, so the other CPUs that got a watchdog NMI and are >>> spinning will never go into the NMI handler and acknowledge that they are >>> stopped. >> >> I thought about this issue and fixed (in my tree) in a different way: >> http://people.freebsd.org/~avg/cpu_stop-race.diff >> >> The need for spinlock_enter in the patch in not entirely clear. >> The main idea is that a CPU which calls cpu_stop and loses a race should >> voluntary enter cpustop_handler. >> I am also not sure about MI-cleanness of this patch. > > It is similar to what I propose but with some differences: > - It is not clean from MI perspective OK. > - I don't think we need to treact it specially, I would just > unconditionally stop all the CPUs entering in the "spinlock zone", > making the patch simpler. I couldn't understand this part. > So I guess you are really fine with the proposal I made? I definitely agree with with the MI cpustop_handler part. I couldn't understand the rest of the proposal. -- Andriy Gapon