From owner-freebsd-hackers@FreeBSD.ORG Tue Nov 23 06:33:25 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6EDE106566B; Tue, 23 Nov 2010 06:33:25 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id BF6B88FC0A; Tue, 23 Nov 2010 06:33:24 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id IAA12023; Tue, 23 Nov 2010 08:33:23 +0200 (EET) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1PKmRO-000CZf-SY; Tue, 23 Nov 2010 08:33:22 +0200 Message-ID: <4CEB6039.2040700@freebsd.org> Date: Tue, 23 Nov 2010 08:33:29 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101029 Lightning/1.0b2 Thunderbird/3.1.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org, "Robert N. M. Watson" X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit Cc: Subject: dtrace/cyclic deadlock X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 06:33:25 -0000 I think that I've run into the known issue of dtrace/cyclic deadlock. Just would like to run my understanding and ideas by you. The problem is that the cyclic_fire() callback is executed in the interrupt filter context (and thus with interrupts disabled) and it tries to obtain a spin mutex lock in the cyclic code. At the same time other CPU may execute a thread that holds that spin mutex and uses smp_rendezvous_cpus() to perform a synchronous function invocation on the first CPU. So, CPU #1 can not make forward progress because it is spinning on the spin-lock and CPU #2 can not make forward progress because it can not interrupt CPU #1. I think that the problem was introduced during the porting of the code. On (Open)Solaris there are no spin-locks in this code, all data structures are per-CPU and data coherency is ensured by (1) accessing the data only from the CPU to which it belongs; and (2) using some modern-day spl*() equivalent[?] to block interrupts. I think that this is quite similar to what we do for per-CPU caches in UMA and so the same approach should work here. That is, as in (Open)Solaris, the data should be accessed only from the owning CPU and spinlock_enter()/spinlock_exit() should be used to prevent races between non-interrupt code and nested interrupt code. What do you think? Thanks! -- Andriy Gapon