From owner-freebsd-net@FreeBSD.ORG Tue Feb 17 21:26:17 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AE459267 for ; Tue, 17 Feb 2015 21:26:17 +0000 (UTC) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [IPv6:2001:4f8:3:ffe0:406a:0:50:2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8254C76D for ; Tue, 17 Feb 2015 21:26:17 +0000 (UTC) Received: from phabric-backend.isc.freebsd.org (phabric-backend.isc.freebsd.org [127.0.1.5]) by phabric-backend.isc.freebsd.org (8.14.9/8.14.9) with ESMTP id t1HLQHit057021 for ; Tue, 17 Feb 2015 21:26:17 GMT (envelope-from root@phabric-backend.isc.freebsd.org) Received: (from root@localhost) by phabric-backend.isc.freebsd.org (8.14.9/8.14.9/Submit) id t1HLQHQf057020; Tue, 17 Feb 2015 21:26:17 GMT (envelope-from root) Date: Tue, 17 Feb 2015 21:26:17 +0000 To: freebsd-net@freebsd.org From: "rrs (Randall Stewart)" Subject: [Differential] [Commented On] D1711: Changes to the callout code to restore active semantics and also add a test-framework and test to validate thecallout code (and potentially for use by other tests). Message-ID: <6a7bdec57d66259006badc37482dd8f7@localhost.localdomain> X-Priority: 3 Thread-Topic: D1711: Changes to the callout code to restore active semantics and also add a test-framework and test to validate the callout code (and potentially for use by other tests). X-Herald-Rules: none X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: In-Reply-To: References: Thread-Index: Y2JjMTcyODJkYzgxM2NkZDFjY2RhOGRmMTlkIFTjsfk= X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Feb 2015 21:26:17 -0000 rrs added a comment. Hans: I think your wrong here. The caller of callout_cpu_switch() is holding the CC_LOCK(). Now there are only two callers of this function. Either the actual callout code itself (softclock_call_cc()) or the callout_reset_sbt_on(). In the case of callout_reset_sbt_on(). So lets look at these two cases Case 1: soft clock_call_cc() - In this case the callout is finished and is being migrated to a different CPU. This would have happened only because we had the case of doing a callout_reset_sbt_on() and finding the callout executing on the wheel *is* the callout running now and we can't do the migration, so the CPU is set and the PENDING/MIGRATION/ACTVIE flags are also present. The callout lock is held from the perspective of the original CPU, and the code goes through and releases the lock only after setting in the c_lock = CPUBLOCK which will prevent any other running callout_reset_sbt_on() from moving forward, they will be stuck waiting for the c_lock to clear to a CPU number. When the migration is complete the function restores the cpu and the other guy can proceed. It may be moving to yet another CPU, that would be fine too since it now has the lock on the new CPU and moves forward out of callout_reset_sbt_on() 2) callout_reset_sbt_on() - here again we have the current CPU callout lock, and we will only do the switch if the callout is not the current guy executing, if it is we set the flags so <1> above occurs. If its not we move it.. again we set the CPUBLOCK on the callout so no there callout_reset_sbt_on() can get in until after we move it. So there is no way we can have two guys moving at the same time. The CPUBLOCK on the c->c_cpu is used to serialize the two guys. This of course is all noise, since it is *not* related in any shape or form to what Hiren is seeing. If it was you would have seen the panic not at line 530 but above that.. So the question is what is happening.. I am still contemplating it.. and hopefully I can figure out a way to recreate it.. without a vmcore/kernel.debug its pretty hard to decipher these things.. Hiren has been a wonderful help but I am afraid I am about out of question to ask for now.. Its very very strange. REVISION DETAIL https://reviews.freebsd.org/D1711 To: rrs, gnn, rwatson, lstewart, jhb, kostikbel, sbruno, imp, adrian, hselasky Cc: julian, hiren, jhb, kostikbel, emaste, delphij, neel, erj, freebsd-net