From owner-freebsd-stable@FreeBSD.ORG Tue Feb 20 11:42:25 2007 Return-Path: X-Original-To: stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E83F4170E40 for ; Tue, 20 Feb 2007 11:40:49 +0000 (UTC) (envelope-from rink@thunderstone.rink.nu) Received: from mx1.rink.nu (thunderstone.rink.nu [80.112.228.34]) by mx1.freebsd.org (Postfix) with ESMTP id 168A413C5FE for ; Tue, 20 Feb 2007 11:20:07 +0000 (UTC) (envelope-from rink@thunderstone.rink.nu) Received: from localhost (localhost [127.0.0.1]) by mx1.rink.nu (Postfix) with ESMTP id 58991D4C5E; Tue, 20 Feb 2007 11:59:12 +0100 (CET) X-Virus-Scanned: amavisd-new at rink.nu Received: from mx1.rink.nu ([127.0.0.1]) by localhost (thunderstone.rink.nu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gRAzH1I0961k; Tue, 20 Feb 2007 11:59:03 +0100 (CET) Received: from thunderstone.rink.nu (localhost [127.0.0.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.rink.nu (Postfix) with ESMTP id 5D37FD4C5D; Tue, 20 Feb 2007 11:59:03 +0100 (CET) Received: (from rink@localhost) by thunderstone.rink.nu (8.13.8/8.13.8/Submit) id l1KAx2fD062779; Tue, 20 Feb 2007 11:59:02 +0100 (CET) (envelope-from rink) Date: Tue, 20 Feb 2007 11:59:02 +0100 From: Rink Springer To: stable@FreeBSD.org Message-ID: <20070220105902.GC39393@rink.nu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Mutt/1.5.13 (2006-08-11) Cc: roel@qsp.nl Subject: Deadlock in state 'sysctl lock' X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2007 11:42:26 -0000 Hi people, At work, one of our SpamAssassin/ClamAV filtering machines just entered a deadlock state: FreeBSD/i386 (xxx.qsp.nl) (cuad0) login: root load: 0.00 cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k load: 0.00 cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k load: 0.00 cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k load: 0.00 cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k load: 0.00 cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k After inspection, I believe the following code in kern/kern_sysctl.c:userland_sysctl() is the culprit: SYSCTL_LOCK(); do { req.oldidx =3D 0; req.newidx =3D 0; error =3D sysctl_root(0, name, namelen, &req); } while (error =3D=3D EAGAIN); if (req.lock =3D=3D REQ_WIRED && req.validlen > 0) vsunlock(req.oldptr, req.validlen); SYSCTL_UNLOCK(); Clearly, should sysctl_root() always return EAGAIN, this will cause a serious deadlock condition. It appears this is possible. The only plausible reference to sysctl's returning EGAIN seems to be in=20 kern/kern_proc.c:sysctl_out_proc(). However, this code returns ESRCH if the process couldn't have been found in the fast place, and since the complete handler function will be called by sysctl_root() every iteration, and thus will do a pfind() and return ESRCH if it failed and not EAGAIN as it will later on in the code path. The machine is a 6.0-STABLE SMP machine of 30-Mar-2006. No debugging options are in the kernel as the machine has quite some load. The only console messages were a lot of 'calcru' messages. Any help is very much appreciated. For now, I'd like to propose a change to kern/kern_sysctl.c:userland_sysctl(), to ensure this will never keep looping on EAGAIN states (preferably, it should trigger a panic or at least a KASSERT should such a condition occour). I know this is a bandaid for a problem we don't really quite understand yet, but this may ease debugging later on (especially as it will help us understand where exactly it is going bad) Any comments? It looks to me this deadlock is quite rare (in fact, I've never seen it before), but I believe it is serious enough to be addressed, even with such a bandaid until the real solution is presented by someone who knows the sysctl internals better than I do. Thanks, --=20 Rink P.W. Springer - http://rink.nu "It is such a quiet thing, to fall. But yet a far more terrible thing, to admit it." - Darth Traya