Skip site navigation (1)Skip section navigation (2) 
Date:      Tue, 20 Feb 2007 11:59:02 +0100
From:      Rink Springer <rink@FreeBSD.org>
To:        stable@FreeBSD.org
Cc:        roel@qsp.nl
Subject:   Deadlock in state 'sysctl lock'
Message-ID:  <20070220105902.GC39393@rink.nu>
next in thread | raw e-mail | index | archive | help 
Hi people,

At work, one of our SpamAssassin/ClamAV filtering machines just entered
a deadlock state:

FreeBSD/i386 (xxx.qsp.nl) (cuad0)

login: root
load: 0.00  cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k
load: 0.00  cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k
load: 0.00  cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k
load: 0.00  cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k
load: 0.00  cmd: login 683 [sysctl lock] 0.00u 0.00s 0% 148k

After inspection, I believe the following code in
kern/kern_sysctl.c:userland_sysctl() is the culprit:

        SYSCTL_LOCK();

        do {
                req.oldidx =3D 0;
                req.newidx =3D 0;
                error =3D sysctl_root(0, name, namelen, &req);
        } while (error =3D=3D EAGAIN);

        if (req.lock =3D=3D REQ_WIRED && req.validlen > 0)
                vsunlock(req.oldptr, req.validlen);

        SYSCTL_UNLOCK();

Clearly, should sysctl_root() always return EAGAIN, this will cause a
serious deadlock condition. It appears this is possible.

The only plausible reference to sysctl's returning EGAIN seems to be in=20
kern/kern_proc.c:sysctl_out_proc(). However, this code returns ESRCH
if the process couldn't have been found in the fast place, and since the
complete handler function will be called by sysctl_root() every
iteration, and thus will do a pfind() and return ESRCH if it failed and
not EAGAIN as it will later on in the code path.

The machine is a 6.0-STABLE SMP machine of 30-Mar-2006. No debugging
options are in the kernel as the machine has quite some load. The only
console messages were a lot of 'calcru' messages.

Any help is very much appreciated. For now, I'd like to propose a change
to kern/kern_sysctl.c:userland_sysctl(), to ensure this will never keep
looping on EAGAIN states (preferably, it should trigger a panic or at
least a KASSERT should such a condition occour). I know this is a
bandaid for a problem we don't really quite understand yet, but this may
ease debugging later on (especially as it will help us understand where
exactly it is going bad)

Any comments? It looks to me this deadlock is quite rare (in fact, I've
never seen it before), but I believe it is serious enough to be
addressed, even with such a bandaid until the real solution is presented
by someone who knows the sysctl internals better than I do.

Thanks,

--=20
Rink P.W. Springer                                - http://rink.nu
"It is such a quiet thing, to fall. But yet a far more terrible thing,
 to admit it."                                    - Darth Traya

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070220105902.GC39393>