From owner-freebsd-mips@freebsd.org Thu Nov 12 22:29:08 2015 Return-Path: Delivered-To: freebsd-mips@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C325A2DD5F for ; Thu, 12 Nov 2015 22:29:08 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 084D41D84; Thu, 12 Nov 2015 22:29:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id tACMT1ia008728 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 12 Nov 2015 23:29:02 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: adrian@freebsd.org Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id tACMSuhp010735; Fri, 13 Nov 2015 05:28:56 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: USB-related problem To: Adrian Chadd , benno , Jeff Roberson , Konstantin Belousov References: <56348063.3090508@grosbein.net> <56370E1D.3040801@grosbein.net> <563F5630.2000407@grosbein.net> <563F6F6F.1010807@grosbein.net> <563F91A8.9080702@grosbein.net> <5640C0FD.2040803@grosbein.net> <5640F315.5020303@grosbein.net> <56410214.3070901@grosbein.net> <564231DE.7090308@grosbein.net> <56423D82.5030203@grosbein.net> Cc: "freebsd-mips@freebsd.org" From: Eugene Grosbein X-Enigmail-Draft-Status: N1110 Message-ID: <564512A8.1000006@grosbein.net> Date: Fri, 13 Nov 2015 05:28:56 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.3 required=5.0 tests=BAYES_00, DATE_IN_FUTURE_24_48, LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * 2.0 DATE_IN_FUTURE_24_48 Date: is 24 to 48 hours after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Nov 2015 22:29:08 -0000 On 11.11.2015 01:57, Adrian Chadd wrote: > + benno, jeff, kib > > cool, so now we know where syncer is hanging. nice work! > > That syncer loop that calls it waiting for the list to be empty is > also a bit suss; it looks like it could also get stuck in a loop and > never yield. I've used new state variables spread all over the kernel (plus KTR facility) to track this down and came to a result I cannot explain. First, here is call sequence leading to kernel hang, always the same: sys/kern/vfs_subr.c, sync_vnode() -> VOP_FSYNC(vp, MNT_LAZY, td) -> VOP_FSYNC_APV() in generated vnode_if.c in the kernel compile directory. Here we have KTR_START3(KTR_VOP, ...) that is latest record that DDB shows for "show ktr /v". There should be KTR_STOP3(KTR_VOP, "VOP", ...) recorded after, but it is not. In fact, execution proceeds to vop_fsync(a) and never returns, precisely: vop_fsync(a) -> sys/fs/devfs/devfs_vnops.c, devfs_fsync() -> "return (vop_stdfsync(ap))" - never returns. It reaches sys/kern/vfs_default.c, vop_stdfsync() and hangs exactly at this KASSERT: https://svnweb.freebsd.org/base/head/sys/kern/vfs_default.c?annotate=288451#l683 I've defined another global variable "volatile unsigned vopsfsyncstate", set it to 15 just before this KASSERT and to 126 just after the KASSERT and it always equals to 15 when kernel hangs. I tested kernel compiled with and without WITNESS (and WITHNESS_SKIPSPIN), no changes. I do not understand how and why does it hang here but that's a fact. Here I'm stuck. Eugene Grosbein