From owner-freebsd-fs@freebsd.org Wed Nov 22 14:40:20 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CD31DED32E for ; Wed, 22 Nov 2017 14:40:20 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: from mail-ot0-x22d.google.com (mail-ot0-x22d.google.com [IPv6:2607:f8b0:4003:c0f::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23C707CB09; Wed, 22 Nov 2017 14:40:20 +0000 (UTC) (envelope-from youzhong@gmail.com) Received: by mail-ot0-x22d.google.com with SMTP id u10so13714863otc.12; Wed, 22 Nov 2017 06:40:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=BcjWCBm6ScrsfF3vaixEzLBBAkSFc67mfnhiq2FuJ4w=; b=kAHPh9+1r8ep9QwP8LSxrCjgGhRDgq0zfbgZqDdsk8fihtvbOZWbw+59QAHhF1mOHJ pxzU7IGzB27LQhyMmXAVaCU59bz99mTOAh/IgWacHLC8ERk1on+0crDSfdm8QPIbuyUD boWfHmlIs718mjaf9Y+ktUNXC/G061YG+bAVMFD5hGdPd61tBpDvRwNACtaAY4KpUwm5 LSQobqmVKhMj86VAIOf1kIGJaDnM54zcUwBc09qrsxQXQPmnrahtAe9QtsC/GO39jpl1 6MP+lrBI6V7rk31u/qBl2WzGENvRMfT1za6LAgrfRv4Wjji6tQ9jDee38s7L3LZZbSlG UHxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=BcjWCBm6ScrsfF3vaixEzLBBAkSFc67mfnhiq2FuJ4w=; b=V1eVwXXQ28c0VuaShJ1wPW8OXph2cjSYKPJj8rMhTNQJZU1gKg9VIHyzUZTLtbdAgH IDveUTA36kZ4zrDhjx8t9W+7JOYuJbNmEK7/tzP7KB8frrsd6UJhkc5VBQbzIWFFCPaG h0dLWSu8JEVa7fbP41IVBJaJm0DRtjW3u+BYsrAt3yX/K1j4rKnp2PsM851UlyOYw5aK bfaHdOgWe85AG0Dwu+1g8VVatSFhTypU9inM6DSto+vKXTvQgN4mTeyqsXDzTfFgHRCR z/v4mVXzbs2JfzqjNF+1BlQVAiUO5WEoGQEqEPUDGbq0hLZSWjCgBP/cYokpTk1QvwJG ekmw== X-Gm-Message-State: AJaThX78YJjqd3K2K/8MaLmzc2mFvEG5qEBSi5nMUrnl7LKHCN+qy1YB TXAv+Zd+6NidW2LIAsDCbnEnvn9QaayUhngMNFPzMA== X-Google-Smtp-Source: AGs4zMYpsag5BObbBd6Xlng0lBrq4seVmXvXyvcvshsgbel+WXicB69rGcafn4p4+9D0aUCKF20hSLmxYnUA2XGhTDE= X-Received: by 10.157.89.173 with SMTP id u45mr3075964oth.341.1511361619262; Wed, 22 Nov 2017 06:40:19 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.140.73 with HTTP; Wed, 22 Nov 2017 06:40:18 -0800 (PST) In-Reply-To: <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> From: Youzhong Yang Date: Wed, 22 Nov 2017 09:40:18 -0500 Message-ID: Subject: Re: zio_done panic in 10.3 To: Andriy Gapon Cc: Shiva Bhanujan , "cem@freebsd.org" , "freebsd-fs@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 14:40:20 -0000 Hi Andriy, This is nice! I am 100% sure it's exactly the same issue I experienced and then reported to illumos mailing list. In all the crash dumps zio->io_done = l2arc_read_done, so I thought the crash must be related to L2ARC. Once I set secondarycache=metadata, the frequency of crash went from one per 2 days down to one per week. I've been puzzled by what could have caused a zio being destroyed while there's still child zio. Your explanation definitely makes sense! By the way, is there a FreeBSD bug report or an illumos bug number tracking this issue? I would be more than happy to create one if needed, and also test your potential fix here in our environment. Thanks, --Youzhong On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon wrote: > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > it did get compressed to 0.5G - still too big to send via email. I did > send some more debug information by running kgdb on the core file to > Andriy, and I'm waiting for any analysis that he might provide. > > Yes, kgdb-over-email turned out to be a far more efficient compression :-) > I already have an analysis based on the information provided by Shiva and > by > another user who has the same problem and contacted me privately. > I am discussing possible ways to fix the problem with George Wilson who > was very > kind to double-check the analysis, complete it and suggest possible fixes. > > A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done > functions > chain new zio-s under the same parent zio (a completion of one child zio > may > create another child zio). They do it using arc_read which can create > either a > logical zio in most cases or a vdev zio for a read from a cache device > (2arc). > zio_done() has a check for the completion of a parent zio's children but > that > check is not completely safe and can be broken by the pattern that > dbuf_prefetch > can create. So, under some specific circumstances the parent zio may > complete > and get destroyed while there is a child zio. > > I believe this problem to be rather rare, but there could be > configurations and > workloads where it's triggered more often. > The problem does not happen if there are no cache devices. > > > From: Conrad Meyer [cem@freebsd.org] > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > To: Shiva Bhanujan > > > > Cc: Andriy Gapon; freebsd-fs@freebsd.org > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >