From owner-freebsd-fs@freebsd.org Wed Nov 22 15:29:07 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5837DEE371 for ; Wed, 22 Nov 2017 15:29:07 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f41.google.com (mail-lf0-f41.google.com [209.85.215.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4211D7E44B; Wed, 22 Nov 2017 15:29:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f41.google.com with SMTP id d10so7634912lfj.7; Wed, 22 Nov 2017 07:29:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=0ea+++xQTlndIlADwX0h1xxTZKV+6wlE2JejoctCa8k=; b=IzI4wLzTVdx3Fwhfz2Btt2e2Ab+MjadmG/7x9VZdCgRgGQgmvfJ+WFt3sy1jdCeqJt NV0a+RJbDjcjs07Qj/62IFm4owfA+Jh1cPAEAGGpYfskMYgi35tKm5iDVrnSeO6CNruD ZM9GwzQMHk8Mb6x2+QzPuygjK9eS+86QdnFbo3gsdLDTZwUEjnQLGsFyXp1WAPriKxtr gs7ROV6PcCIfdi0hZxG4ag0n5m58ZLp6zHWEzXFsOpKY+PZUQFnfyf7lDmp/ByKW0LXJ PURCdcwtIWs94flq22eFJs1Mft3NTIsAppf9TjHz7mme+xAJoIAHPGovkHS6ZezAoRkE uPAw== X-Gm-Message-State: AJaThX49K/laZrR54r7Jh4LQSUzGTsNnNpnRVDzZhadCiiYN82eycMz9 h9yjrDb6W3XM3qUbRErgpNZ0LY38pWw= X-Google-Smtp-Source: AGs4zMYjal7xoujmft9eM0Ej1jhNKRiq50gn3TmQEtBXk6K8JZpyuR7TOcKAJQsfeK5WiSGVtR5gnQ== X-Received: by 10.25.26.4 with SMTP id a4mr4589824lfa.105.1511364176211; Wed, 22 Nov 2017 07:22:56 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 17sm3704514ljh.73.2017.11.22.07.22.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Nov 2017 07:22:55 -0800 (PST) Subject: Re: zio_done panic in 10.3 To: Youzhong Yang Cc: Shiva Bhanujan , "cem@freebsd.org" , "freebsd-fs@freebsd.org" References: <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D3D1@QLEXC01.Quorum.local> <5021a016-9193-b626-78cf-54ffa3929e22@FreeBSD.org> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D562@QLEXC01.Quorum.local> <3A5A10BE32AC9E45B4A22F89FC90EC0701C367D636@QLEXC01.Quorum.local> <41e2465d-e1b5-33ce-57b5-49bea6087d9a@FreeBSD.org> From: Andriy Gapon Message-ID: <78d712d9-dda3-0411-262e-bb64f9ab46eb@FreeBSD.org> Date: Wed, 22 Nov 2017 17:22:54 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 15:29:07 -0000 On 22/11/2017 16:40, Youzhong Yang wrote: > Hi Andriy, > > This is nice! I am 100% sure it's exactly the same issue I experienced and then > reported to illumos mailing list. In all the crash dumps zio->io_done = > l2arc_read_done, so I thought the crash must be related to L2ARC. Once I set > secondarycache=metadata, the frequency of crash went from one per 2 days down to > one per week. I've been puzzled by what could have caused a zio being destroyed > while there's still child zio. Your explanation definitely makes sense! Oh, I now recall seeing your report: https://illumos.topicbox.com/groups/zfs/Tccd8b4463865899e I remember that it raised my interest, but then I forgot about it and didn't correlate it with the latest reports. > By the way, is there a FreeBSD bug report or an illumos bug number tracking this > issue? I would be more than happy to create one if needed, and also test your > potential fix here in our environment. I am not aware of any existing bug report. It would be great if you could open one [ or two :-) ] If you open an illumos issue, please also add George Wilson as a watcher. I think that George is also interested in fixing this issue and he knows the relevant code better than me. Thank you! > On Tue, Nov 21, 2017 at 3:46 PM, Andriy Gapon > wrote: > > > > > On 21/11/2017 21:30, Shiva Bhanujan wrote: > > it did get compressed to 0.5G - still too big to send via email.  I did send some more debug information by running kgdb on the core file to Andriy, and I'm waiting for any analysis that he might provide. > > Yes, kgdb-over-email turned out to be a far more efficient compression :-) > I already have an analysis based on the information provided by Shiva and by > another user who has the same problem and contacted me privately. > I am discussing possible ways to fix the problem with George Wilson who was very > kind to double-check the analysis, complete it and suggest possible fixes. > > A short version is that dbuf_prefetch and dbuf_prefetch_indirect_done functions > chain new zio-s under the same parent zio (a completion of one child zio may > create another child zio).  They do it using arc_read which can create either a > logical zio in most cases or a vdev zio for a read from a cache device (2arc). > zio_done() has a check for the completion of a parent zio's children but that > check is not completely safe and can be broken by the pattern that dbuf_prefetch > can create.  So, under some specific circumstances the parent zio may complete > and get destroyed while there is a child zio. > > I believe this problem to be rather rare, but there could be configurations and > workloads where it's triggered more often. > The problem does not happen if there are no cache devices. > > > From: Conrad Meyer [cem@freebsd.org ] > > > > Sent: Tuesday, November 21, 2017 9:04 AM > > > > To: Shiva Bhanujan > > > > Cc: Andriy Gapon; freebsd-fs@freebsd.org > > > > Subject: Re: zio_done panic in 10.3 > > > > > > > > > > > > > > > > Have you tried compressing it with e.g. xz or zstd? > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > " > > -- Andriy Gapon