From owner-freebsd-fs@freebsd.org Sun Oct 11 04:51:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97A18A11816 for ; Sun, 11 Oct 2015 04:51:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E0211B35 for ; Sun, 11 Oct 2015 04:51:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9B4pJAh069265 for ; Sun, 11 Oct 2015 04:51:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 202607] [panic] Poudriere umounting file systems causes 'solaris assert: avl_is_empty(&dn->dn_dbufs)' panic Date: Sun, 11 Oct 2015 04:51:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: gibbs@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 04:51:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202607 --- Comment #37 from Justin T. Gibbs --- (In reply to Sean Bruno from comment #35) The patch has been accepted for integration upstream in illumos and should land in the next day or two. I'm sure Xin will pull it into FreeBSD as soon as it does. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sun Oct 11 08:54:22 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6514A11EA3 for ; Sun, 11 Oct 2015 08:54:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B32249FF for ; Sun, 11 Oct 2015 08:54:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9B8sMeZ033002 for ; Sun, 11 Oct 2015 08:54:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203588] [ufs] fsck_ufs segfaults during journals after powerloss Date: Sun, 11 Oct 2015 08:54:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 08:54:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203588 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sun Oct 11 09:37:17 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86DDF9D3997 for ; Sun, 11 Oct 2015 09:37:17 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [91.118.6.139]) by mx1.freebsd.org (Postfix) with ESMTP id 0ABFB1BCA for ; Sun, 11 Oct 2015 09:37:16 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [172.20.10.3]) by mail.azedo.at (Postfix) with ESMTP id 5A9E9A83018; Sun, 11 Oct 2015 11:22:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at azedo.at Received: from mail.azedo.at ([172.20.10.3]) by mail.azedo.at (mail.azedo.at [172.20.10.3]) (amavisd-new, port 10024) with ESMTP id farAmkuhiwG9; Sun, 11 Oct 2015 11:22:20 +0200 (CEST) Received: from lynx.local (80-121-99-80.adsl.highway.telekom.at [80.121.99.80]) by mail.azedo.at (Postfix) with ESMTPSA id 448EFA83015; Sun, 11 Oct 2015 11:22:20 +0200 (CEST) Subject: Re: Zfs locking up process To: freebsd-fs@freebsd.org References: From: Michael Ranner X-Enigmail-Draft-Status: N1110 Message-ID: <561A2A4B.4080704@ranner.eu> Date: Sun, 11 Oct 2015 11:22:19 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 09:37:17 -0000 Am 07.10.15 um 17:11 schrieb Rajil Saraswat: > Hello > > I have server running Freenas 9.3 with a few jails. The machine has two new > disks setup in mirror. I have a dataset (/mnt/tank/media) which is shared > in two jails. > > Unfortunately, sometimes when I do a ls in a jail in the shared directory I > see that the process just hangs. > > Today in the jail I did an 'su' and process just hung. On the host if i do > ls /mnt/tank/media it also hangs. > > The su process (pid 77477) is taking up 100% cpu in the jail. It seems that > zfs is holding up the process. Any idea what could be wrong? > > It is a known problem with ZFS and nullfs. I had no problems under FreeBSD 8 witch such a setup, but since FreeBSD 9 it is very unstable to mount_nullfs on ZFS. I experienced the same behaviour with Apache jails and PHP, mostly PHP running with 100% CPU inside the jail. So you cannot "share" the dataset between to jails without the risk of hangs - its a problem inside the VFS. You should set the mountpoint via "zfs set" inside your jail. But you can do this only for one jail. Regards Michael From owner-freebsd-fs@freebsd.org Sun Oct 11 12:21:34 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9908899A860 for ; Sun, 11 Oct 2015 12:21:34 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 369361358 for ; Sun, 11 Oct 2015 12:21:33 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wiclk2 with SMTP id lk2so120030582wic.1 for ; Sun, 11 Oct 2015 05:21:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=EHghyLp7Fx3sC5fTl3zPUjcQvM0GGuDxYJFWks8AzHw=; b=G6RYLToDh5fb9bK+xKjT4ZghTdjk19lzzXxYf4nScmSWlw+gOvPyon+AohzNMApgG2 TsminseAnHtE+wgiF7K7VH/1LbOMtjc6o/I9Ttu5U46+KbsfEmHOx7TXLVwxzW1r6AIV x99vZfKemASORVOfd137UlNqmcF93ouhp5diSLZPxFghgTlZQCtDDbbJxfK10zMMrKQI Fj+sYXrSazP0d33iPoRU/kWEHCKuZvoHpekPJSTYDoamI0tfwTlmrwkjtNqiqPwY4fec J5FrO9YrMc/F72tGET76fjdf1j+JJRGdOAhO6HpWW1EgV1C3LQOdb58d2Jw412n1D2Q/ 0IYA== X-Gm-Message-State: ALoCoQlO+IwlUbGb+ITiQ/Dab65iVw9DqqIVmmV5qkFqukSG9cOhKJpKySX6UIChNAGUFUHY3wqM X-Received: by 10.194.116.197 with SMTP id jy5mr25064957wjb.86.1444566085787; Sun, 11 Oct 2015 05:21:25 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by smtp.gmail.com with ESMTPSA id h6sm6657085wiy.14.2015.10.11.05.21.24 for (version=TLSv1/SSLv3 cipher=OTHER); Sun, 11 Oct 2015 05:21:24 -0700 (PDT) Subject: Re: Zfs locking up process To: freebsd-fs@freebsd.org References: <561A2A4B.4080704@ranner.eu> From: Steven Hartland Message-ID: <561A5445.40603@multiplay.co.uk> Date: Sun, 11 Oct 2015 13:21:25 +0100 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561A2A4B.4080704@ranner.eu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 12:21:34 -0000 On 11/10/2015 10:22, Michael Ranner wrote: > Am 07.10.15 um 17:11 schrieb Rajil Saraswat: >> Hello >> >> I have server running Freenas 9.3 with a few jails. The machine has two new >> disks setup in mirror. I have a dataset (/mnt/tank/media) which is shared >> in two jails. >> >> Unfortunately, sometimes when I do a ls in a jail in the shared directory I >> see that the process just hangs. >> >> Today in the jail I did an 'su' and process just hung. On the host if i do >> ls /mnt/tank/media it also hangs. >> >> The su process (pid 77477) is taking up 100% cpu in the jail. It seems that >> zfs is holding up the process. Any idea what could be wrong? >> >> > It is a known problem with ZFS and nullfs. I had no problems under > FreeBSD 8 witch such a setup, but since FreeBSD 9 it is very unstable to > mount_nullfs on ZFS. I experienced the same behaviour with Apache jails > and PHP, mostly PHP running with 100% CPU inside the jail. I'd have to disagree with this we have hundreds of machines on 10.1 which uses nullfs every day and we've never seen a lockup. Given that do you have more information about this e.g. PR? > So you cannot "share" the dataset between to jails without the risk of > hangs - its a problem inside the VFS. > > You should set the mountpoint via "zfs set" inside your jail. But you > can do this only for one jail. > > Regards > Michael > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Sun Oct 11 13:37:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F38CA0FC9A for ; Sun, 11 Oct 2015 13:37:36 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [91.118.6.139]) by mx1.freebsd.org (Postfix) with ESMTP id D4CBB139 for ; Sun, 11 Oct 2015 13:37:35 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [172.20.10.3]) by mail.azedo.at (Postfix) with ESMTP id 2FE3CA83015; Sun, 11 Oct 2015 15:37:33 +0200 (CEST) X-Virus-Scanned: amavisd-new at azedo.at Received: from mail.azedo.at ([172.20.10.3]) by mail.azedo.at (mail.azedo.at [172.20.10.3]) (amavisd-new, port 10024) with ESMTP id 9aRig--XbHjZ; Sun, 11 Oct 2015 15:37:19 +0200 (CEST) Received: from lynx.local (80-121-99-80.adsl.highway.telekom.at [80.121.99.80]) by mail.azedo.at (Postfix) with ESMTPSA id 6E0C6A83018; Sun, 11 Oct 2015 15:37:19 +0200 (CEST) Subject: Re: Zfs locking up process To: Steven Hartland , freebsd-fs@freebsd.org References: <561A2A4B.4080704@ranner.eu> <561A5445.40603@multiplay.co.uk> From: Michael Ranner Message-ID: <561A660E.1020905@ranner.eu> Date: Sun, 11 Oct 2015 15:37:18 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561A5445.40603@multiplay.co.uk> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 13:37:36 -0000 Am 11.10.15 um 14:21 schrieb Steven Hartland: > > > On 11/10/2015 10:22, Michael Ranner wrote: >> Am 07.10.15 um 17:11 schrieb Rajil Saraswat: >>> Hello >>> >>> I have server running Freenas 9.3 with a few jails. The machine has >>> two new >>> disks setup in mirror. I have a dataset (/mnt/tank/media) which is >>> shared >>> in two jails. >>> >>> Unfortunately, sometimes when I do a ls in a jail in the shared >>> directory I >>> see that the process just hangs. >>> >>> Today in the jail I did an 'su' and process just hung. On the host >>> if i do >>> ls /mnt/tank/media it also hangs. >>> >>> The su process (pid 77477) is taking up 100% cpu in the jail. It >>> seems that >>> zfs is holding up the process. Any idea what could be wrong? >>> >>> >> It is a known problem with ZFS and nullfs. I had no problems under >> FreeBSD 8 witch such a setup, but since FreeBSD 9 it is very unstable to >> mount_nullfs on ZFS. I experienced the same behaviour with Apache jails >> and PHP, mostly PHP running with 100% CPU inside the jail. > I'd have to disagree with this we have hundreds of machines on 10.1 > which uses nullfs every day and we've never seen a lockup. > > Given that do you have more information about this e.g. PR? There are some posts to freebsd-fs in 2014 like this: https://lists.freebsd.org/pipermail/freebsd-fs/2014-November/020482.html And an in depth insight von Andriy Gapon: https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020072.html The problem will become more frequently with heavy snapshot usage on the underlying ZFS datasets. Regards Michael From owner-freebsd-fs@freebsd.org Sun Oct 11 13:49:38 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDE74A1105A for ; Sun, 11 Oct 2015 13:49:38 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 609519EC for ; Sun, 11 Oct 2015 13:49:38 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wiclk2 with SMTP id lk2so121646461wic.1 for ; Sun, 11 Oct 2015 06:49:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type; bh=wD2KHuP/P5yaz7sWnQSB38wHLCdr3JciLZEg8R+WGPc=; b=Ux1+n2NtVZfFOTbPiA8mcnNyAojIyl40j32gLaU6JMMtKK4MCDWoOxK+W0mRLGkq0f QEoEEN11UI9FJHzmuVZF++NH89/LJZA0VGxr1LELo4u6bLJ/kwhhsxrDT1nvW4vMNTRw MMNMyhwb/l259EPsxSR1MXRTuAywkMSv12TSWjJmMa38ZJIsElB5rYLRrjfquV3GkuPq 8v7et/KLdm9tGagSBaCTFyOud1RuZDWqZcddbniNExiHgzSGmh9ovPHAVWJWnrTemMF9 UqMub4T6+y/c8fvu29MysPIgbW/Q9+V7LisMDLbXRN+ust28yRtf5WtFks+9rH6SemCP d+Tw== X-Gm-Message-State: ALoCoQn83I4SROuqsCuPlOoiC7Hx+lVzUKfBENfByxqunaC1BfvxaNVwGVhHifrMh2sxr0XVnVOY X-Received: by 10.180.12.241 with SMTP id b17mr10249465wic.55.1444571370777; Sun, 11 Oct 2015 06:49:30 -0700 (PDT) Received: from [10.10.1.68] (82-69-141-170.dsl.in-addr.zen.co.uk. [82.69.141.170]) by smtp.gmail.com with ESMTPSA id fv5sm6929040wic.7.2015.10.11.06.49.29 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 11 Oct 2015 06:49:29 -0700 (PDT) Subject: Re: Zfs locking up process To: Michael Ranner , freebsd-fs@freebsd.org References: <561A2A4B.4080704@ranner.eu> <561A5445.40603@multiplay.co.uk> <561A660E.1020905@ranner.eu> From: Steven Hartland Message-ID: <561A68EB.5080202@multiplay.co.uk> Date: Sun, 11 Oct 2015 14:49:31 +0100 User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561A660E.1020905@ranner.eu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 13:49:38 -0000 On 11/10/2015 14:37, Michael Ranner wrote: > Am 11.10.15 um 14:21 schrieb Steven Hartland: >> >> >> On 11/10/2015 10:22, Michael Ranner wrote: >>> Am 07.10.15 um 17:11 schrieb Rajil Saraswat: >>>> Hello >>>> >>>> I have server running Freenas 9.3 with a few jails. The machine has >>>> two new >>>> disks setup in mirror. I have a dataset (/mnt/tank/media) which is >>>> shared >>>> in two jails. >>>> >>>> Unfortunately, sometimes when I do a ls in a jail in the shared >>>> directory I >>>> see that the process just hangs. >>>> >>>> Today in the jail I did an 'su' and process just hung. On the host >>>> if i do >>>> ls /mnt/tank/media it also hangs. >>>> >>>> The su process (pid 77477) is taking up 100% cpu in the jail. It >>>> seems that >>>> zfs is holding up the process. Any idea what could be wrong? >>>> >>>> >>> It is a known problem with ZFS and nullfs. I had no problems under >>> FreeBSD 8 witch such a setup, but since FreeBSD 9 it is very >>> unstable to >>> mount_nullfs on ZFS. I experienced the same behaviour with Apache jails >>> and PHP, mostly PHP running with 100% CPU inside the jail. >> I'd have to disagree with this we have hundreds of machines on 10.1 >> which uses nullfs every day and we've never seen a lockup. >> >> Given that do you have more information about this e.g. PR? > There are some posts to freebsd-fs in 2014 like this: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-November/020482.html > > And an in depth insight von Andriy Gapon: > > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020072.html > > The problem will become more frequently with heavy snapshot usage on > the underlying ZFS datasets. There's been lots a movement in ZFS since 9.x so it would be good to confirm what FreeBSD version the original poster is using. I suspect it is 9.3 and if so the first action would be to check on latest 10 release i.e. 10.2 at this time, to ensure its not already been addressed. Looking at the FreeNAS site FreeNAS-10 ALPHA is out which is based off 10.2 so it would be worth testing that, given the announcement post though this should be done with caution. Regards Steve From owner-freebsd-fs@freebsd.org Sun Oct 11 14:00:18 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27FDBA1128C for ; Sun, 11 Oct 2015 14:00:18 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [91.118.6.139]) by mx1.freebsd.org (Postfix) with ESMTP id A48ACD04 for ; Sun, 11 Oct 2015 14:00:17 +0000 (UTC) (envelope-from michael@ranner.eu) Received: from mail.azedo.at (mail.azedo.at [172.20.10.3]) by mail.azedo.at (Postfix) with ESMTP id 9C531A83018; Sun, 11 Oct 2015 16:00:16 +0200 (CEST) X-Virus-Scanned: amavisd-new at azedo.at Received: from mail.azedo.at ([172.20.10.3]) by mail.azedo.at (mail.azedo.at [172.20.10.3]) (amavisd-new, port 10024) with ESMTP id ksX_OmwxQSWM; Sun, 11 Oct 2015 16:00:02 +0200 (CEST) Received: from lynx.local (80-121-99-80.adsl.highway.telekom.at [80.121.99.80]) by mail.azedo.at (Postfix) with ESMTPSA id 9C158A83015; Sun, 11 Oct 2015 16:00:02 +0200 (CEST) Subject: Re: Zfs locking up process To: Steven Hartland , freebsd-fs@freebsd.org References: <561A2A4B.4080704@ranner.eu> <561A5445.40603@multiplay.co.uk> <561A660E.1020905@ranner.eu> <561A68EB.5080202@multiplay.co.uk> From: Michael Ranner X-Enigmail-Draft-Status: N1110 Message-ID: <561A6B62.5020905@ranner.eu> Date: Sun, 11 Oct 2015 16:00:02 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561A68EB.5080202@multiplay.co.uk> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 14:00:18 -0000 Am 11.10.15 um 15:49 schrieb Steven Hartland: > > On 11/10/2015 14:37, Michael Ranner wrote: >> Am 11.10.15 um 14:21 schrieb Steven Hartland: >>> >>> >>> On 11/10/2015 10:22, Michael Ranner wrote: >>>> Am 07.10.15 um 17:11 schrieb Rajil Saraswat: >>>>> Hello >>>>> >>>>> I have server running Freenas 9.3 with a few jails. The machine >>>>> has two new >>>>> disks setup in mirror. I have a dataset (/mnt/tank/media) which is >>>>> shared >>>>> in two jails. >>>>> >>>>> Unfortunately, sometimes when I do a ls in a jail in the shared >>>>> directory I >>>>> see that the process just hangs. >>>>> >>>>> Today in the jail I did an 'su' and process just hung. On the >>>>> host if i do >>>>> ls /mnt/tank/media it also hangs. >>>>> >>>>> The su process (pid 77477) is taking up 100% cpu in the jail. It >>>>> seems that >>>>> zfs is holding up the process. Any idea what could be wrong? >>>>> >>>>> >>>> It is a known problem with ZFS and nullfs. I had no problems under >>>> FreeBSD 8 witch such a setup, but since FreeBSD 9 it is very >>>> unstable to >>>> mount_nullfs on ZFS. I experienced the same behaviour with Apache >>>> jails >>>> and PHP, mostly PHP running with 100% CPU inside the jail. >>> I'd have to disagree with this we have hundreds of machines on 10.1 >>> which uses nullfs every day and we've never seen a lockup. >>> >>> Given that do you have more information about this e.g. PR? >> There are some posts to freebsd-fs in 2014 like this: >> >> https://lists.freebsd.org/pipermail/freebsd-fs/2014-November/020482.html >> >> And an in depth insight von Andriy Gapon: >> >> https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020072.html >> >> The problem will become more frequently with heavy snapshot usage on >> the underlying ZFS datasets. > > There's been lots a movement in ZFS since 9.x so it would be good to > confirm what FreeBSD version the original poster is using. I suspect > it is 9.3 and if so the first action would be to check on latest 10 > release i.e. 10.2 at this time, to ensure its not already been addressed. > > Looking at the FreeNAS site FreeNAS-10 ALPHA is out which is based off > 10.2 so it would be worth testing that, given the announcement post > though this should be done with caution. > > Regards > Steve I have this problem (exact the same stack trace) in all 9.* and 10.1 releases, but the problem does not occur on all servers. Some systems run stable for 100eds of days and specific systems trigger the problem several times in one week. It seems, that some workload will trigger it, especially when snapshot operations are also in progress. Since I removed all nullfs mounts for my web datasets (Apache 2.2/2.4, PHP 5.*) the systems are rock stable. But fyi: I have no problems with nullfs mounts of the ports tree to my jails and building ports. I have/had also no problems on my poudriere build system, running poudriere since early days. -- Mit freundlichen Grüßen Ing. Michael Ranner GSM: +43 676 4155044 Mail: michael@ranner.eu WWW: http://www.azedo.at/ From owner-freebsd-fs@freebsd.org Sun Oct 11 21:00:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D0829B10C8 for ; Sun, 11 Oct 2015 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 40E051BF7 for ; Sun, 11 Oct 2015 21:00:36 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9BL0aX0069061 for ; Sun, 11 Oct 2015 21:00:36 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201510112100.t9BL0aX0069061@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 11 Oct 2015 21:00:36 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 21:00:36 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Oct 12 01:50:51 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F7DD9D2747 for ; Mon, 12 Oct 2015 01:50:51 +0000 (UTC) (envelope-from petehodur@gmail.com) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF72518BE for ; Mon, 12 Oct 2015 01:50:50 +0000 (UTC) (envelope-from petehodur@gmail.com) Received: by lbcao8 with SMTP id ao8so128947779lbc.3 for ; Sun, 11 Oct 2015 18:50:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=Hbm37fLc8Z9Zz20hI9dBgHG2SrygP3uhb+jwJ6is06Q=; b=Iq2CwvCgDTX1BMXuCj9MEDAJqBo/TNgIze9FNsn/uJR5Ub+ebTUdezZBnZ/Zl/ebjr EJZSs+oW73kCybO8I0iS6+za7NXXZRdsUMJDhNNC0KyMwV7w9tKg2D9CM+eDzeX+52QC lAvn8RKCZtViP7/syxtIg9QxLpQ8I6qa30OFK0IimEMw6fY9Ntx5mruAXqo+L7H1ocmW kWHeORQ+L9Mg16yISisXRg0MQPfksNv5n70EbAtnutMmuGtb/fZIWiLBdYjWF1khEGwc Oz4yLn24FgSaO8QhBiCik2nUfopNP2mbBnPoTdtRJWt7C4iOxd6R7/MjiVfOmAoRYEsz tW5w== X-Received: by 10.25.218.205 with SMTP id r196mr7381470lfg.82.1444614648696; Sun, 11 Oct 2015 18:50:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.206.19 with HTTP; Sun, 11 Oct 2015 18:50:28 -0700 (PDT) From: Peter Hodur Date: Mon, 12 Oct 2015 03:50:28 +0200 Message-ID: Subject: FBSD 10.1 with NFSv4 / high nfsd CPU load (> 300%) To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 01:50:51 -0000 Hello, I have an issue with new NFS daemon. I have switch to FreeBSD 10.1 RELENG and tried also move from NFSv3 to NFSv4. With NFSv3 everything looks ok. But when I try mount same FS from client with "-o nfs4", I can see VERY high load on server - more than 300% within couple of seconds. And exports does not works - everything is too slow. NFSD state in "top" is rpcsvc. I have tried it with disabled DRC but with same result. On client is FBSD 9.1. Have someone same issue? Thanks Peter From owner-freebsd-fs@freebsd.org Mon Oct 12 18:39:18 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 22D1EA11F1E for ; Mon, 12 Oct 2015 18:39:18 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E32A814BD for ; Mon, 12 Oct 2015 18:39:17 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id t9CIRmAM016591; Mon, 12 Oct 2015 13:27:48 -0500 (CDT) Date: Mon, 12 Oct 2015 13:27:48 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Quartz cc: FreeBSD FS Subject: Re: A couple ZFS questions In-Reply-To: <56174374.1040609@sneakertech.com> Message-ID: References: <56174374.1040609@sneakertech.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Mon, 12 Oct 2015 13:27:48 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 18:39:18 -0000 On Fri, 9 Oct 2015, Quartz wrote: > Inside a thread on -questions, it was asked if was a bad idea to have a ZFS > array that spanned different controllers (ie; motherboard sata + pci-e sata). > I answered that AFAIK it was ok as long as the speed of the onboard > ports+drives and card+drives aren't drastically different and that the drives > are the same. But it occurred to me that maybe that's not true [anymore]. Can > anyone with more hardware knowledge chime in? Different should not be a problem. Keep in mind that vdev performance is driven by the slowest device in the vdev. If you have multiple vdevs then overall performance is improved by putting similar performance devices in each vdev since zfs will load-share across them, taking current requests, observed performance, and how full the vdev is into account. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Mon Oct 12 23:57:59 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87E09A12FE6 for ; Mon, 12 Oct 2015 23:57:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 3BB989E3 for ; Mon, 12 Oct 2015 23:57:58 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:GMPTBh883lTbrP9uRHKM819IXTAuvvDOBiVQ1KB91OwcTK2v8tzYMVDF4r011RmSDdmdu6gP0rqL+4nbGkU+or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuStKU3578jrDvs7ToICx2xxOFKYtoKxu3qQiD/uI3uqBFbpgL9x3Sv3FTcP5Xz247bXianhL7+9vitMU7q3cYk7sb+sVBSaT3ebgjBfwdVWx+cjN92Mq+jRTfQBHHxnwQT39exgJFHwXF6x3nRL/+tyL7sqx23yzMbuPsSrVhYzWp7O9OQRTrjCoCf2oj9Wjcich9iYpGpx28qhhnw8jfadfGZ7JFYqrBcIZCFiJ6VcFLWnkEW9vkYg== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2BsAgB3SBxW/61jaINdg3puBr1/AQ2BWhcKgnKCCjVKAoFsFAEBAQEBAQEBgQmCH4IHAQEBAwEBAQEgBCcgCwULAgEIDgoCAg0ZAgInAQkmAgQIBwQBHASIBQgNrEaTVwEBAQEBAQEDAQEBAQEBARcEgSKFUYR+hDsBAQUXNAeCaYFFBZYUhRmFGYRAljyDbQIfAQFCghABHYFwIjMHhSo6gQYBAQE X-IronPort-AV: E=Sophos;i="5.17,675,1437451200"; d="scan'208";a="244135823" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 12 Oct 2015 19:57:57 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 5FC4F15F55D; Mon, 12 Oct 2015 19:57:57 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id iBdpsRLSr6Hv; Mon, 12 Oct 2015 19:57:56 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id AE05215F565; Mon, 12 Oct 2015 19:57:56 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id xAD0TRZZyoxI; Mon, 12 Oct 2015 19:57:56 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 865ED15F55D; Mon, 12 Oct 2015 19:57:56 -0400 (EDT) Date: Mon, 12 Oct 2015 19:57:56 -0400 (EDT) From: Rick Macklem To: Peter Hodur Cc: freebsd-fs@freebsd.org Message-ID: <1527559687.33304726.1444694276368.JavaMail.zimbra@uoguelph.ca> In-Reply-To: References: Subject: Re: FBSD 10.1 with NFSv4 / high nfsd CPU load (> 300%) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.11] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: FBSD 10.1 with NFSv4 / high nfsd CPU load (> 300%) Thread-Index: D3MLMO8U0k2Wq4TxcxUa3N03AwoJnA== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 23:57:59 -0000 Peter Hodur wrote: > Hello, > > I have an issue with new NFS daemon. I have switch to FreeBSD 10.1 RELENG > and tried also move from NFSv3 to NFSv4. With NFSv3 everything looks ok. > But when I try mount same FS from client with "-o nfs4", I can see VERY > high load on server - more than 300% within couple of seconds. And exports > does not works - everything is too slow. NFSD state in "top" is rpcsvc. > > I have tried it with disabled DRC but with same result. > So, are the mounts working but just slow or are they hung trying to mount? In general, NFSv4 is a very different protocol than NFSv3 and not really a replacement for NFSv3 imho. It does offer features that some need, like better byte range locking, ACLs, user names instead of uids... However, if you don't need these features, there really isn't a good reason to use it, again imho. If you want to try and isolate what is going on: - I'd start with "nfsstat -e -s" on the server: --> Look for which RPCs are being done a lot and how many Opens/ClientIDs allocated on the server. For example, if the server is seeing a large number of Open/Close RPCs, that could just be unavoidable overhead. (There is something called delegations, which can be used to avoid repeated Opens of the same files, but if your clients are opening large #s of different files, delegations won't help.) If there are a large # of Opens allocated (near the bottom of the output, after the RPC/Op counts), then some of the overhead can be avoided by making the hash tables much larger. (There are tunables for these, but I can't remember if they are in 10.1 or only 10.2?) - If the "nfsstat -e -s" on the server doesn't seem to give you a hint w.r.t. what is causing the overhead, then I'd use tcpdump to capture packets for a short period of time and then look at them in wireshark. (Unlike tcpdump, wireshark knows NFSv4.) Hopefully you'll see a large number of the same RPCs, possibly failing, which will give you some insight. - I'd also do a "ps axHl" on the server and see if any of the other daemons, like nfsuserd seem to be busy. (A busy nfsused would indicate a high miss rate on the cache of uid<->username mappings in the kernel.) If you post your "nfsstat -e -s", then I might be able to spot something that won't be obvious to you. On the other hand, if your mounts aren't succeeding (or you gets lots of files owned by "nobody" when you "ls -l" the mount point), then something is busted in your setup. You should check /var/log/messages to see if there are problems with your /etc/exports. You can also post your /etc/exports. (For example, it should look just the same as what you use for NFSv3, but with the addition of a "V4: ..." line. The "V4: ..." line does not replace any of the other lines, which are still required for NFSv4.) Don't know if any of this will help, rick > On client is FBSD 9.1. > > > Have someone same issue? > > > Thanks > > Peter > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Tue Oct 13 09:17:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 010C9A1134B for ; Tue, 13 Oct 2015 09:17:48 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (imap.slu.se [77.235.224.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "webmail.slu.se", Issuer "TERENA SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88724ED0 for ; Tue, 13 Oct 2015 09:17:46 +0000 (UTC) (envelope-from karli.sjoberg@slu.se) Received: from exch2-4.slu.se (77.235.224.124) by exch2-4.slu.se (77.235.224.124) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Tue, 13 Oct 2015 11:02:33 +0200 Received: from exch2-4.slu.se ([fe80::bc24:95e8:1e1e:f3ae]) by exch2-4.slu.se ([fe80::bc24:95e8:1e1e:f3ae%22]) with mapi id 15.00.1104.000; Tue, 13 Oct 2015 11:02:33 +0200 From: =?utf-8?B?S2FybGkgU2rDtmJlcmc=?= To: "freebsd-fs@freebsd.org" Subject: 'zfs send/recv' during resilver/scrub Thread-Topic: 'zfs send/recv' during resilver/scrub Thread-Index: AQHRBZXlywmiCAxx9U+oQigRKBAukg== Date: Tue, 13 Oct 2015 09:02:33 +0000 Message-ID: <1444726953.3939.12.camel@data-b104.adm.slu.se> Accept-Language: sv-SE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [77.235.228.63] Content-Type: text/plain; charset="utf-8" Content-ID: <4090178181CD5D4588AFB43F46562AE7@ad.slu.se> Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 09:17:48 -0000 SGV5IGFsbCENCg0KSSBwb3N0ZWQgdGhpcyBxdWVzdGlvbiBvbiB0aGUgZm9ydW1zIHllc3RlcmRh eSBidXQgc2VuZGluZyBpdCBoZXJlIGFzDQp3ZWxsIHRvIG1heWJlIHJlYWNoIGEgZGlmZmVyZW50 IGF1ZGllbmNlOg0KaHR0cHM6Ly9mb3J1bXMuZnJlZWJzZC5vcmcvdGhyZWFkcy96ZnMtc2VuZC1y ZWN2LWR1cmluZy1yZXNpbHZlci1zY3J1Yi41MzU1OS8NCg0KQSBsb25nIHRpbWUgYWdvIG5vdywg d2hlbiBJIHN0YXJ0ZWQgd3JpdGluZyBteSBiYWNrdXAgc2NyaXB0IGNhbGxlZA0KJ3JlcGxpY2F0 ZScsIEkgaGFkIHRoaXMgbm90aW9uIHRoYXQgeW91IHNob3VsZG7CtHQgZG8gYW55dGhpbmcgd2hp bGUNCnRoZXJlIHdhcyBhIHJlc2lsdmVyL3NjcnViIGluIHByb2dyZXNzLCBiYXNlZCBvbiBzb21l dGhpbmcgSSByZWFkIHRoYXQNCndhcyBldmVuIG9sZGVyIGluZm9ybWF0aW9uLiBTbyB0aGUgc2Ny aXB0IGhhbHRzIGlmIHRoZXJlIGlzIGFuIG9uZ29pbmcNCnJlc2lsdmVyIG9yIHNjcnViIGluIGVp dGhlciBzb3VyY2Ugb3IgZGVzdGluYXRpb24gc3lzdGVtLg0KDQpOb3cgd2hlbiBzZWFyY2hpbmcg Zm9yIGZhY3RzIHJlZ2FyZGluZyB0aGlzIG1hdHRlciwgSSBjb21lIHVwIGVtcHR5Li4uDQpTbyB0 aGlua2luZyBhYm91dCBpdCwgaXMgaXQgcmVhbGx5IG5lY2Vzc2FyeT8gSXQgbWlnaHQganVzdCBi ZQ0Kc3VwZXJzdGl0aW9uLCBiYXNlZCBvbiBvbGQgRlVEIHRoYXQgScK0dmUganVzdCBiZWVuIHRh a2luZyBmb3IgZ3JhbnRlZC4NCk9yIGhhdmUgSSBiZWVuIGltYWdpbmluZyB0aGlzLCBhbmQgaXQg aGFzIG5ldmVyIGJlZW4gYW4gaXNzdWUgaW4gdGhlDQpmaXJzdCBwbGFjZT8NCg0KVGhpbmcgaXMg dGhhdCB3aXRoIGJpZ2dlciwgaGVhdmlseSBsb2FkZWQgc3lzdGVtcywgc2NydWJzIGFuZCByZXNp bHZlcnMNCmNhbiB0YWtlIGRheXMsIGV2ZW4gd2Vla3MsIGVzcGVjaWFsbHkgaWYgaXTCtHMgdXAg YXQsIHNheSA4MC05MCUNCmNhcGFjaXR5LiBBbmQgdG8gbm90IGhhdmUgYW55IHJlY2VudCBiYWNr dXBzIGZvciB0aGF0IGxvbmcgcGVyaW9kIG9mDQp0aW1lIGp1c3QgaXNuwrR0IHJpZ2h0LiBTbyBl dmVuIGlmIGl0IGNvdWxkIGNhdXNlIHByb2JsZW1zIHJlcGxpY2F0aW5nDQp3aGlsZSB0aGVyZSBp cyBhIHJlc2lsdmVyIG9yIHNjcnViIHJ1bm5pbmcsIGl0wrRzIHN0aWxsIHByZWZlcmFibGUgdG8g bm90DQpoYXZpbmcgYW55IGJhY2t1cHMuDQoNCkhvdyBhcmUgb3RoZXJzIGhhbmRsaW5nIGl0LCBs aWtlIEZyZWVOQVMgZS5nLiBEbyB0aGV5IHJlcGxpY2F0ZSB3aGlsZQ0Kc2NydWJiaW5nPw0KDQpX aGF0IGFyZSB5b3VyIHRob3VnaHRzIGFib3V0IHRoaXM/DQoNCkJlc3QgUmVnYXJkcw0KS2FybGkg U2rDtmJlcmcNCg== From owner-freebsd-fs@freebsd.org Tue Oct 13 10:34:47 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC615A1207A for ; Tue, 13 Oct 2015 10:34:46 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay05.ispgateway.de (smtprelay05.ispgateway.de [80.67.31.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7EA99896 for ; Tue, 13 Oct 2015 10:34:45 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.182.8] (helo=fabiankeil.de) by smtprelay05.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1Zlwkl-0005uM-So for freebsd-fs@FreeBSD.org; Tue, 13 Oct 2015 12:24:20 +0200 Date: Tue, 13 Oct 2015 12:24:18 +0200 From: Fabian Keil To: Subject: Re: panic: solaris assert: rt->rt_space == 0 (0xe000 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c, line: 153 Message-ID: <6fe311f0.24af1e99@fabiankeil.de> In-Reply-To: <04f3092d.6fdfad8a@fabiankeil.de> References: <04f3092d.6fdfad8a@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/k=v7MkUpi.YtOIU9a2z6A5C"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 10:34:47 -0000 --Sig_/k=v7MkUpi.YtOIU9a2z6A5C Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Fabian Keil wrote: > Using an 11.0-CURRENT based on r276255 I just got a panic > after trying to export a certain ZFS pool: >=20 > (kgdb) where > #0 doadump (textdump=3D0) at pcpu.h:219 > #1 0xffffffff80313e8e in db_dump (dummy=3D, dummy2= =3D0, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:533 > #2 0xffffffff8031396d in db_command (cmd_table=3D0x0) at /usr/src/sys/dd= b/db_command.c:440 > #3 0xffffffff803136e4 in db_command_loop () at /usr/src/sys/ddb/db_comma= nd.c:493 > #4 0xffffffff803161f0 in db_trap (type=3D, code=3D0= ) at /usr/src/sys/ddb/db_main.c:251 > #5 0xffffffff805f63c1 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 > #6 0xffffffff80878717 in trap (frame=3D0xfffffe0094a62540) at /usr/src/s= ys/amd64/amd64/trap.c:542 > #7 0xffffffff8085c472 in calltrap () at /usr/src/sys/amd64/amd64/excepti= on.S:235 > #8 0xffffffff805f5abe in kdb_enter (why=3D0xffffffff80995a6d "panic", ms= g=3D) at cpufunc.h:63 > #9 0xffffffff805b3b81 in panic (fmt=3D) at /usr/src= /sys/kern/kern_shutdown.c:739 > #10 0xffffffff81bdd22f in assfail3 (a=3D, lv=3D, op=3D, rv=3D, f= =3D, l=3D) > at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91 > #11 0xffffffff8194afc4 in range_tree_destroy (rt=3D0xfffff80011586000) at= /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:153 > #12 0xffffffff819488bc in metaslab_fini (msp=3D0xfffff800611a9800) at /us= r/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1398 > #13 0xffffffff81965841 in vdev_free (vd=3D0xfffff8000696d800) at /usr/src= /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:994 > #14 0xffffffff819657e1 in vdev_free (vd=3D0xfffff80040532000) at /usr/src= /sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:683 > #15 0xffffffff81953948 in spa_unload (spa=3D0xfffff800106af000) at /usr/s= rc/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:1314 > #16 0xffffffff81957a58 in spa_export_common (pool=3D= , new_state=3D1, oldconfig=3D0x0, force=3D, hardforce= =3D0) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:4540 > #17 0xffffffff81957b08 in spa_export (pool=3D0x0, oldconfig=3D0xfffffe009= 4a624f0, force=3D128, hardforce=3D50) at /usr/src/sys/cddl/contrib/opensola= ris/uts/common/fs/zfs/spa.c:4574 > #18 0xffffffff8199ed50 in zfs_ioc_pool_export (zc=3D0xfffffe0006fbf000) a= t /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:1618 > #19 0xffffffff8199c2aa in zfsdev_ioctl (dev=3D, zcmd= =3D, arg=3D, flag=3D, td=3D) > at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.= c:6198 > #20 0xffffffff8047d32b in devfs_ioctl_f (fp=3D0xfffff8002adcb280, com=3D3= 222821379, data=3D0xfffffe0094a62a20, cred=3D, td=3D0x= fffff80056f15000) at /usr/src/sys/fs/devfs/devfs_vnops.c:775 Just for the archives, the panic is fixed by: https://www.illumos.org/issues/6292 Fabian --Sig_/k=v7MkUpi.YtOIU9a2z6A5C Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlYc28oACgkQBYqIVf93VJ3kZgCglSnqxCvZZtB+FcF0IPmCvl3V jyIAn3xXhb7Z/O0AFM75dzuRad609uHx =Kmmm -----END PGP SIGNATURE----- --Sig_/k=v7MkUpi.YtOIU9a2z6A5C-- From owner-freebsd-fs@freebsd.org Wed Oct 14 17:11:57 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14DBBA151FB for ; Wed, 14 Oct 2015 17:11:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 016011634 for ; Wed, 14 Oct 2015 17:11:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9EHBuPj025689 for ; Wed, 14 Oct 2015 17:11:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 202607] [panic] Poudriere umounting file systems causes 'solaris assert: avl_is_empty(&dn->dn_dbufs)' panic Date: Wed, 14 Oct 2015 17:11:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: gibbs@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2015 17:11:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=202607 Justin T. Gibbs changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Progress |Closed Resolution|--- |FIXED --- Comment #38 from Justin T. Gibbs --- https://svnweb.freebsd.org/changeset/base/289309 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Thu Oct 15 14:33:52 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E8DEA155BF for ; Thu, 15 Oct 2015 14:33:52 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E357C13B9 for ; Thu, 15 Oct 2015 14:33:51 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id E81051C1195 for ; Thu, 15 Oct 2015 09:27:32 -0500 (CDT) Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) To: freebsd-fs@freebsd.org References: <55BB443E.8040801@denninger.net> <55CF7926.1030901@denninger.net> <55DF7191.2080409@denninger.net> <55DF76AA.3040103@denninger.net> From: Karl Denninger Message-ID: <561FB7D0.4080107@denninger.net> Date: Thu, 15 Oct 2015 09:27:28 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <55DF76AA.3040103@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020606080808060507010408" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 14:33:52 -0000 This is a cryptographically signed message in MIME format. --------------ms020606080808060507010408 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Got another one of these this morning, after a long absence... Same symptom -- happened during a backup (send/receive) and it's in the same place -- when the snapshot is unmounted. I have a clean dump and this is against a quite-recent checkout, so the most-currently-rolled forward ZFS changes are in this kernel. Fatal trap 12: page fault while in kernel mode cpuid =3D 14; apic id =3D 34 fault virtual address =3D 0x378 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80940ae0 stack pointer =3D 0x28:0xfffffe066796c680 frame pointer =3D 0x28:0xfffffe066796c700 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 81187 (zfs) trap number =3D 12 panic: page fault cpuid =3D 14 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe066796c160 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe066796c210 vpanic() at vpanic+0x126/frame 0xfffffe066796c250 panic() at panic+0x43/frame 0xfffffe066796c2b0 trap_fatal() at trap_fatal+0x36b/frame 0xfffffe066796c310 trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe066796c3b0 trap() at trap+0x47a/frame 0xfffffe066796c5c0 calltrap() at calltrap+0x8/frame 0xfffffe066796c5c0 --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe066796c680, rbp= =3D 0xfffffe 066796c700 --- __mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe066796c700 dounmount() at dounmount+0x58/frame 0xfffffe066796c780 zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe066796c7a0 dsl_dataset_user_release_impl() at dsl_dataset_user_release_impl+0x103/frame 0xf ffffe066796c920 dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x5c/frame 0xfffffe066796c940 zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066796c970 zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe066796c990 devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe066796c9b0 devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe066796c9e0 _fdrop() at _fdrop+0x29/frame 0xfffffe066796ca00 closef() at closef+0x21e/frame 0xfffffe066796ca90 closefp() at closefp+0x98/frame 0xfffffe066796cad0 amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe066796cbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe066796cbf0 --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D 0x7fffffffd728, rbp =3D 0x7fffffffd740 --- Uptime: 3d15h37m26s On 8/27/2015 15:44, Karl Denninger wrote: > No, but that does sound like it might be involved..... > > And yeah, this did start when I moved the root pool to a mirrored pair > of Intel 530s off a pair of spinning-rust WD RE4s.... (The 530s are dar= n > nice performance-wise, reasonably inexpensive and thus very suitable fo= r > a root filesystem drive and they also pass the "pull the power cord" > test, incidentally.) > > You may be onto something -- I'll try shutting it off, but due to the > fact that I can't make this happen and it's a "every week or two" panic= , > but ALWAYS when the zfs send | zfs recv is running AND it's always on > the same filesystem it will be a fair while before I know if it's fixed= > (like over a month, given the usual pattern here, as that would be 4 > "average" periods without a panic)..... > > I also wonder if I could tune this out with some of the other TRIM > parameters instead of losing it entirely. > > vfs.zfs.trim.max_interval: 1 > vfs.zfs.trim.timeout: 30 > vfs.zfs.trim.txg_delay: 32 > vfs.zfs.trim.enabled: 1 > vfs.zfs.vdev.trim_max_pending: 10000 > vfs.zfs.vdev.trim_max_active: 64 > vfs.zfs.vdev.trim_min_active: 1 > > That it's panic'ing on a mtx_lock_sleep might point this way.... the > trace shows it coming from a zfs_onexit_destroy, which ends up calling > zfs_unmount_snap() and then it blows in dounmount() while executing > mtx_lock_sleep(). > > I do wonder if I'm begging for new and innovative performance issues if= > I run with TRIM off for an extended period of time, however..... :-) > > > On 8/27/2015 15:30, Sean Chittenden wrote: >> Have you tried disabling TRIM? We recently ran in to an issue where a= `zfs delete` on a large dataset caused the host to panic because TRIM wa= s tripping over the ZFS deadman timer. Disabling TRIM worked as valid w= orkaround for us. ? You mentioned a recent move to SSDs, so this can ha= ppen, esp after the drive has experienced a little bit of actual work. ?= -sc >> >> >> -- >> Sean Chittenden >> sean@chittenden.org >> >> >>> On Aug 27, 2015, at 13:22, Karl Denninger wrote:= >>> >>> On 8/15/2015 12:38, Karl Denninger wrote: >>>> Update: >>>> >>>> This /appears /to be related to attempting to send or receive a >>>> /cloned /snapshot. >>>> >>>> I use /beadm /to manage boot environments and the crashes have all >>>> come while send/recv-ing the root pool, which is the one where these= >>>> clones get created. It is /not /consistent within a given snapshot >>>> when it crashes and a second attempt (which does a "recovery" >>>> send/receive) succeeds every time -- I've yet to have it panic twice= >>>> sequentially. >>>> >>>> I surmise that the problem comes about when a file in the cloned >>>> snapshot is modified, but this is a guess at this point. >>>> >>>> I'm going to try to force replication of the problem on my test syst= em. >>>> >>>> On 7/31/2015 04:47, Karl Denninger wrote: >>>>> I have an automated script that runs zfs send/recv copies to bring = a >>>>> backup data set into congruence with the running copies nightly. T= he >>>>> source has automated snapshots running on a fairly frequent basis >>>>> through zfs-auto-snapshot. >>>>> >>>>> Recently I have started having a panic show up about once a week du= ring >>>>> the backup run, but it's inconsistent. It is in the same place, bu= t I >>>>> cannot force it to repeat. >>>>> >>>>> The trap itself is a page fault in kernel mode in the zfs code at >>>>> zfs_unmount_snap(); here's the traceback from the kvm (sorry for th= e >>>>> image link but I don't have a better option right now.) >>>>> >>>>> I'll try to get a dump, this is a production machine with encrypted= swap >>>>> so it's not normally turned on. >>>>> >>>>> Note that the pool that appears to be involved (the backup pool) ha= s >>>>> passed a scrub and thus I would assume the on-disk structure is ok.= =2E... >>>>> but that might be an unfair assumption. It is always occurring in = the >>>>> same dataset although there are a half-dozen that are sync'd -- if = this >>>>> one (the first one) successfully completes during the run then all = the >>>>> rest will as well (that is, whenever I restart the process it has a= lways >>>>> failed here.) The source pool is also clean and passes a scrub. >>>>> >>>>> traceback is at http://www.denninger.net/kvmimage.png; apologies fo= r the >>>>> image traceback but this is coming from a remote KVM. >>>>> >>>>> I first saw this on 10.1-STABLE and it is still happening on FreeBS= D >>>>> 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to se= e if >>>>> the problem was something that had been addressed. >>>>> >>>>> >>>> --=20 >>>> Karl Denninger >>>> karl@denninger.net >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ >>> Second update: I have now taken another panic on 10.2-Stable, same de= al, >>> but without any cloned snapshots in the source image. I had thought t= hat >>> removing cloned snapshots might eliminate the issue; that is now out = the >>> window. >>> >>> It ONLY happens on this one filesystem (the root one, incidentally) >>> which is fairly-recently created as I moved this machine from spinnin= g >>> rust to SSDs for the OS and root pool -- and only when it is being >>> backed up by using zfs send | zfs recv (with the receive going to a >>> different pool in the same machine.) I have yet to be able to provok= e >>> it when using zfs send to copy to a different machine on the same LAN= , >>> but given that it is not able to be reproduced on demand I can't be >>> certain it's timing related (e.g. performance between the two pools i= n >>> question) or just that I haven't hit the unlucky combination. >>> >>> This looks like some sort of race condition and I will continue to se= e >>> if I can craft a case to make it occur "on demand" >>> >>> --=20 >>> Karl Denninger >>> karl@denninger.net >>> /The Market Ticker/ >>> /[S/MIME encrypted email preferred]/ >> >> %SPAMBLOCK-SYS: Matched [+Sean Chittenden ], mess= age ok >> --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020606080808060507010408 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEwMTUxNDI3MjhaME8GCSqGSIb3DQEJBDFCBEBu QCAmBvB+c3K7ThkDiy/LmOi12lW4s8uaVSDO57SbEaaAFfpiDF+Zwt9p5OPwMxlq5vMd18zw F3zR++kZ1OENMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAHE5bjQS9 dKEsZ6t8TBlQ4s9DeIOEtyea+erDtC/yPslXWz1iw47AOsoZN5XyaPKD09EqjIrBKtK7JRN5 dWoFP9m+G2UVzYGSSdwLBsO2MVJDQ9TTRWpJCIiPz+50MEByOUtViynUT3wbmw96wNIw+z8X VItfBdxG3UXo8rqNzYRokjmhXMBWwuSaWDLMFQlEPDUtLV1AfH4mo0/oDutr7pEuT+xXau1u 5Ojie0X1D9vfB2rkaC+dkShiQue5e6K/Y2OWjD/12zBcJ/prbiAKvcbEs8ZGEr3TVKa/67Wg KgpP3xPNEZPi7gNkdqLdgnDQhGvP5HSyAuJRWCnZuMsfBRflWT7N/xylf4pXsbpSRxeNo6Xz GWZmXaMK/vb/fhkXSXhwqYLzEQr7vL79Gayz17ZXBwxnPjEpfmSWCBl/JjSkQSbTLwEz66Kj uLxWFn/5j6W9Vx3O5CPD3MRngz3yj+E7sL+81tI73oRd1uU6GgWZ8OFFnZ9XDExg3H/FDyW0 JsYhaeXdLsd4/EqLv0U2K1C9POGJQK2WYIND2Y9USGnlA+mxyDBqPRthc3aXt4McPnI+EA7E 6pSxkOiv6q0YviYVnvZaCgHLxJwpyA0g5wlCImfDar11u04hZy+iPBhKABZe3S8XiHN+PPaQ nO+qgVbEZgB5Vrpl9CNRMawzy8cAAAAAAAA= --------------ms020606080808060507010408-- From owner-freebsd-fs@freebsd.org Thu Oct 15 14:56:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE4C7A15AD8 for ; Thu, 15 Oct 2015 14:56:27 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97D40809 for ; Thu, 15 Oct 2015 14:56:26 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id E3EA11C13C1 for ; Thu, 15 Oct 2015 09:56:24 -0500 (CDT) Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) To: freebsd-fs@freebsd.org References: <55BB443E.8040801@denninger.net> <55CF7926.1030901@denninger.net> <55DF7191.2080409@denninger.net> <55DF76AA.3040103@denninger.net> <561FB7D0.4080107@denninger.net> From: Karl Denninger X-Enigmail-Draft-Status: N1110 Message-ID: <561FBE95.6010107@denninger.net> Date: Thu, 15 Oct 2015 09:56:21 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561FB7D0.4080107@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms070409030504010001090508" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 14:56:28 -0000 This is a cryptographically signed message in MIME format. --------------ms070409030504010001090508 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable More data on this -- it is happening during this sequence (snippet from my backup scripts w/my comments) "$i" is the ZFS filesystem that is being backed up; in each case the crashes occur _*only*_ when the root (mounted) filesystem is being copied; it has never happened on any of the other datasets. zfs rename -r $i@zfs-base $i@zfs-old (rename the previous backup base to the "start" point) zfs snapshot -r $i@zfs-base (take a snapshot to checkpoint to on the back= up) zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP result=3D$? if [ $result -eq 0 ]; then _*zfs destroy -r $i@zfs-old*_ -->> *PANIC OCCURS HERE* zfs destroy -r $BACKUP/$FILESYS@zfs-old else echo "STOP - - ERROR RUNNING COPY on $i!!!!" exit 1 fi On 10/15/2015 09:27, Karl Denninger wrote: > Got another one of these this morning, after a long absence... > > Same symptom -- happened during a backup (send/receive) and it's in > the same place -- when the snapshot is unmounted and destroyed. I > have a clean dump and this is against a quite-recent checkout, so the > most-currently-rolled forward ZFS changes are in this kernel. > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 14; apic id =3D 34 > fault virtual address =3D 0x378 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80940ae0 > stack pointer =3D 0x28:0xfffffe066796c680 > frame pointer =3D 0x28:0xfffffe066796c700 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 81187 (zfs) > trap number =3D 12 > panic: page fault > cpuid =3D 14 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe066796c160 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe066796c210 > vpanic() at vpanic+0x126/frame 0xfffffe066796c250 > panic() at panic+0x43/frame 0xfffffe066796c2b0 > trap_fatal() at trap_fatal+0x36b/frame 0xfffffe066796c310 > trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe066796c3b0 > trap() at trap+0x47a/frame 0xfffffe066796c5c0 > calltrap() at calltrap+0x8/frame 0xfffffe066796c5c0 > --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe066796c680, r= bp > =3D 0xfffffe > 066796c700 --- > *__mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe066796c700*= > dounmount() at dounmount+0x58/frame 0xfffffe066796c780 > zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe066796c7a0 > dsl_dataset_user_release_impl() at > dsl_dataset_user_release_impl+0x103/frame 0xf > ffffe066796c920 > dsl_dataset_user_release_onexit() at > dsl_dataset_user_release_onexit+0x5c/frame > 0xfffffe066796c940 > zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066796c97= 0 > zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe066796c990 > devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe066796c9b0 > devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe066796c9e0 > _fdrop() at _fdrop+0x29/frame 0xfffffe066796ca00 > closef() at closef+0x21e/frame 0xfffffe066796ca90 > closefp() at closefp+0x98/frame 0xfffffe066796cad0 > amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe066796cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe066796cbf0 > --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= > 0x7fffffffd728, rbp =3D 0x7fffffffd740 --- > Uptime: 3d15h37m26s > > > On 8/27/2015 15:44, Karl Denninger wrote: >> No, but that does sound like it might be involved..... >> >> And yeah, this did start when I moved the root pool to a mirrored pair= >> of Intel 530s off a pair of spinning-rust WD RE4s.... (The 530s are da= rn >> nice performance-wise, reasonably inexpensive and thus very suitable f= or >> a root filesystem drive and they also pass the "pull the power cord" >> test, incidentally.) >> >> You may be onto something -- I'll try shutting it off, but due to the >> fact that I can't make this happen and it's a "every week or two" pani= c, >> but ALWAYS when the zfs send | zfs recv is running AND it's always on >> the same filesystem it will be a fair while before I know if it's fixe= d >> (like over a month, given the usual pattern here, as that would be 4 >> "average" periods without a panic)..... >> >> I also wonder if I could tune this out with some of the other TRIM >> parameters instead of losing it entirely. >> >> vfs.zfs.trim.max_interval: 1 >> vfs.zfs.trim.timeout: 30 >> vfs.zfs.trim.txg_delay: 32 >> vfs.zfs.trim.enabled: 1 >> vfs.zfs.vdev.trim_max_pending: 10000 >> vfs.zfs.vdev.trim_max_active: 64 >> vfs.zfs.vdev.trim_min_active: 1 >> >> That it's panic'ing on a mtx_lock_sleep might point this way.... the >> trace shows it coming from a zfs_onexit_destroy, which ends up calling= >> zfs_unmount_snap() and then it blows in dounmount() while executing >> mtx_lock_sleep(). >> >> I do wonder if I'm begging for new and innovative performance issues i= f >> I run with TRIM off for an extended period of time, however..... :-) >> >> >> On 8/27/2015 15:30, Sean Chittenden wrote: >>> Have you tried disabling TRIM? We recently ran in to an issue where = a `zfs delete` on a large dataset caused the host to panic because TRIM w= as tripping over the ZFS deadman timer. Disabling TRIM worked as valid = workaround for us. ? You mentioned a recent move to SSDs, so this can h= appen, esp after the drive has experienced a little bit of actual work. = ? -sc >>> >>> >>> -- >>> Sean Chittenden >>> sean@chittenden.org >>> >>> >>>> On Aug 27, 2015, at 13:22, Karl Denninger wrote= : >>>> >>>> On 8/15/2015 12:38, Karl Denninger wrote: >>>>> Update: >>>>> >>>>> This /appears /to be related to attempting to send or receive a >>>>> /cloned /snapshot. >>>>> >>>>> I use /beadm /to manage boot environments and the crashes have all >>>>> come while send/recv-ing the root pool, which is the one where thes= e >>>>> clones get created. It is /not /consistent within a given snapshot= >>>>> when it crashes and a second attempt (which does a "recovery" >>>>> send/receive) succeeds every time -- I've yet to have it panic twic= e >>>>> sequentially. >>>>> >>>>> I surmise that the problem comes about when a file in the cloned >>>>> snapshot is modified, but this is a guess at this point. >>>>> >>>>> I'm going to try to force replication of the problem on my test sys= tem. >>>>> >>>>> On 7/31/2015 04:47, Karl Denninger wrote: >>>>>> I have an automated script that runs zfs send/recv copies to bring= a >>>>>> backup data set into congruence with the running copies nightly. = The >>>>>> source has automated snapshots running on a fairly frequent basis >>>>>> through zfs-auto-snapshot. >>>>>> >>>>>> Recently I have started having a panic show up about once a week d= uring >>>>>> the backup run, but it's inconsistent. It is in the same place, b= ut I >>>>>> cannot force it to repeat. >>>>>> >>>>>> The trap itself is a page fault in kernel mode in the zfs code at >>>>>> zfs_unmount_snap(); here's the traceback from the kvm (sorry for t= he >>>>>> image link but I don't have a better option right now.) >>>>>> >>>>>> I'll try to get a dump, this is a production machine with encrypte= d swap >>>>>> so it's not normally turned on. >>>>>> >>>>>> Note that the pool that appears to be involved (the backup pool) h= as >>>>>> passed a scrub and thus I would assume the on-disk structure is ok= =2E.... >>>>>> but that might be an unfair assumption. It is always occurring in= the >>>>>> same dataset although there are a half-dozen that are sync'd -- if= this >>>>>> one (the first one) successfully completes during the run then all= the >>>>>> rest will as well (that is, whenever I restart the process it has = always >>>>>> failed here.) The source pool is also clean and passes a scrub. >>>>>> >>>>>> traceback is at http://www.denninger.net/kvmimage.png; apologies f= or the >>>>>> image traceback but this is coming from a remote KVM. >>>>>> >>>>>> I first saw this on 10.1-STABLE and it is still happening on FreeB= SD >>>>>> 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to s= ee if >>>>>> the problem was something that had been addressed. >>>>>> >>>>>> >>>>> --=20 >>>>> Karl Denninger >>>>> karl@denninger.net >>>>> /The Market Ticker/ >>>>> /[S/MIME encrypted email preferred]/ >>>> Second update: I have now taken another panic on 10.2-Stable, same d= eal, >>>> but without any cloned snapshots in the source image. I had thought = that >>>> removing cloned snapshots might eliminate the issue; that is now out= the >>>> window. >>>> >>>> It ONLY happens on this one filesystem (the root one, incidentally) >>>> which is fairly-recently created as I moved this machine from spinni= ng >>>> rust to SSDs for the OS and root pool -- and only when it is being >>>> backed up by using zfs send | zfs recv (with the receive going to a >>>> different pool in the same machine.) I have yet to be able to provo= ke >>>> it when using zfs send to copy to a different machine on the same LA= N, >>>> but given that it is not able to be reproduced on demand I can't be >>>> certain it's timing related (e.g. performance between the two pools = in >>>> question) or just that I haven't hit the unlucky combination. >>>> >>>> This looks like some sort of race condition and I will continue to s= ee >>>> if I can craft a case to make it occur "on demand" >>>> >>>> --=20 >>>> Karl Denninger >>>> karl@denninger.net >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ >>> %SPAMBLOCK-SYS: Matched [+Sean Chittenden ], mes= sage ok >>> > > --=20 > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070409030504010001090508 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEwMTUxNDU2MjFaME8GCSqGSIb3DQEJBDFCBEB4 3DB4wJHGvHDPgNx5fUUOw4hKGEKCCIasQykFb/OMA/GsM6rTpjHcYcY8jIdlTzV8OZzMMbdk bhS6uAn2HCpZMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAeODoQB3f 52IrW9FEYbHbpenZJ1dVfft8ZtowOD7eHivRGWlPiRaym6RPPHhmLibZ/Gy09piVckZ6Dejy ySBnrAjbT8k2LIGkhIq3DrXW8ZUbPQDz+SsJD1NeXwIPbYLpgl1L/JlFueecRA2lcBFiXbM1 2UyGpBTH9XvhuTCEZpwW50fdHUqzS2eeXwE8wA+51LsBsMmFGJNo1RAIsxOVvIao1PudpVRM 1PxTCzpNvZ9u94r3FO4BH1Y8sKIBYn0ofPlr3Ij0/j09Z4paaavsFTA/vTHFi8MY6rImTkp1 yjmwC+9Cvj+C9z4vi0bNuRyvrVC91s5Gz+YPHjTtbk9jOYwTg5onWH+JiSD5Lh356wTTiZg7 hohx7sDUzK7vc9gJS1H74O6qv1pcQAAxIBQNPmkB3vKk7M7v6Hu/XXeugNuGstQH+YsgKtiq Je0WDJQYukl2Z1+R8gBT0fAH4GkLlLeAuvlGu3DTPJ7kklzEHY43NljlS+tb5PJNJmZsJT4r +Rmj8NMl80yA5bEGT2EiiFxOG3MOI1xEDk4/EyisfjmTe59q0oDmFAnWxZ0GgyLVtwzjWX1j YNL9CJxt/hlTL93lPAlrIW+lF+MeQ0qpIx0TBWGDn3SyEbExpCyKArkJMV395Q9XsgMVIF5H P3Et9bydEDwSSJ8IK9laXnBWpikAAAAAAAA= --------------ms070409030504010001090508-- From owner-freebsd-fs@freebsd.org Fri Oct 16 02:15:23 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C0D8A13ABF for ; Fri, 16 Oct 2015 02:15:23 +0000 (UTC) (envelope-from ff-wuestmark-de@web1.netbeat.de) Received: from web1.netbeat.de (web1.netbeat.de [83.243.58.172]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BBA11288 for ; Fri, 16 Oct 2015 02:15:22 +0000 (UTC) (envelope-from ff-wuestmark-de@web1.netbeat.de) Received: from ff-wuestmark-de by web1.netbeat.de with local (Exim 4.69) (envelope-from ) id 1ZmuY4-0003BA-1o for freebsd-fs@freebsd.org; Fri, 16 Oct 2015 04:15:12 +0200 X-Caller: /home/ff-wuestmark-de/htdocs/post.php To: freebsd-fs@freebsd.org Subject: Your ticket order #00000998436 approved Date: Fri, 16 Oct 2015 04:15:11 +0200 From: "America Airlines" Reply-To: "America Airlines" Message-ID: <34d9d766f70c53ce5f25eb1ebb33bb65@ff-wuestmark.de> X-Priority: 3 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 02:15:23 -0000 Dear customer, Your payment has been processed and your credit card has been charged. Please check your e-ticket in the attachment to this e-mail. Order details and e-ticket information: FLIGHT NUMBER / UC157344 DATE & TIME / Oct 20 2015, 10:30 DEPARTING / Baltimore TOTAL PRICE / $ 540.00 Thank you for choosing America Airlines. From owner-freebsd-fs@freebsd.org Fri Oct 16 12:29:02 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 157E5A169AB for ; Fri, 16 Oct 2015 12:29:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0189A9FE for ; Fri, 16 Oct 2015 12:29:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9GCT1pE079460 for ; Fri, 16 Oct 2015 12:29:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 172942] [smbfs] Unmounting a smb mount when the server became unavailable causes kernel panic Date: Fri, 16 Oct 2015 12:29:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 1.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 12:29:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=172942 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@FreeBSD.org --- Comment #2 from Rick Macklem --- Created attachment 162116 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=162116&action=edit Fix a race that causes crashes in smbfs that destroyed mutexes prematurely This patch seems to have fixed a similar problem w.r.t. smbfs crashes for someone who reported the problem to a mailing list. I believe it fixes the problem. It will be committed in a few weeks if nothing is reported to indicate that it doesn't fix the problem. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 16 12:34:06 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82ED3A16B83 for ; Fri, 16 Oct 2015 12:34:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F5F6F2F for ; Fri, 16 Oct 2015 12:34:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9GCY6g2064197 for ; Fri, 16 Oct 2015 12:34:06 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 172942] [smbfs] Unmounting a smb mount when the server became unavailable causes kernel panic Date: Fri, 16 Oct 2015 12:34:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 1.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 12:34:06 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=172942 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |rmacklem@FreeBSD.org --- Comment #3 from Rick Macklem --- I believe there is a race caused by smb_iod_destroy() where it calls sbm_iod_request() to shutdown the connection/iod thread. smb_iod_request() does an msleep(..PDROP..), which can return as soon as smb_iod_main() does the wakeup(). After returning from the msleep(), it returns to smb_iod_destroy(), which then destroys the mutexes and frees the iod structure. Unfortunately, smb_iod_main() is not done with the mutexes when it calls wakeup(). I believe this patch fixes the problem by moving the code that destroys the mutexs and frees the iod structure to the end of the smb_iod thread. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Fri Oct 16 12:54:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3FB50A16062 for ; Fri, 16 Oct 2015 12:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2CDF71B86 for ; Fri, 16 Oct 2015 12:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9GCsQrf006298 for ; Fri, 16 Oct 2015 12:54:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 201912] panic in smbfs during mount Date: Fri, 16 Oct 2015 12:54:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 12:54:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201912 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rmacklem@FreeBSD.org --- Comment #5 from Rick Macklem --- Created attachment 162117 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=162117&action=edit fix a race between smb_iod_destroy() and the smd_iod thread that destroys mutexes and frees the iod structure prematurely I think this patch might fix the problem that caused your crash. Your crash does look somewhat different than PR#172942, but it does call smb_iod_destroy() via smb_vc_gone() and that is where the race was. Unfortunately your crash does suggest that the mount was trying to do another smb_iod_destroy() when it had already happened and this might suggest an additional race that the patch doesn't address. Since it is somewhat different, I haven't marked it as a duplicate of PR#172942. I will do that if Martin reports back that the patch seems to have stopped the crashes from occurring. (I have no idea how reproducible these crashes are? -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sat Oct 17 01:40:36 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DFCF8A168CE for ; Sat, 17 Oct 2015 01:40:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCD4D25C for ; Sat, 17 Oct 2015 01:40:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9H1eaII056783 for ; Sat, 17 Oct 2015 01:40:36 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 201912] panic in smbfs during mount Date: Sat, 17 Oct 2015 01:40:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 01:40:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201912 --- Comment #6 from Rick Macklem --- Created attachment 162138 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=162138&action=edit fixes smbfs so that it doesn't do a disconnect when vc_iod == NULL This patch (which includes the 162117 one) adds a check for vc_iod != NULL to the code in smb_vc_disconnect(), since this function is called when smb_vc_create() fails and vc_iod == NULL for that case. It also fixes smb_iod_create() so it returns with vc_iod == NULL when it fails, since it has free'd the iod and it also adds code to destroy the mutexes for this case. I believe this patch will fix the crash reported here. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sat Oct 17 01:41:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E4124A1691F for ; Sat, 17 Oct 2015 01:41:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D12DC387 for ; Sat, 17 Oct 2015 01:41:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9H1fJLv060138 for ; Sat, 17 Oct 2015 01:41:19 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 201912] panic in smbfs during mount Date: Sat, 17 Oct 2015 01:41:20 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 01:41:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201912 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |In Progress -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Sat Oct 17 01:43:58 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E086DA16A9B for ; Sat, 17 Oct 2015 01:43:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD60D6E5 for ; Sat, 17 Oct 2015 01:43:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9H1hw4a066180 for ; Sat, 17 Oct 2015 01:43:58 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 201912] panic in smbfs during mount Date: Sat, 17 Oct 2015 01:43:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: rmacklem@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 01:43:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=201912 Rick Macklem changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |rmacklem@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug.