From owner-freebsd-fs@freebsd.org Sun Oct 18 16:57:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6116DA18547 for ; Sun, 18 Oct 2015 16:57:03 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-ig0-x22e.google.com (mail-ig0-x22e.google.com [IPv6:2607:f8b0:4001:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FB7EC44 for ; Sun, 18 Oct 2015 16:57:02 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by igbhv6 with SMTP id hv6so43153373igb.0 for ; Sun, 18 Oct 2015 09:57:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=qb4GyI1AfwV0Kcb/iE9irWTyGZiqm/jAcv7r5aTiiVk=; b=B0z6p5P4b7QZsUToBDqIfVlirlWZlizdzi07gR6ReA+qTjB4Im/SJVkshCY/MNpFVE ne3ccHx9z9PcYLahBVhHgtfJVxV4Uhwu/djLXlUVr1XvJPq781ZPHxuoq5A7FSrTF+F6 h5Xhu46XFIS4Usv8zhmctM0xSkTrlm9f1q3DI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=qb4GyI1AfwV0Kcb/iE9irWTyGZiqm/jAcv7r5aTiiVk=; b=SQWYabksrT1Ig8UJ/9NBm/mP2Lk6WukEd0AY9FgOnhQBCRyBUk+Rawz3xVTRL8Ec2v aBIxgkRjaokGb72qCB5LwEedPkfrEDkfxd1NaGv0PyIUhTVOCLvofvYZQtaF+l0Z/m6N kYSJL6kZnR1dnnTs73eVlrB8hn+Q8/xf0R6ykmMOIoxUqebEu+skrHuzSNOxOD8b3Yll 7PQGHk2aYMjdxnCWmrCpPu46EkYTeTBTJ6EH5S3d+Pe2wtFgRMppD4LlznatNgpaE90t mdONzOSaTivUZ+CA1KIbru2AemW1ojw8kO41DWqErR3ZjqVGmhoanl50bAHHJSi/y/6g T8Fg== X-Gm-Message-State: ALoCoQlPukyQr4tUQ+bUL3B6zPxS+nYj5Yq2kiee0d9tWZ6tkPeoZoLOPZ3rUVX/x9sEBOyTQhus MIME-Version: 1.0 X-Received: by 10.50.78.194 with SMTP id d2mr3677517igx.72.1445187421974; Sun, 18 Oct 2015 09:57:01 -0700 (PDT) Received: by 10.36.86.211 with HTTP; Sun, 18 Oct 2015 09:57:01 -0700 (PDT) Date: Sun, 18 Oct 2015 09:57:01 -0700 Message-ID: Subject: OpenZFS Developer Summit live streaming From: Matthew Ahrens To: developer , illumos-zfs , freebsd-fs , zfs-discuss Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 16:57:03 -0000 If you can't make it to the OpenZFS Developer Summit this year, we invite you to view the live webcast, beginning Monday at 9:30 AM PST, at http://open-zfs.org. Discuss on IRC (#openzfs ) or twitter (#openzfs ). Thanks to our genero= us sponsors, this year the live stream will feature audio you can hear, and no interruptions for advertisements! The recordings will also be available on YouTube later this month. Full schedule below: *Monday, October 19, 2015*TimeTitleSpeakerCompany 9:30am =E2=80=93 9:45amKeynoteMatt AhrensDelphix 9= :45am =E2=80=93 10:00amOpenZFS Success StoriesTarkan ManerNexenta 10:0= 0am =E2=80=93 10:45amZFS Internals OverviewKirk McKusickIndependent 10:45am =E2=80=93 11:00am*Break*11:00am =E2=80=93= 11:30amZFS Send and ReceivePaul DagnelieDelphix 11:30am =E2=80=93 11:45amCompressed Send and ReceiveDan KimmelDelphix 11:45am =E2=80= =93 12:00pmLive Migration with ZmotionFrancois LesageOVH 12:00pm =E2= =80=93 12:45pm*Lunch*12:45pm =E2=80=93 1:00pmThe Birth of ZFSJeff BonwickDSSD , EMC 1:00pm =E2=80=93 1:30pmParity Declustered RAID-Z/MirrorIsaac HuangIntel 1:30pm =E2=80=93 1:45= pmImprove Performance on AWS with Eager ZeroJoe SteinDelphix 1:45pm =E2=80=93 2:15pm*Break*2:15pm =E2=80=93 2:45pmCompressed ARCGeorge WilsonDe= lphix 2:45pm =E2=80=93 3:00pmDiscontiguous Caching with = ABDDavid ChenOSNexus 3:00pm =E2=80=93 3:15pmPersistent L2AR= CSaso KiselkovNexenta 3:15pm =E2=80=93 3:30pmDedup CeilingSa= so KiselkovNexenta 3:30pm =E2=80=93 4:00pm*Break*4:00pm = =E2=80=93 4:30pmWriteback Cache Alex AizmanNexenta 4:30pm =E2=80=93 4:45pmSandboxing OpenZFS on LinuxAlbe= rt Lee OmniTI 4:45pm =E2=80=93 5:00pmSPA Metadata Allocation C= lassesDon BradyIntel 5:00pm =E2= =80=93 5:15pmZtourDon BradyIntel 5:15pm =E2=80=93 5:30pmClosingMatt AhrensDelphix From owner-freebsd-fs@freebsd.org Sun Oct 18 21:00:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EC33A182C1 for ; Sun, 18 Oct 2015 21:00:54 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 22428AE9 for ; Sun, 18 Oct 2015 21:00:54 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9IL0suE000283 for ; Sun, 18 Oct 2015 21:00:54 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201510182100.t9IL0suE000283@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 18 Oct 2015 21:00:54 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Oct 2015 21:00:54 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic Open | 136470 | [nfs] Cannot mount / in read-only, over NFS Open | 139651 | [nfs] mount(8): read-only remount of NFS volume d Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f 4 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Oct 19 07:25:03 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 529F4A107DE for ; Mon, 19 Oct 2015 07:25:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3FBF1C20 for ; Mon, 19 Oct 2015 07:25:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J7P3vJ077593 for ; Mon, 19 Oct 2015 07:25:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Mon, 19 Oct 2015 07:25:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 07:25:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203864 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org CC| |ngie@FreeBSD.org --- Comment #2 from Garrett Cooper,425-314-3911 --- uname -a? FWIW, I ran into ZFS deadlocks with r288943 when running tests/sys/acl . When I upgrade to r289441, this went away. I was running into apparent deadlocks with the spa namespace lock though, not tx->tx_sync_done_cv. mav's modifications to zfs send/recv might need to be fixed. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 19 07:29:37 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D67FCA10973 for ; Mon, 19 Oct 2015 07:29:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C32CCE65 for ; Mon, 19 Oct 2015 07:29:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J7Tbft084198 for ; Mon, 19 Oct 2015 07:29:37 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Mon, 19 Oct 2015 07:29:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: peter@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 07:29:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203864 --- Comment #3 from Peter Wemm --- It's later than the revision you indicated: FreeBSD halo.ysv.freebsd.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r289488: Sun Oct 18 06:47:35 UTC 2015 peter@build-11.freebsd.org:/usr/obj/usr/src/sys/CLUSTER11 amd64 -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 19 07:43:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB278A10D49 for ; Mon, 19 Oct 2015 07:43:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9824714ED for ; Mon, 19 Oct 2015 07:43:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J7hQOv011811 for ; Mon, 19 Oct 2015 07:43:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Mon, 19 Oct 2015 07:43:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 07:43:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203864 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |dogfood, needs-qa CC| |koobs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 19 07:47:34 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A8524A10F32 for ; Mon, 19 Oct 2015 07:47:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 957D018E8 for ; Mon, 19 Oct 2015 07:47:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9J7lYV0017242 for ; Mon, 19 Oct 2015 07:47:34 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203864] ZFS deadlock between zfs send, zfs rename and ctrl-C Date: Mon, 19 Oct 2015 07:47:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: dogfood, needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 07:47:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203864 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |avg@FreeBSD.org --- Comment #4 from Andriy Gapon --- https://wiki.freebsd.org/AvgZfsDeadlockDebug -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 19 14:43:33 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74048A196DB for ; Mon, 19 Oct 2015 14:43:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60FEEE9C for ; Mon, 19 Oct 2015 14:43:33 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9JEhXT7069041 for ; Mon, 19 Oct 2015 14:43:33 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203649] makefs: Coverity CID 1305659: Unclear whether reaction on malloc failure suffices. Date: Mon, 19 Oct 2015 14:43:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 14:43:33 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203649 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |patch Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Mon Oct 19 14:44:26 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6A55A19741 for ; Mon, 19 Oct 2015 14:44:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B3A19F6D for ; Mon, 19 Oct 2015 14:44:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9JEiQ1i070066 for ; Mon, 19 Oct 2015 14:44:26 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203649] makefs: Coverity CID 1305659: Unclear whether reaction on malloc failure suffices. Date: Mon, 19 Oct 2015 14:44:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.0-CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-bugs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 14:44:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203649 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-fs@FreeBSD.org |freebsd-bugs@FreeBSD.org --- Comment #2 from Mark Linimon --- Hmm, this is in bin, not kern. Oops. -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Tue Oct 20 15:51:19 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB2DAA195A2 for ; Tue, 20 Oct 2015 15:51:19 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5C1A0CF8 for ; Tue, 20 Oct 2015 15:51:18 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id F03621C1350 for ; Tue, 20 Oct 2015 10:51:10 -0500 (CDT) Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) To: freebsd-fs@freebsd.org References: <55BB443E.8040801@denninger.net> <55CF7926.1030901@denninger.net> <55DF7191.2080409@denninger.net> <55DF76AA.3040103@denninger.net> <561FB7D0.4080107@denninger.net> From: Karl Denninger X-Enigmail-Draft-Status: N1110 Message-ID: <562662ED.1070302@denninger.net> Date: Tue, 20 Oct 2015 10:51:09 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561FB7D0.4080107@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020804010306020008070202" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 15:51:19 -0000 This is a cryptographically signed message in MIME format. --------------ms020804010306020008070202 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable More data on this crash from this morning; I caught it in-process this time and know exactly where it was in the backup script when it detonated= =2E Here's the section of the script that was running when it blew up: for i in $ZFS do echo Processing $i FILESYS=3D`echo $i|cut -d/ -f2` zfs list $i@zfs-base >/dev/null 2>&1 result=3D$? if [ $result -eq 1 ]; then echo "Make and send zfs-base snapshot" zfs snapshot -r $i@zfs-base zfs send -R $i@zfs-base | zfs receive -Fuvd $BACKUP else base_only=3D`zfs get -H com.backup:base-only $i|grep true= ` result=3D$? if [ $result -eq 1 ]; then echo "Bring incremental backup up to date" old_present=3D`zfs list $i@zfs-old >/dev/null 2>&= 1` old=3D$? if [ $old -eq 0 ]; then echo "Previous incremental sync was interrupted; resume" else zfs rename -r $i@zfs-base $i@zfs-old zfs snapshot -r $i@zfs-base fi zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP result=3D$? if [ $result -eq 0 ]; then zfs destroy -r $i@zfs-old zfs destroy -r $BACKUP/$FILESYS@zfs-old else echo "STOP - - ERROR RUNNING COPY on $i!!= !!" exit 1 fi else echo "Base-only backup configured for $i" fi fi echo done And the output on the console when it happened: Begin local ZFS backup by SEND Run backups of default [zsr/R/10.2-STABLE zsr/ssd-txlog zs/archive zs/colo-archive zs/disk zs/pgsql zs/pics dbms/ticker] Tue Oct 20 10:14:09 CDT 2015 Import backup pool Imported; ready to proceed Processing zsr/R/10.2-STABLE Bring incremental backup up to date _*Previous incremental sync was interrupted; resume*_ attempting destroy backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-13-00= h07 success attempting destroy backup/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-18-12h04 success *local fs backup/R/10.2-STABLE does not have fromsnap (zfs-old in stream)= ;** **must have been deleted locally; ignoring* receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 into backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 snap backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 already exists; ignoring received 0B stream in 1 seconds (0B/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 into backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 snap backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 already exists; ignoring received 0B stream in 1 seconds (0B/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-base into backup/R/10.2-STABLE@zfs-base snap backup/R/10.2-STABLE@zfs-base already exists; ignoring That bolded pair of lines should _*not*_ be there. It means that the snapshot "zfs-old" is on the source volume, but it shouldn't be there because after the copy is complete we destroy it on both the source and destination. Further, when it is attempted to be sent while the snapshot name (zfs-old) was found in a zfs list /*the data allegedly associated with the phantom snapshot that shouldn't exist was not there */(second bolded line) zfs send -RI $i@zfs-old $i@zfs-base | zfs receive -Fudv $BACKUP result=3D$? if [ $result -eq 0 ]; then *zfs destroy -r $i@zfs-old** ** zfs destroy -r $BACKUP/$FILESYS@zfs-old= * else echo "STOP - - ERROR RUNNING COPY on $i!!= !!" exit 1 fi I don't have the trace from the last run (I didn't save it) *but there were no errors returned by it *as it was run by hand (from the console) while I was watching it. This STRONGLY implies that the zfs destroy /allegedly /succeeded (it ran without an error and actually removed the snapshot) but left the snapshot _*name*_ on the volume as it was able to be found by the script, and then when that was referenced by the backup script in the incremental send the data set was invalid producing the resulting panic. The bad news is that the pool shows no faults and a scrub which took place a few days ago says the pool is fine. If I re-run the backup I suspect it will properly complete (it always has in the past as a resume from the interrupted one) but clearly, whatever is going on here looks like some sort of race in the destroy commands (which _*are*_ being run recursively) that leaves the snapshot name on the volume but releases the data stored in it, thus the panic when it is attempted to be referenced. I have NOT run a scrub on the pool in an attempt to preserve whatever evidence may be there; if there is something that I can look at with zdb or similar I'll leave this alone for a bit -- the machine came back up and is running fine. This is a production system but I can take it down off-hours for a short time if there is a need to run something in single-user mode using zdb to try to figure out what's going on. There have been no known hardware issues with it and all the other pools (including the ones on the same host adapter) are and have been running without incident. Ideas? Fatal trap 12: page fault while in kernel mode cpuid =3D 9; apic id =3D 21 fault virtual address =3D 0x378 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80940ae0 stack pointer =3D 0x28:0xfffffe0668018680 frame pointer =3D 0x28:0xfffffe0668018700 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 70921 (zfs) trap number =3D 12 panic: page fault cpuid =3D 9 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0668018160 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0668018210 vpanic() at vpanic+0x126/frame 0xfffffe0668018250 panic() at panic+0x43/frame 0xfffffe06680182b0 trap_fatal() at trap_fatal+0x36b/frame 0xfffffe0668018310 trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe06680183b0 trap() at trap+0x47a/frame 0xfffffe06680185c0 calltrap() at calltrap+0x8/frame 0xfffffe06680185c0 --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe0668018680, rbp= =3D 0xfffffe0668018700 --- *__mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe0668018700** **dounmount() at dounmount+0x58/frame 0xfffffe0668018780** **zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe06680187a0**= **dsl_dataset_user_release_impl() at dsl_dataset_user_release_impl+0x103/frame 0xfffffe0668018920** **dsl_dataset_user_release_onexit() at dsl_dataset_user_release_onexit+0x5c/frame 0xfffffe0668018940** **zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066801897= 0** **zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe0668018990* devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe06680189b0 devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe06680189e0 _fdrop() at _fdrop+0x29/frame 0xfffffe0668018a00 closef() at closef+0x21e/frame 0xfffffe0668018a90 closefp() at closefp+0x98/frame 0xfffffe0668018ad0 amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe0668018bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0668018bf0 --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D 0x7fffffffd1e8, rbp =3D 0x7fffffffd200 --- Uptime: 5d0h58m0s On 10/15/2015 09:27, Karl Denninger wrote: > Got another one of these this morning, after a long absence... > > Same symptom -- happened during a backup (send/receive) and it's in > the same place -- when the snapshot is unmounted. I have a clean dump > and this is against a quite-recent checkout, so the > most-currently-rolled forward ZFS changes are in this kernel. > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 14; apic id =3D 34 > fault virtual address =3D 0x378 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80940ae0 > stack pointer =3D 0x28:0xfffffe066796c680 > frame pointer =3D 0x28:0xfffffe066796c700 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 81187 (zfs) > trap number =3D 12 > panic: page fault > cpuid =3D 14 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe066796c160 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe066796c210 > vpanic() at vpanic+0x126/frame 0xfffffe066796c250 > panic() at panic+0x43/frame 0xfffffe066796c2b0 > trap_fatal() at trap_fatal+0x36b/frame 0xfffffe066796c310 > trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe066796c3b0 > trap() at trap+0x47a/frame 0xfffffe066796c5c0 > calltrap() at calltrap+0x8/frame 0xfffffe066796c5c0 > --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe066796c680, r= bp > =3D 0xfffffe > 066796c700 --- > __mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe066796c700 > dounmount() at dounmount+0x58/frame 0xfffffe066796c780 > zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe066796c7a0 > dsl_dataset_user_release_impl() at > dsl_dataset_user_release_impl+0x103/frame 0xf > ffffe066796c920 > dsl_dataset_user_release_onexit() at > dsl_dataset_user_release_onexit+0x5c/frame > 0xfffffe066796c940 > zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066796c97= 0 > zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe066796c990 > devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe066796c9b0 > devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe066796c9e0 > _fdrop() at _fdrop+0x29/frame 0xfffffe066796ca00 > closef() at closef+0x21e/frame 0xfffffe066796ca90 > closefp() at closefp+0x98/frame 0xfffffe066796cad0 > amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe066796cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe066796cbf0 > --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= > 0x7fffffffd728, rbp =3D 0x7fffffffd740 --- > Uptime: 3d15h37m26s > > > On 8/27/2015 15:44, Karl Denninger wrote: >> No, but that does sound like it might be involved..... >> >> And yeah, this did start when I moved the root pool to a mirrored pair= >> of Intel 530s off a pair of spinning-rust WD RE4s.... (The 530s are da= rn >> nice performance-wise, reasonably inexpensive and thus very suitable f= or >> a root filesystem drive and they also pass the "pull the power cord" >> test, incidentally.) >> >> You may be onto something -- I'll try shutting it off, but due to the >> fact that I can't make this happen and it's a "every week or two" pani= c, >> but ALWAYS when the zfs send | zfs recv is running AND it's always on >> the same filesystem it will be a fair while before I know if it's fixe= d >> (like over a month, given the usual pattern here, as that would be 4 >> "average" periods without a panic)..... >> >> I also wonder if I could tune this out with some of the other TRIM >> parameters instead of losing it entirely. >> >> vfs.zfs.trim.max_interval: 1 >> vfs.zfs.trim.timeout: 30 >> vfs.zfs.trim.txg_delay: 32 >> vfs.zfs.trim.enabled: 1 >> vfs.zfs.vdev.trim_max_pending: 10000 >> vfs.zfs.vdev.trim_max_active: 64 >> vfs.zfs.vdev.trim_min_active: 1 >> >> That it's panic'ing on a mtx_lock_sleep might point this way.... the >> trace shows it coming from a zfs_onexit_destroy, which ends up calling= >> zfs_unmount_snap() and then it blows in dounmount() while executing >> mtx_lock_sleep(). >> >> I do wonder if I'm begging for new and innovative performance issues i= f >> I run with TRIM off for an extended period of time, however..... :-) >> >> >> On 8/27/2015 15:30, Sean Chittenden wrote: >>> Have you tried disabling TRIM? We recently ran in to an issue where = a `zfs delete` on a large dataset caused the host to panic because TRIM w= as tripping over the ZFS deadman timer. Disabling TRIM worked as valid = workaround for us. ? You mentioned a recent move to SSDs, so this can h= appen, esp after the drive has experienced a little bit of actual work. = ? -sc >>> >>> >>> -- >>> Sean Chittenden >>> sean@chittenden.org >>> >>> >>>> On Aug 27, 2015, at 13:22, Karl Denninger wrote= : >>>> >>>> On 8/15/2015 12:38, Karl Denninger wrote: >>>>> Update: >>>>> >>>>> This /appears /to be related to attempting to send or receive a >>>>> /cloned /snapshot. >>>>> >>>>> I use /beadm /to manage boot environments and the crashes have all >>>>> come while send/recv-ing the root pool, which is the one where thes= e >>>>> clones get created. It is /not /consistent within a given snapshot= >>>>> when it crashes and a second attempt (which does a "recovery" >>>>> send/receive) succeeds every time -- I've yet to have it panic twic= e >>>>> sequentially. >>>>> >>>>> I surmise that the problem comes about when a file in the cloned >>>>> snapshot is modified, but this is a guess at this point. >>>>> >>>>> I'm going to try to force replication of the problem on my test sys= tem. >>>>> >>>>> On 7/31/2015 04:47, Karl Denninger wrote: >>>>>> I have an automated script that runs zfs send/recv copies to bring= a >>>>>> backup data set into congruence with the running copies nightly. = The >>>>>> source has automated snapshots running on a fairly frequent basis >>>>>> through zfs-auto-snapshot. >>>>>> >>>>>> Recently I have started having a panic show up about once a week d= uring >>>>>> the backup run, but it's inconsistent. It is in the same place, b= ut I >>>>>> cannot force it to repeat. >>>>>> >>>>>> The trap itself is a page fault in kernel mode in the zfs code at >>>>>> zfs_unmount_snap(); here's the traceback from the kvm (sorry for t= he >>>>>> image link but I don't have a better option right now.) >>>>>> >>>>>> I'll try to get a dump, this is a production machine with encrypte= d swap >>>>>> so it's not normally turned on. >>>>>> >>>>>> Note that the pool that appears to be involved (the backup pool) h= as >>>>>> passed a scrub and thus I would assume the on-disk structure is ok= =2E.... >>>>>> but that might be an unfair assumption. It is always occurring in= the >>>>>> same dataset although there are a half-dozen that are sync'd -- if= this >>>>>> one (the first one) successfully completes during the run then all= the >>>>>> rest will as well (that is, whenever I restart the process it has = always >>>>>> failed here.) The source pool is also clean and passes a scrub. >>>>>> >>>>>> traceback is at http://www.denninger.net/kvmimage.png; apologies f= or the >>>>>> image traceback but this is coming from a remote KVM. >>>>>> >>>>>> I first saw this on 10.1-STABLE and it is still happening on FreeB= SD >>>>>> 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to s= ee if >>>>>> the problem was something that had been addressed. >>>>>> >>>>>> >>>>> --=20 >>>>> Karl Denninger >>>>> karl@denninger.net >>>>> /The Market Ticker/ >>>>> /[S/MIME encrypted email preferred]/ >>>> Second update: I have now taken another panic on 10.2-Stable, same d= eal, >>>> but without any cloned snapshots in the source image. I had thought = that >>>> removing cloned snapshots might eliminate the issue; that is now out= the >>>> window. >>>> >>>> It ONLY happens on this one filesystem (the root one, incidentally) >>>> which is fairly-recently created as I moved this machine from spinni= ng >>>> rust to SSDs for the OS and root pool -- and only when it is being >>>> backed up by using zfs send | zfs recv (with the receive going to a >>>> different pool in the same machine.) I have yet to be able to provo= ke >>>> it when using zfs send to copy to a different machine on the same LA= N, >>>> but given that it is not able to be reproduced on demand I can't be >>>> certain it's timing related (e.g. performance between the two pools = in >>>> question) or just that I haven't hit the unlucky combination. >>>> >>>> This looks like some sort of race condition and I will continue to s= ee >>>> if I can craft a case to make it occur "on demand" >>>> >>>> --=20 >>>> Karl Denninger >>>> karl@denninger.net >>>> /The Market Ticker/ >>>> /[S/MIME encrypted email preferred]/ >>> %SPAMBLOCK-SYS: Matched [+Sean Chittenden ], mes= sage ok >>> > > --=20 > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020804010306020008070202 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEwMjAxNTUxMDlaME8GCSqGSIb3DQEJBDFCBECH IaPMFjd/MssLYsIDpBu5o9tqiWyZ9FLXzccK7FKFmvMZeOo3sc7eJ3ucNGPDOr5VYsB6mPzF 5OY0f8m3w3kfMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAAUeeayWj BC8PtJ1u3b8hUpYDaLZy2hc5uYyhqJxALjYoIA1VLo+sVpSa0CXZ+TsRQL4Ir0lGevmSqCGP 1AsG6XuXPCdAbkp8Jz5v2LckIgw765Q2lfv3vCzaO4qP5hNcTHxrrHmAXd8NiXxKi4TDyax4 f9RmF1lGZikToARHp17dpVAo3do6aql3pZP1Rir4FPa//h9zh6NsIx8VVLknIxkYaLPKQI4d v6y73mD/EenYx9KiYn4ACWIonbMrrj2OuZJssAzRXGvqCY1IuaIuEb53ipNhhA0rNGIs/3Un l5UvTPaGtiW75Z8n5JbcMShZyr7Hd0v3la8MLViYsHCWKEqutB9aaOS5f6r07EchiebW8Zj0 p6seT4gZ2NUMFjg0XrrkskbRYCiNNx7nnBxygCaDoIAd6Pwt1GePqQKiGW66z4IL83d/x0nQ Sjj/ERIFUD6pY1Rn13QOebBpLtG6v1CrWfLvsG8IuSjMF9n7frIJrR2rlCsH8AMJhtx3s4ky jccGhWmJm6hf6llKt7IW9K65J3mmvZ0VARvT1K6X1V4vLYSLhQjLTPNedmMZTUXZcD5y/ZVu 6cb8v3CWIw31EzT4CrEZckT0T0GzPqVa9ntotX/Ka5ypC7dHZS0qEyIA0EtA21rOfmY20ZU0 hdV3azCXdVYSyQXPhrsBqsbIfeIAAAAAAAA= --------------ms020804010306020008070202-- From owner-freebsd-fs@freebsd.org Tue Oct 20 23:43:54 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57108A15AF6 for ; Tue, 20 Oct 2015 23:43:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4453211C4 for ; Tue, 20 Oct 2015 23:43:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9KNhsQY027739 for ; Tue, 20 Oct 2015 23:43:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 203906] ZFS lockup, spa_namespace_lock Date: Tue, 20 Oct 2015 23:43:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2015 23:43:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203906 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-fs@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@freebsd.org Wed Oct 21 01:29:00 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7FB25A195AB for ; Wed, 21 Oct 2015 01:29:00 +0000 (UTC) (envelope-from jason@broken.net) Received: from broken.net (broken.net [198.134.7.18]) by mx1.freebsd.org (Postfix) with ESMTP id 6B61A1A8 for ; Wed, 21 Oct 2015 01:29:00 +0000 (UTC) (envelope-from jason@broken.net) Received: from [192.168.253.69] (c-73-158-46-10.hsd1.ca.comcast.net [73.158.46.10]) by broken.net (Postfix) with ESMTPSA id 9C2D3D798; Tue, 20 Oct 2015 18:04:54 -0700 (PDT) Subject: Re: [zfs] RE: granularity of performance penalty from resilvering To: zfs@lists.illumos.org, Fred Liu , "zfs-discuss@list.zfsonlinux.org" , developer , freebsd-fs , zfs-discuss References: From: Jason Matthews Message-ID: <5626E4B3.2020904@broken.net> Date: Tue, 20 Oct 2015 18:04:51 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:29:00 -0000 you could look at these tunables (not the settings themselves)... these settings actually make resilvers have a higher priority. You obviously would want to go the other way. j. |* Prioritize resilvering by setting the delay to zero set zfs:zfs_resilver_delay = 0|| * Prioritize scrubs by setting the delay to zero set zfs:zfs_scrub_delay = 0 ||* resilver for five seconds per TXG set zfs:zfs_resilver_min_time_ms = 5000 | |echo zfs_resilver_delay/w0 | mdb -kw echo zfs_scrub_delay/w0 |mdb -kw echo zfs_top_maxinflight/w7f |mdb -kw echo zfs_resilver_min_time_ms/w1388 |mdb -kw | On 10/19/2015 11:49 PM, Fred Liu wrote: > > Yes, “zpool scrub –s” can stop the resilvering. > > *From:*Fred Liu > *Sent:* 星 期二, 十月20, 2015 12:15 > *To:* 'zfs-discuss@list.zfsonlinux.org'; developer; illumos-zfs; > freebsd-fs; zfs-discuss > *Subject:* granularity of performance penalty from resilvering > > Sorry if is a duplicate thread. > > The last suffering has been lasted for two weeks for we replaced a 6TB > HDD. > > There should be some IO throttle measure from ZFS software stack. At > least, we can try to stop resilvering like scrubbing > > if the realization is quiet complicated. > > Besides that, will nice zil/cache be relief? > > Thanks. > > Fred > > *illumos-zfs* | Archives > > > | Modify > > Your Subscription [Powered by Listbox] > From owner-freebsd-fs@freebsd.org Wed Oct 21 01:52:53 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17DB8A19E59 for ; Wed, 21 Oct 2015 01:52:53 +0000 (UTC) (envelope-from jason@broken.net) Received: from broken.net (broken.net [198.134.7.18]) by mx1.freebsd.org (Postfix) with ESMTP id 02DCA2B6 for ; Wed, 21 Oct 2015 01:52:52 +0000 (UTC) (envelope-from jason@broken.net) Received: from [192.168.253.69] (c-73-158-46-10.hsd1.ca.comcast.net [73.158.46.10]) by broken.net (Postfix) with ESMTPSA id 823CFDA82; Tue, 20 Oct 2015 18:52:51 -0700 (PDT) Subject: Re: [zfs] RE: granularity of performance penalty from resilvering To: zfs@lists.illumos.org, "zfs-discuss@list.zfsonlinux.org" , developer , freebsd-fs , zfs-discuss , "developer@lists.illumos.org" References: <5626E4B3.2020904@broken.net> From: Jason Matthews Message-ID: <5626EFF1.9020208@broken.net> Date: Tue, 20 Oct 2015 18:52:49 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:52:53 -0000 That is the only way to fly... j. On 10/20/2015 6:35 PM, Fred Liu wrote: > > Yeah. I want to go the other way. Plus, these settings are only > applicable in illumos. > > Therefore I decide to give up the hybrid( ssd+sata) solution to > underpin applications which need decent RAS. > > I am gonna go all-flash array. > > Thanks. > > Fred > > *From:*Jason Matthews [mailto:jason@broken.net] > *Sent:* 星期三, 十月21, 2015 9:05 > *To:* zfs@lists.illumos.org; Fred Liu; > zfs-discuss@list.zfsonlinux.org; developer; freebsd-fs; zfs-discuss > *Subject:* Re: [zfs] RE: granularity of performance penalty from > resilvering > > > > you could look at these tunables (not the settings themselves)... > > these settings actually make resilvers have a higher priority. You > obviously would want to go the other way. > > j. > > |* Prioritize resilvering by setting the delay to zero| > |set zfs:zfs_resilver_delay = 0 | > > * Prioritize scrubs by setting the delay to zero > set zfs:zfs_scrub_delay = 0 > > |* resilver for five seconds per TXG| > |set zfs:zfs_resilver_min_time_ms = 5000| > > > |echo zfs_resilver_delay/w0 | mdb -kw| > |echo zfs_scrub_delay/w0 |mdb -kw| > |echo zfs_top_maxinflight/w7f |mdb -kw| > |echo zfs_resilver_min_time_ms/w1388 |mdb -kw| > > On 10/19/2015 11:49 PM, Fred Liu wrote: > > Yes, “zpool scrub –s” can stop the resilvering. > > *From:*Fred Liu > *Sent:* 星 期二, 十月20, 2015 12:15 > *To:* 'zfs-discuss@list.zfsonlinux.org > '; developer; illumos-zfs; > freebsd-fs; zfs-discuss > *Subject:* granularity of performance penalty from resilvering > > Sorry if is a duplicate thread. > > The last suffering has been lasted for two weeks for we replaced a > 6TB HDD. > > There should be some IO throttle measure from ZFS software stack. > At least, we can try to stop resilvering like scrubbing > > if the realization is quiet complicated. > > Besides that, will nice zil/cache be relief? > > Thanks. > > Fred > > *illumos-zfs* | Archives > > > | Modify > > Your Subscription [Powered by Listbox] > From owner-freebsd-fs@freebsd.org Wed Oct 21 01:59:53 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 38ACFA1A081; Wed, 21 Oct 2015 01:59:53 +0000 (UTC) (envelope-from jason@broken.net) Received: from broken.net (broken.net [198.134.7.18]) by mx1.freebsd.org (Postfix) with ESMTP id 1B2365F5; Wed, 21 Oct 2015 01:59:53 +0000 (UTC) (envelope-from jason@broken.net) Received: from [192.168.253.69] (c-73-158-46-10.hsd1.ca.comcast.net [73.158.46.10]) by broken.net (Postfix) with ESMTPSA id E5CE2DAF6; Tue, 20 Oct 2015 18:59:52 -0700 (PDT) Subject: Re: [zfs] RE: granularity of performance penalty from resilvering To: zfs@lists.illumos.org, "zfs-discuss@list.zfsonlinux.org" , developer , freebsd-fs , zfs-discuss , "developer@lists.illumos.org" , "zfs-devel@freebsd.org" References: <5626E4B3.2020904@broken.net> <5626EFF1.9020208@broken.net> From: Jason Matthews Message-ID: <5626F196.5080606@broken.net> Date: Tue, 20 Oct 2015 18:59:50 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 01:59:53 -0000 i just sent you the tunables. turn the knobs in the other direction. j. On 10/20/2015 6:58 PM, Fred Liu wrote: > This feature is really needed. Even in some low-end RAID controllers, the rebuilding penalty > is tunable more than the high-end enterprise storage.... > >> -----Original Message----- >> From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] >> On Behalf Of Jason Matthews >> Sent: 星期三, 十月 21, 2015 9:53 >> To: zfs@lists.illumos.org; zfs-discuss@list.zfsonlinux.org; developer; >> freebsd-fs; zfs-discuss; developer@lists.illumos.org >> Subject: Re: [zfs] RE: granularity of performance penalty from >> resilvering >> >> >> That is the only way to fly... >> >> j. >> >> On 10/20/2015 6:35 PM, Fred Liu wrote: >>> Yeah. I want to go the other way. Plus, these settings are only >>> applicable in illumos. >>> >>> Therefore I decide to give up the hybrid( ssd+sata) solution to >>> underpin applications which need decent RAS. >>> >>> I am gonna go all-flash array. >>> >>> Thanks. >>> >>> Fred >>> >>> *From:*Jason Matthews [mailto:jason@broken.net] >>> *Sent:* 星期三, 十月21, 2015 9:05 >>> *To:* zfs@lists.illumos.org; Fred Liu; >>> zfs-discuss@list.zfsonlinux.org; developer; freebsd-fs; zfs-discuss >>> *Subject:* Re: [zfs] RE: granularity of performance penalty from >>> resilvering >>> >>> >>> >>> you could look at these tunables (not the settings themselves)... >>> >>> these settings actually make resilvers have a higher priority. You >>> obviously would want to go the other way. >>> >>> j. >>> >>> |* Prioritize resilvering by setting the delay to zero| set >>> |zfs:zfs_resilver_delay = 0 | >>> >>> * Prioritize scrubs by setting the delay to zero set >>> zfs:zfs_scrub_delay = 0 >>> >>> |* resilver for five seconds per TXG| >>> |set zfs:zfs_resilver_min_time_ms = 5000| >>> >>> >>> |echo zfs_resilver_delay/w0 | mdb -kw| echo zfs_scrub_delay/w0 |mdb >>> |-kw| echo zfs_top_maxinflight/w7f |mdb -kw| echo >>> |zfs_resilver_min_time_ms/w1388 |mdb -kw| >>> >>> On 10/19/2015 11:49 PM, Fred Liu wrote: >>> >>> Yes, “zpool scrub –s” can stop the resilvering. >>> >>> *From:*Fred Liu >>> *Sent:* 星 期二, 十月20, 2015 12:15 >>> *To:* 'zfs-discuss@list.zfsonlinux.org >>> '; developer; illumos-zfs; >>> freebsd-fs; zfs-discuss >>> *Subject:* granularity of performance penalty from resilvering >>> >>> Sorry if is a duplicate thread. >>> >>> The last suffering has been lasted for two weeks for we replaced >> a >>> 6TB HDD. >>> >>> There should be some IO throttle measure from ZFS software stack. >>> At least, we can try to stop resilvering like scrubbing >>> >>> if the realization is quiet complicated. >>> >>> Besides that, will nice zil/cache be relief? >>> >>> Thanks. >>> >>> Fred >>> >>> *illumos-zfs* | Archives >>> >>> >>> | Modify >>> >> > f5b912c9> >>> Your Subscription [Powered by Listbox] >>> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > ------------------------------------------- > illumos-zfs > Archives: https://www.listbox.com/member/archive/182191/=now > RSS Feed: https://www.listbox.com/member/archive/rss/182191/22567878-8480fd5f > Modify Your Subscription: https://www.listbox.com/member/?member_id=22567878&id_secret=22567878-f5b912c9 > Powered by Listbox: http://www.listbox.com From owner-freebsd-fs@freebsd.org Wed Oct 21 04:46:20 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5A1FA1ACEB; Wed, 21 Oct 2015 04:46:20 +0000 (UTC) (envelope-from jason@broken.net) Received: from broken.net (broken.net [198.134.7.18]) by mx1.freebsd.org (Postfix) with ESMTP id A123530F; Wed, 21 Oct 2015 04:46:20 +0000 (UTC) (envelope-from jason@broken.net) Received: from [192.168.253.69] (c-73-158-46-10.hsd1.ca.comcast.net [73.158.46.10]) by broken.net (Postfix) with ESMTPSA id 81CA1D3AE; Tue, 20 Oct 2015 21:46:18 -0700 (PDT) Subject: Re: [zfs] RE: granularity of performance penalty from resilvering To: zfs@lists.illumos.org, "zfs-discuss@list.zfsonlinux.org" , developer , freebsd-fs , zfs-discuss , "developer@lists.illumos.org" , "zfs-devel@freebsd.org" References: <5626E4B3.2020904@broken.net> <5626EFF1.9020208@broken.net> <5626F196.5080606@broken.net> From: Jason Matthews Message-ID: <56271897.4030209@broken.net> Date: Tue, 20 Oct 2015 21:46:15 -0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 04:46:20 -0000 I am not sure we are communicating. Instead of prioritizing resilvering, de-prioritize resilvering using those tunables. j. On 10/20/2015 7:14 PM, Fred Liu wrote: > Yeah, that is a trade-off. Prioritize resilvering high will shorten the time window. > That means brake your application "temporarily??". Even in current situation, the resilvering > eats almost all the IOs. > >> -----Original Message----- >> From: owner-zfs-devel@freebsd.org [mailto:owner-zfs-devel@freebsd.org] >> On Behalf Of Jason Matthews >> Sent: 星期三, 十月 21, 2015 10:00 >> To: zfs@lists.illumos.org; zfs-discuss@list.zfsonlinux.org; developer; >> freebsd-fs; zfs-discuss; developer@lists.illumos.org; zfs- >> devel@freebsd.org >> Subject: Re: [zfs] RE: granularity of performance penalty from >> resilvering >> >> >> >> i just sent you the tunables. turn the knobs in the other direction. >> >> j. >> >> On 10/20/2015 6:58 PM, Fred Liu wrote: >>> This feature is really needed. Even in some low-end RAID controllers, >>> the rebuilding penalty is tunable more than the high-end enterprise >> storage.... >>>> -----Original Message----- >>>> From: owner-freebsd-fs@freebsd.org >>>> [mailto:owner-freebsd-fs@freebsd.org] >>>> On Behalf Of Jason Matthews >>>> Sent: 星期三, 十月 21, 2015 9:53 >>>> To: zfs@lists.illumos.org; zfs-discuss@list.zfsonlinux.org; >>>> developer; freebsd-fs; zfs-discuss; developer@lists.illumos.org >>>> Subject: Re: [zfs] RE: granularity of performance penalty from >>>> resilvering >>>> >>>> >>>> That is the only way to fly... >>>> >>>> j. >>>> >>>> On 10/20/2015 6:35 PM, Fred Liu wrote: >>>>> Yeah. I want to go the other way. Plus, these settings are only >>>>> applicable in illumos. >>>>> >>>>> Therefore I decide to give up the hybrid( ssd+sata) solution to >>>>> underpin applications which need decent RAS. >>>>> >>>>> I am gonna go all-flash array. >>>>> >>>>> Thanks. >>>>> >>>>> Fred >>>>> >>>>> *From:*Jason Matthews [mailto:jason@broken.net] >>>>> *Sent:* 星期三, 十月21, 2015 9:05 >>>>> *To:* zfs@lists.illumos.org; Fred Liu; >>>>> zfs-discuss@list.zfsonlinux.org; developer; freebsd-fs; zfs-discuss >>>>> *Subject:* Re: [zfs] RE: granularity of performance penalty from >>>>> resilvering >>>>> >>>>> >>>>> >>>>> you could look at these tunables (not the settings themselves)... >>>>> >>>>> these settings actually make resilvers have a higher priority. You >>>>> obviously would want to go the other way. >>>>> >>>>> j. >>>>> >>>>> |* Prioritize resilvering by setting the delay to zero| set >>>>> |zfs:zfs_resilver_delay = 0 | >>>>> >>>>> * Prioritize scrubs by setting the delay to zero set >>>>> zfs:zfs_scrub_delay = 0 >>>>> >>>>> |* resilver for five seconds per TXG| set >>>>> |zfs:zfs_resilver_min_time_ms = 5000| >>>>> >>>>> >>>>> |echo zfs_resilver_delay/w0 | mdb -kw| echo zfs_scrub_delay/w0 |mdb >>>>> |-kw| echo zfs_top_maxinflight/w7f |mdb -kw| echo >>>>> |zfs_resilver_min_time_ms/w1388 |mdb -kw| >>>>> >>>>> On 10/19/2015 11:49 PM, Fred Liu wrote: >>>>> >>>>> Yes, “zpool scrub –s” can stop the resilvering. >>>>> >>>>> *From:*Fred Liu >>>>> *Sent:* 星 期二, 十月20, 2015 12:15 >>>>> *To:* 'zfs-discuss@list.zfsonlinux.org >>>>> '; developer; illumos- >> zfs; >>>>> freebsd-fs; zfs-discuss >>>>> *Subject:* granularity of performance penalty from resilvering >>>>> >>>>> Sorry if is a duplicate thread. >>>>> >>>>> The last suffering has been lasted for two weeks for we >>>>> replaced >>>> a >>>>> 6TB HDD. >>>>> >>>>> There should be some IO throttle measure from ZFS software >> stack. >>>>> At least, we can try to stop resilvering like scrubbing >>>>> >>>>> if the realization is quiet complicated. >>>>> >>>>> Besides that, will nice zil/cache be relief? >>>>> >>>>> Thanks. >>>>> >>>>> Fred >>>>> >>>>> *illumos-zfs* | Archives >>>>> >>>>> > 8480fd5f >>>>> | Modify >>>>> >>>> >>> f5b912c9> >>>>> Your Subscription [Powered by Listbox] >>>>> >>>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs- >> unsubscribe@freebsd.org" >>> ------------------------------------------- >>> illumos-zfs >>> Archives: https://www.listbox.com/member/archive/182191/=now >>> RSS Feed: >>> https://www.listbox.com/member/archive/rss/182191/22567878-8480fd5f >>> Modify Your Subscription: >>> >> https://www.listbox.com/member/?& >>> f5b912c9 Powered by Listbox: http://www.listbox.com >> _______________________________________________ >> zfs-devel@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/zfs-devel >> To unsubscribe, send any mail to "zfs-devel-unsubscribe@freebsd.org" > > ------------------------------------------- > illumos-zfs > Archives: https://www.listbox.com/member/archive/182191/=now > RSS Feed: https://www.listbox.com/member/archive/rss/182191/22567878-8480fd5f > Modify Your Subscription: https://www.listbox.com/member/?member_id=22567878&id_secret=22567878-f5b912c9 > Powered by Listbox: http://www.listbox.com From owner-freebsd-fs@freebsd.org Wed Oct 21 08:07:37 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FE72A1A279 for ; Wed, 21 Oct 2015 08:07:37 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB6B017A0 for ; Wed, 21 Oct 2015 08:07:36 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wicll6 with SMTP id ll6so62112518wic.1 for ; Wed, 21 Oct 2015 01:07:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type; bh=nuRIeMbXTj7Fn0cEHL6xaNw8kdsPd3EOJpCdtgOuMLY=; b=UWfj2eXZyAK9Z9wAVNyM9d2XcTAtWOlPd9kGtXzHul/H6BIjoAoj8iiKZ2DVstmMyC 7u123YUc+DjXp9jpqf2I8pwIgMvFQnnFhWlZyw6r9XlXJkXSRtJ/XZsuda9pr1vkdiRM sRz7FwCgAcnVTSHIGpAdtSOzUGrX8SGSELVGTTY932MVMywH9GlsgdJzjzqF+FQJLBI5 lCMqZ0VWUZ5X/AmBK/eAJntAprN0wDr+AjJ6uBIOU7bB60YJyG6hwZHdjwqa/cYC1byo 2XKbFABxrfSSPRe7Nc+Td/CX3qh4I34Epq3h6zRQPjdSTq2AN8lFr4sKwYshTqX1H+rK nKpA== X-Gm-Message-State: ALoCoQkPTojbbs8lI9DlHh3Bhltny8CkX+nBjE+cdB1alIH58RbeYE1whks7+mdpJqenz30g5Az+ X-Received: by 10.194.178.196 with SMTP id da4mr10655879wjc.41.1445414849480; Wed, 21 Oct 2015 01:07:29 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id lv4sm8705910wjb.43.2015.10.21.01.07.28 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 21 Oct 2015 01:07:28 -0700 (PDT) Subject: Re: [developer] RE: [zfs] RE: granularity of performance penalty from resilvering To: Fred Liu , Jason Matthews , "zfs@lists.illumos.org" , "zfs-discuss@list.zfsonlinux.org" , developer , freebsd-fs , zfs-discuss , "developer@lists.illumos.org" References: <5626E4B3.2020904@broken.net> From: Steven Hartland Message-ID: <562747C6.7010802@multiplay.co.uk> Date: Wed, 21 Oct 2015 09:07:34 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 08:07:37 -0000 These are availabe on FreeBSD as well via sysctls: * vfs.zfs.resilver_delay * vfs.zfs.scrub_delay * vfs.zfs.top_maxinflight * vfs.zfs.resilver_min_time_ms Regards Steve On 21/10/2015 02:35, Fred Liu wrote: > > Yeah. I want to go the other way. Plus, these settings are only > applicable in illumos. > > Therefore I decide to give up the hybrid( ssd+sata) solution to > underpin applications which need decent RAS. > > I am gonna go all-flash array. > > Thanks. > > Fred > > *From:*Jason Matthews [mailto:jason@broken.net] > *Sent:* 星期三, 十月21, 2015 9:05 > *To:* zfs@lists.illumos.org; Fred Liu; > zfs-discuss@list.zfsonlinux.org; developer; freebsd-fs; zfs-discuss > *Subject:* Re: [zfs] RE: granularity of performance penalty from > resilvering > > > > you could look at these tunables (not the settings themselves)... > > these settings actually make resilvers have a higher priority. You > obviously would want to go the other way. > > j. > > |* Prioritize resilvering by setting the delay to zero| > |set zfs:zfs_resilver_delay = 0 | > > * Prioritize scrubs by setting the delay to zero > set zfs:zfs_scrub_delay = 0 > > |* resilver for five seconds per TXG| > |set zfs:zfs_resilver_min_time_ms = 5000| > > > |echo zfs_resilver_delay/w0 | mdb -kw| > |echo zfs_scrub_delay/w0 |mdb -kw| > |echo zfs_top_maxinflight/w7f |mdb -kw| > |echo zfs_resilver_min_time_ms/w1388 |mdb -kw| > > On 10/19/2015 11:49 PM, Fred Liu wrote: > > Yes, “zpool scrub –s” can stop the resilvering. > > *From:*Fred Liu > *Sent:* 星 期二, 十月20, 2015 12:15 > *To:* 'zfs-discuss@list.zfsonlinux.org > '; developer; illumos-zfs; > freebsd-fs; zfs-discuss > *Subject:* granularity of performance penalty from resilvering > > Sorry if is a duplicate thread. > > The last suffering has been lasted for two weeks for we replaced a > 6TB HDD. > > There should be some IO throttle measure from ZFS software stack. > At least, we can try to stop resilvering like scrubbing > > if the realization is quiet complicated. > > Besides that, will nice zil/cache be relief? > > Thanks. > > Fred > > *illumos-developer* | Archives > > > | Modify > > Your Subscription [Powered by Listbox] > From owner-freebsd-fs@freebsd.org Thu Oct 22 04:24:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBA0CA1BBED for ; Thu, 22 Oct 2015 04:24:27 +0000 (UTC) (envelope-from fred.fliu@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B572A15DF for ; Thu, 22 Oct 2015 04:24:27 +0000 (UTC) (envelope-from fred.fliu@gmail.com) Received: by obbwb3 with SMTP id wb3so58984816obb.0 for ; Wed, 21 Oct 2015 21:24:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=pTJIc1RgskkpAEU//98oD0t6+8e7eiCTFvjqS+T+oys=; b=HzaGloYU7U31k1wqup51ZYO/qtGp1FJgkDo2JgSYK3HQSk04VaKlo+MXHY8pd+P+Li T79n5rJ1kfudH8v4l4OUeJvNJa1ZLvJ/qpcYY8/N8Xcy+wx5d/p2l75FxN+bA7BObxhx UPMoNaVEK1n3YFkpII/W+MilCppYTE3N0l3w7NqAeenBrc/8ibzy3kYWdDeLaRrRpqhT CS6u9iCXLHBf148/h4UaKy22lHPtbhb/jgXnYGN5ZP8liyMESxCTUtK8/okIRlvFIdrz fjQmg3aLBlbaRSstJ9XhpGjQF0Tef1jmeRvL5y6yP9CRAb6BfQqHdHvm71s02g8ctp7v w2vw== MIME-Version: 1.0 X-Received: by 10.60.142.227 with SMTP id rz3mr677497oeb.30.1445487866972; Wed, 21 Oct 2015 21:24:26 -0700 (PDT) Received: by 10.202.173.216 with HTTP; Wed, 21 Oct 2015 21:24:26 -0700 (PDT) In-Reply-To: References: Date: Thu, 22 Oct 2015 12:24:26 +0800 Message-ID: Subject: Fwd: FW: granularity of performance penalty from resilvering From: Fred Liu To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 04:24:28 -0000 ---------- Forwarded message ---------- From: Fred Liu Date: 2015-10-22 12:21 GMT+08:00 Subject: FW: granularity of performance penalty from resilvering To: "fred.fliu@gmail.com" _____________________________________________ From: Microsoft Exchange Sent: =E6=98=9F=E6=9C=9F=E5=9B=9B, =E5=8D=81=E6=9C=88 22, 2015 12:14 To: Fred Liu Subject: =E6=9C=AA=E9=80=81=E8=BE=BE: granularity of performance penalty fr= om resilvering =E5=90=91=E8=BF=99=E4=BA=9B=E6=94=B6=E4=BB=B6=E4=BA=BA=E6=88=96=E9=80=9A=E8= =AE=AF=E7=BB=84=E5=88=97=E8=A1=A8=E4=BC=A0=E9=80=92=E9=82=AE=E4=BB=B6=E5=A4= =B1=E8=B4=A5: freebsd-fs Microsoft Exchange =E5=B7=B2=E5=B0=9D=E8=AF=95=E4=BC=A0=E9=80=92=E6=AD=A4= =E9=82=AE=E4=BB=B6=EF=BC=8C=E4=BD=86=E6=9C=AA=E6=88=90=E5=8A=9F=EF=BC=8C=E5= =B7=B2=E5=81=9C=E6=AD=A2=E5=B0=9D=E8=AF=95=E3=80=82=E8=AF=B7=E7=A8=8D=E5=90= =8E=E5=86=8D=E6=AC=A1=E5=B0=9D=E8=AF=95=E5=8F=91=E9=80=81=E6=AD=A4=E9=82=AE= =E4=BB=B6=EF=BC=8C=E6=88=96=E5=90=91=E7=B3=BB=E7=BB=9F=E7=AE=A1=E7=90=86=E5= =91=98=E6=8F=90=E4=BE=9B=E4=BB=A5=E4=B8=8B=E8=AF=8A=E6=96=AD=E6=96=87=E6=9C= =AC=E3=80=82 _____ =E9=80=9A=E8=BF=87 Microsoft Exchange Server 2007 =E5=8F=91=E9=80=81 =E4=BE=9B=E7=AE=A1=E7=90=86=E5=91=98=E4=BD=BF=E7=94=A8=E7=9A=84=E8=AF=8A=E6= =96=AD=E4=BF=A1=E6=81=AF: =E7=94=9F=E6=88=90=E6=9C=8D=E5=8A=A1=E5=99=A8: SH-CAS.ISSI.COM freebsd-fs@freebsd.org #550 4.4.7 QUEUE.Expired; message expired ## =E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6=E6=A0=87=E9=A2=98: Received: from SH-MAIL.ISSI.COM ([169.254.1.216]) by SH-CAS.ISSI.COM ([::1]= ) with mapi; Mon, 19 Oct 2015 21:14:41 -0700 From: Fred Liu To: "zfs-discuss@list.zfsonlinux.org" , developer , illumos-zfs , freebsd-fs , zfs-discuss Date: Mon, 19 Oct 2015 21:14:59 -0700 Subject: granularity of performance penalty from resilvering Thread-Topic: granularity of performance penalty from resilvering Thread-Index: AdEJxgqT8nDB80fTSNyo/vaI2njIVABJqEaA Message-ID: References: In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: zh-CN, en-US Content-Type: multipart/alternative; boundary=3D"_000_A5A6EA4AE9DCC44F8E7FCB4D6317B1D202B43A690D8DSHMAIL= ISSIC_" MIME-Version: 1.0 From owner-freebsd-fs@freebsd.org Thu Oct 22 22:31:08 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A060A1CCD7 for ; Thu, 22 Oct 2015 22:31:08 +0000 (UTC) (envelope-from fusionfoto@gmail.com) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CBE214A6 for ; Thu, 22 Oct 2015 22:31:08 +0000 (UTC) (envelope-from fusionfoto@gmail.com) Received: by qgeo38 with SMTP id o38so70633712qge.0 for ; Thu, 22 Oct 2015 15:31:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=ykc01bcSSWO6ZKqkKzf2Bm5X19IluFfTX6CN4k2lsPU=; b=m/6kZsw+GSHkS+IBi8tKvH5Kuq2I42f7xWskEddoikKBVPJNcrJovquI9drG+TYOBT XauQFASbMYK5MFDYolCJ/avQhei/OXN61KmrvLbiC0RhnJB0my5lbzg3T9vkNCVbRkqn 4fE3Im81NGUpIZ0sZEYkvgkRiXOmrHBtZxNaHvk73sIpAWjbDszqu6xMmFV452CITBHO ABdbjmhcNVpSOjm8HZ5vIrCBTnJ8LrkVgxqXAhPylQlNi/bcrg0R7J4LUIt9MeNdU54m 42BuZiHmf7JAmXmUplGVJDEHXHY8BrE2sJjang02wC3wTURg8bWYgV00Y97wuynkp/hE fnYg== MIME-Version: 1.0 X-Received: by 10.140.86.42 with SMTP id o39mr21315150qgd.102.1445553067148; Thu, 22 Oct 2015 15:31:07 -0700 (PDT) Received: by 10.55.57.70 with HTTP; Thu, 22 Oct 2015 15:31:07 -0700 (PDT) Date: Thu, 22 Oct 2015 18:31:07 -0400 Message-ID: Subject: Adjusting zvol_immediate_write_sz From: FF To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Oct 2015 22:31:08 -0000 I couldn't find anything on google doing this with FreeBSD. Without getting into the reasons for why someone may want to alter the threshold for varying the way the ZIL interacts with writes. Is there any obvious reason why creating a boot-tunable or runtime-tunable for this variable in zvol.c is a bad thing? Would a set command in the loader already override this without any additional code otherwise I'm happy to write the code and ask for it to be committed. Thanks! -- FF From owner-freebsd-fs@freebsd.org Fri Oct 23 07:42:48 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69E9EA1960F for ; Fri, 23 Oct 2015 07:42:48 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F32DAE39 for ; Fri, 23 Oct 2015 07:42:47 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wikq8 with SMTP id q8so64346346wik.1 for ; Fri, 23 Oct 2015 00:42:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=2nvHppcRj2I8HmDAcz41HiBQQ5YUijGHhtYXjx+0rTY=; b=ijsLGAgLGkx34HPBDlv8zWQHjnafesaiU5FZ8h9HuIylaH74ZAzlRsWSDsaD053ksY 0vNdrrGeK8klN6fI6PBBGzabvbQocy4KJpCyE30qafp5HdT4Aq3nxm8WEXuPUlI7GtdM thA4A/U95NvZ6MJE8hg3u53hpvwpU8a6rZGFPekDEaY0ItdpSNY8aRCIml/351/ZJf44 Mx15mC2Z3GKr91EV63fIBIrNgMND7sDDWZJrSBUWE5474GrqxDkRdNQ4FIde04mm/9ZD 9BzUHo7R+5KGPkmLn6+1il2Ickhqe0IWctd+1kmHnc2UX8HuKb2z56dy08zy6SPRrA6+ To8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=2nvHppcRj2I8HmDAcz41HiBQQ5YUijGHhtYXjx+0rTY=; b=LecipN6uxGDzp22MXi5Oq9Q7PMOW/pbES4ll+H6QGdr3MGRSECpMOIaUWVaEaoLSTy KZduiCPe4qud1aZD4PthQiCkJ4G5Y49wClwvlFzNKRXHJyqSoWxTPX0GVgSXg1kUwE2C WyslEndG1fSPJRJvSYmcoRh8cZf5XxjW0CVsPZp15wkQhD5iUXfpPUSy9BaCsajjiVlh 4RLYLthd0OmkBlQ/WnYII+epTxoGwm7oRBiBBwKE3V4HTt4bLibb0SxgHBJafcWlgDn5 SPRb1J5Mx2bggBqALj434o6vHelZ0gxZ+8A3pCaKFMZbTqoUGjWrkQvhu37uARGawzPQ CowA== X-Gm-Message-State: ALoCoQn5/kioG+hWGBiItQ9kgvBRttrwnsgZ02XdYNWc2d9Gq2/cgfr5wwan/lDzdw+gFgBWDrww X-Received: by 10.194.190.19 with SMTP id gm19mr3321591wjc.0.1445586166220; Fri, 23 Oct 2015 00:42:46 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id t7sm1965371wib.1.2015.10.23.00.42.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Oct 2015 00:42:45 -0700 (PDT) Subject: Re: Adjusting zvol_immediate_write_sz To: freebsd-fs@freebsd.org References: From: Steven Hartland Message-ID: <5629E4F5.3030500@multiplay.co.uk> Date: Fri, 23 Oct 2015 08:42:45 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 07:42:48 -0000 I don't see why not FF, if you would like to throw us a quick patch then I'd be happy to review and commit. On 22/10/2015 23:31, FF wrote: > I couldn't find anything on google doing this with FreeBSD. > > Without getting into the reasons for why someone may want to alter the > threshold for varying the way the ZIL interacts with writes. Is there any > obvious reason why creating a boot-tunable or runtime-tunable for this > variable in zvol.c is a bad thing? > > Would a set command in the loader already override this without any > additional code otherwise I'm happy to write the code and ask for it to be > committed. > > Thanks! > From owner-freebsd-fs@freebsd.org Fri Oct 23 12:04:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F71CA1C89F for ; Fri, 23 Oct 2015 12:04:40 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E29FF1EB6; Fri, 23 Oct 2015 12:04:39 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:0L5tLBa6E2JNMRqUmnH9/pv/LSx+4OfEezUN459isYplN5qZpcu8bnLW6fgltlLVR4KTs6sC0LqL9fi+EjBfqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh730o8OYP1oArQH+SI0xBS3+lR/WuMgSjNkqAYcK4TyNnEF1ff9Lz3hjP1OZkkW0zM6x+Jl+73YY4Kp5pIZoGJ/3dKUgTLFeEC9ucyVsvJWq5lH/Sl6h/HYReF462jRzS1zL9hz3VIz99yXhnuRn1SSQJsGwSqo7D2eM9aBuHSXpgyRPEjcy82Xaj4QklqdSqxGlqhlX3onbfYyRLPo4daqLLoBSfnZIQssED38JOYi7dYZaSrNZZes= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2B7AgC+ISpW/61jaINehApvBr4jDoFZGYYEAoF/FAEBAQEBAQEBgQmCK4IJBSMEUhIBIBEZAgRVAgSIQ7JhkmEBAQEBAQEEAQEBAQEBAQESCYZ3iUAWATQHgmmBRQWWK4JOgkuJXUiDd4MkkmkCHwFDghEdgXEiNIU9gQYBAQE X-IronPort-AV: E=Sophos;i="5.20,186,1444708800"; d="scan'208";a="246411735" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 23 Oct 2015 08:04:24 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 0D4E715F55D; Fri, 23 Oct 2015 08:04:25 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id EZI_MB8O2F7D; Fri, 23 Oct 2015 08:04:24 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 07B5815F565; Fri, 23 Oct 2015 08:04:24 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Tb-99-PTqemN; Fri, 23 Oct 2015 08:04:23 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id D9EE215F55D; Fri, 23 Oct 2015 08:04:23 -0400 (EDT) Date: Fri, 23 Oct 2015 08:04:23 -0400 (EDT) From: Rick Macklem To: FreeBSD FS Cc: Josh Paetzel , Alexander Motin , ken@freebsd.org Message-ID: <1927021065.49882210.1445601863864.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <2144282175.49880310.1445601737139.JavaMail.zimbra@uoguelph.ca> Subject: NFS FHA issue and possible change to the algorithm MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_49882208_271473258.1445601863862" X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: NFS FHA issue and possible change to the algorithm Thread-Index: pEHcTY22qkgnw6m7/neUQejW0d+Nlw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 12:04:40 -0000 ------=_Part_49882208_271473258.1445601863862 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, An off list discussion occurred where a site running an NFS server found that they needed to disable File Handle Affinity (FHA) to get good performance. Here is a re-post of some of that (with Josh's permission): First what was observed w.r.t. the machine. Josh Paetzel wrote: >>>> It's all good. >>>> >>>> It's a 96GB RAM machine and I have 2 million nmbclusters, so 8GB RAM, >>>> and we've tried 1024 NFS threads. >>>> >>>> It might be running out of network memory but we can't really afford to >>>> give it any more, for this use case disabling FHA might end up being the >>>> way to go. >>>> I wrote: >>> Just to fill mav@ in, the person that reported a serious performance >>> problem >>> to Josh was able to fix it by disabling FHA. Josh Paetzel wrote: >> >> There's about 300 virtual machines that mount root from a read only NFS >> share. >> >> There's also another few hundred users that mount their home directories >> over NFS. When things went sideways it is always the virtual machines >> that get unusable. 45 seconds to log in via ssh, 15 minutes to boot, >> stuff like that. >> >> root@head2] ~# nfsstat -s 1 >> GtAttr Lookup Rdlink Read Write Rename Access Rddir >> 4117 17 0 124 689 4 680 0 >> 4750 31 5 121 815 3 950 1 >> 4168 16 0 109 659 9 672 0 >> 4416 24 0 112 771 3 748 0 >> 5038 86 0 76 728 4 825 0 >> 5602 21 0 76 740 3 702 6 >> >> [root@head2] ~# arcstat.py 1 >> time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c >> 18:25:36 21 0 0 0 0 0 0 0 0 65G 65G >> 18:25:37 1.8K 23 1 23 1 0 0 7 0 65G 65G >> 18:25:38 1.9K 88 4 32 1 56 32 3 0 65G 65G >> 18:25:39 2.2K 67 3 62 2 5 5 2 0 65G 65G >> 18:25:40 2.7K 132 4 39 1 93 17 8 0 65G 65G >> >> last pid: 7800; load averages: 1.44, 1.65, 1.68 >> up >> 0+19:22:29 18:26:16 >> 69 processes: 1 running, 68 sleeping >> CPU: 0.1% user, 0.0% nice, 1.8% system, 0.9% interrupt, 97.3% idle >> Mem: 297M Active, 180M Inact, 74G Wired, 140K Cache, 565M Buf, 19G Free >> ARC: 66G Total, 39G MFU, 24G MRU, 53M Anon, 448M Header, 1951M Other >> Swap: 28G Total, 28G Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 9915 root 37 52 0 9900K 2060K rpcsvc 16 16.7H 24.02% nfsd >> 6402 root 1 52 0 85352K 20696K select 8 47:17 3.08% >> python2.7 >> 43178 root 1 20 0 70524K 30752K select 7 31:04 0.59% >> rsync >> 7363 root 1 20 0 49512K 6456K CPU16 16 0:00 0.59% top >> 37968 root 1 20 0 70524K 31432K select 7 16:53 0.00% >> rsync >> 37969 root 1 20 0 55752K 11052K select 1 9:11 0.00% ssh >> 13516 root 12 20 0 176M 41152K uwait 23 4:14 0.00% >> collectd >> 31375 root 12 20 0 176M 42432K uwa >> >> This is a quick peek at the system at the end of the day, so load has >> dropped off considerably, however the main takeaway is it has plenty of >> free RAM, and ZFS ARC hit percentage is > 99%. >> I wrote: >>> I took a look at it and I wonder if it is time to consider changing the >>> algorithm >>> somewhat? >>> >>> The main thing that I wonder about is doing FHA for all the RPCs other than >>> Read and Write. >>> >>> In particular, Getattr is often the most frequent RPC and doing FHA for it >>> seems >>> like wasted overhead to me? Normally separate Getattr RPCs wouldn't be done >>> for >>> FHs are being Read/Written, since the Read/Write reply has updated >>> attributes in it. >>> Although the load is mostly Getattr RPCs and I think the above statement is correct, I don't know if the overhead of doing FHA for all the Getattr RPCs explains the observed performance problem? I don't see how doing FHA for RPCs like Getattr will improve their performance. Note that when the FHA algorithm was originally done, there wasn't a shared vnode lock and, as such, all RPCs on a given FH/vnode would have been serialized by the vnode lock anyhow. Now, with shared vnode locks, this isn't the case for frequently performed RPCs like Getattr, Read (Write for ZFS), Lookup and Access. I have always felt that doing FHA for RPCs other than Read and Write didn't make much sense to me, but I don't have any evidence that it causes a significant performance penalty. Anyhow, the attached simple patch limits FHA to Read and Write RPCs. The simple testing I've done shows it to be about performance neutral (0-1% improvement), but I have only small hardware and no ZFS or any easy way to emulate a load of mostly Getattr RPCs. As such, unless others can determine if this patch (or some other one) helps w.r.t. this, I don't think committing it makes much sense? If anyone can test this or have comments w.r.t. this or suggestions for other possible changes to the FHA algorithm, please do so. Thanks, rick ------=_Part_49882208_271473258.1445601863862 Content-Type: text/x-patch; name=nfsfha.patch Content-Disposition: attachment; filename=nfsfha.patch Content-Transfer-Encoding: base64 LS0tIG5mcy9uZnNfZmhhLmMuc2F2CTIwMTUtMTAtMjEgMTk6Mjk6NTMuMDAwMDAwMDAwIC0wNDAw CisrKyBuZnMvbmZzX2ZoYS5jCTIwMTUtMTAtMjIgMTk6MzE6NDMuMDAwMDAwMDAwIC0wNDAwCkBA IC00Miw2ICs0Miw4IEBAIF9fRkJTRElEKCIkRnJlZUJTRDogaGVhZC9zeXMvbmZzL25mc19maGEK IAogc3RhdGljIE1BTExPQ19ERUZJTkUoTV9ORlNfRkhBLCAiTkZTIEZIQSIsICJORlMgRkhBIik7 CiAKK3N0YXRpYyB1X2ludAluZnNmaGFfZW5hYmxlYWxscnBjcyA9IDA7CisKIC8qCiAgKiBYWFgg bmVlZCB0byBjb21tb25pemUgZGVmaW5pdGlvbnMgYmV0d2VlbiBvbGQgYW5kIG5ldyBORlMgY29k ZS4gIERlZmluZQogICogdGhpcyBoZXJlIHNvIHdlIGRvbid0IGluY2x1ZGUgb25lIG5mc3Byb3Rv Lmggb3ZlciB0aGUgb3RoZXIuCkBAIC0xMDksNiArMTExLDEwIEBAIGZoYV9pbml0KHN0cnVjdCBm aGFfcGFyYW1zICpzb2Z0YykKIAkgICAgT0lEX0FVVE8sICJmaGVfc3RhdHMiLCBDVExUWVBFX1NU UklORyB8IENUTEZMQUdfUkQsIDAsIDAsCiAJICAgIHNvZnRjLT5jYWxsYmFja3MuZmhlX3N0YXRz X3N5c2N0bCwgIkEiLCAiIik7CiAKKwlTWVNDVExfQUREX1VJTlQoJnNvZnRjLT5zeXNjdGxfY3R4 LCBTWVNDVExfQ0hJTERSRU4oc29mdGMtPnN5c2N0bF90cmVlKSwKKwkgICAgT0lEX0FVVE8sICJl bmFibGVfYWxscnBjcyIsIENUTEZMQUdfUlcsCisJICAgICZuZnNmaGFfZW5hYmxlYWxscnBjcywg MCwgIkVuYWJsZSBGSEEgZm9yIGFsbCBSUENzIik7CisKIH0KIAogdm9pZApAQCAtMzgzLDYgKzM4 OSw3IEBAIGZoYV9hc3NpZ24oU1ZDVEhSRUFEICp0aGlzX3RocmVhZCwgc3RydWMKIAlzdHJ1Y3Qg ZmhhX2luZm8gaTsKIAlzdHJ1Y3QgZmhhX2hhc2hfZW50cnkgKmZoZTsKIAlzdHJ1Y3QgZmhhX2Nh bGxiYWNrcyAqY2I7CisJcnBjcHJvY190IHByb2NudW07CiAKIAljYiA9ICZzb2Z0Yy0+Y2FsbGJh Y2tzOwogCkBAIC0zOTksNiArNDA2LDI0IEBAIGZoYV9hc3NpZ24oU1ZDVEhSRUFEICp0aGlzX3Ro cmVhZCwgc3RydWMKIAlpZiAocmVxLT5ycV92ZXJzICE9IDIgJiYgcmVxLT5ycV92ZXJzICE9IDMp CiAJCWdvdG8gdGhpc3Q7CiAKKwkvKgorCSAqIFRoZSBtYWluIHJlYXNvbiBmb3IgdXNlIG9mIEZI QSBub3cgdGhhdCBGcmVlQlNEIHN1cHBvcnRzIHNoYXJlZAorCSAqIHZub2RlIGxvY2tzIGlzIHRv IHRyeSBhbmQgbWFpbnRhaW4gc2VxdWVudGlhbCBvcmRlcmluZyBvZiBSZWFkCisJICogYW5kIFdy aXRlIG9wZXJhdGlvbnMuICBBbHNvLCBpdCBoYXMgYmVlbiBvYnNlcnZlZCB0aGF0IHNvbWUKKwkg KiBSUEMgbG9hZHMsIHN1Y2ggYXMgb25lIG1vc3RseSBvZiBHZXRhdHRyIFJQQ3MsIHBlcmZvcm0g YmV0dGVyCisJICogd2l0aG91dCBGSEEgYXBwbGllZCB0byB0aGVtLiAgQXMgc3VjaCwgRkhBIGlz IG9ubHkgYXBwbGllZCB0bworCSAqIFJlYWQgYW5kIFdyaXRlIFJQQ3MgYnkgZGVmYXVsdC4KKwkg KiBUaGUgc3lzY3RsICJmaGEuZW5hYmxlX2FsbHJwY3MiIGNhbiBiZSBzZXQgbm9uemVybyBzbyB0 aGF0IEZIQSBpcworCSAqIGFwcGxpZWQgdG8gYWxsIFJQQ3MgZm9yIGJhY2t3YXJkcyBjb21wYXRp YmlsaXR5IHdpdGggdGhlIG9sZCBGSEEKKwkgKiBjb2RlLgorCSAqLworCXByb2NudW0gPSByZXEt PnJxX3Byb2M7CisJaWYgKHJlcS0+cnFfdmVycyA9PSAyKQorCQlwcm9jbnVtID0gY2ItPmdldF9w cm9jbnVtKHByb2NudW0pOworCWlmIChjYi0+aXNfcmVhZChwcm9jbnVtKSA9PSAwICYmIGNiLT5p c193cml0ZShwcm9jbnVtKSA9PSAwICYmCisJICAgIG5mc2ZoYV9lbmFibGVhbGxycGNzID09IDAp CisJCWdvdG8gdGhpc3Q7CisKIAlmaGFfZXh0cmFjdF9pbmZvKHJlcSwgJmksIGNiKTsKIAogCS8q Cg== ------=_Part_49882208_271473258.1445601863862-- From owner-freebsd-fs@freebsd.org Fri Oct 23 12:10:40 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C014A1CA45 for ; Fri, 23 Oct 2015 12:10:40 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (wsip-70-169-168-7.pn.at.cox.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB0E420F for ; Fri, 23 Oct 2015 12:10:39 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id A777D133591 for ; Fri, 23 Oct 2015 07:10:31 -0500 (CDT) Subject: Re: Panic in ZFS during zfs recv (while snapshots being destroyed) To: freebsd-fs@freebsd.org References: <55BB443E.8040801@denninger.net> <55CF7926.1030901@denninger.net> <55DF7191.2080409@denninger.net> <55DF76AA.3040103@denninger.net> <561FB7D0.4080107@denninger.net> <562662ED.1070302@denninger.net> From: Karl Denninger X-Enigmail-Draft-Status: N1110 Message-ID: <562A23B0.4050306@denninger.net> Date: Fri, 23 Oct 2015 07:10:24 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <562662ED.1070302@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms080404050900010009090303" X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 12:10:40 -0000 This is a cryptographically signed message in MIME format. --------------ms080404050900010009090303 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable While continuing to run this down I think I've managed to duplicate the state that produces the panic, but I don't know exactly how or why. The script (modified to check all returns) now produced this on a test ca= se: =2E..... receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-04h00 into backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-04h00 received 559MB stream in 22 seconds (25.4MB/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-23-04h04 into backup/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-23-04h04 received 4.25MB stream in 1 seconds (4.25MB/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-05h00 into backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-05h00 received 12.5MB stream in 1 seconds (12.5MB/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-06h00 into backup/R/10.2-STABLE@zfs-auto-snap_frequent-2015-10-23-06h00 received 13.4MB stream in 1 seconds (13.4MB/sec) receiving incremental stream of zsr/R/10.2-STABLE@zfs-base into backup/R/10.2-STABLE@zfs-base received 14.8MB stream in 1 seconds (14.8MB/sec) *will destroy zsr/R/10.2-STABLE@zfs-old** **will reclaim 6.50M** **cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy** **Snapshot destroy FAILED with code 1* And there it stopped, as I have it trapped. But, there's nothing holding that dataset open: root@NewFS:~ # zfs holds zsr/R/10.2-STABLE@zfs-old NAME TAG TIMESTAMP There is also no send or receive command still running, and an attempt to force (or defer) the destroy fails too: root@NewFS:~ # zfs destroy zsr/R/10.2-STABLE@zfs-old cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy root@NewFS:~ # zfs destroy -d zsr/R/10.2-STABLE@zfs-old cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy root@NewFS:~ # zfs destroy -f zsr/R/10.2-STABLE@zfs-old cannot destroy snapshot zsr/R/10.2-STABLE@zfs-old: dataset is busy Parameters: root@NewFS:~ # zfs get all zsr/R/10.2-STABLE@zfs-old NAME PROPERTY VALUE =20 SOURCE zsr/R/10.2-STABLE@zfs-old type snapshot = - zsr/R/10.2-STABLE@zfs-old creation Thu Oct 22 10:14 2015 = - zsr/R/10.2-STABLE@zfs-old used 6.50M = - zsr/R/10.2-STABLE@zfs-old referenced 25.7G = - zsr/R/10.2-STABLE@zfs-old compressratio 1.86x = - zsr/R/10.2-STABLE@zfs-old devices on =20 default zsr/R/10.2-STABLE@zfs-old exec on =20 default zsr/R/10.2-STABLE@zfs-old setuid on =20 default zsr/R/10.2-STABLE@zfs-old xattr off =20 temporary zsr/R/10.2-STABLE@zfs-old version 5 = - zsr/R/10.2-STABLE@zfs-old utf8only off = - zsr/R/10.2-STABLE@zfs-old normalization none = - zsr/R/10.2-STABLE@zfs-old casesensitivity sensitive = - zsr/R/10.2-STABLE@zfs-old nbmand off =20 default zsr/R/10.2-STABLE@zfs-old primarycache all =20 default zsr/R/10.2-STABLE@zfs-old secondarycache all =20 default zsr/R/10.2-STABLE@zfs-old defer_destroy off = - zsr/R/10.2-STABLE@zfs-old userrefs 0 = - zsr/R/10.2-STABLE@zfs-old mlslabel = - zsr/R/10.2-STABLE@zfs-old refcompressratio 1.86x = - zsr/R/10.2-STABLE@zfs-old written 169M = - zsr/R/10.2-STABLE@zfs-old clones = - zsr/R/10.2-STABLE@zfs-old logicalused 0 = - zsr/R/10.2-STABLE@zfs-old logicalreferenced 42.7G = - zsr/R/10.2-STABLE@zfs-old volmode default =20 default zsr/R/10.2-STABLE@zfs-old com.sun:auto-snapshot true =20 inherited from zsr/R/10.2-STABLE Nothing here that looks like a problem; specifically, no clones. If I run the script again and it attempts recovery (since zfs-old is present) the machine panics with the below. Once the machine has panic'd I can remove the snapshot. This looks like some sort of bug internally in the zfs state -- but the question is, now that I'm in this state how do I get out without a crash and why did it happen, given that I can't find any reason for the snapshot to be non-removable (e.g. an active clone, etc) Ideas or anything that would help find the source of the problem using zd= b? On 10/20/2015 10:51, Karl Denninger wrote: > More data on this crash from this morning; I caught it in-process this > time and know exactly where it was in the backup script when it detonat= ed. > > Here's the section of the script that was running when it blew up: > > for i in $ZFS > do > echo Processing $i > > FILESYS=3D`echo $i|cut -d/ -f2` > > zfs list $i@zfs-base >/dev/null 2>&1 > result=3D$? > if [ $result -eq 1 ]; > then > echo "Make and send zfs-base snapshot" > zfs snapshot -r $i@zfs-base > zfs send -R $i@zfs-base | zfs receive -Fuvd $BACKUP > else > base_only=3D`zfs get -H com.backup:base-only $i|grep tr= ue` > result=3D$? > if [ $result -eq 1 ]; > then > echo "Bring incremental backup up to date" > old_present=3D`zfs list $i@zfs-old >/dev/null 2= >&1` > old=3D$? > if [ $old -eq 0 ]; > then > echo "Previous incremental sync was > interrupted; resume" > else > zfs rename -r $i@zfs-base $i@zfs-old > zfs snapshot -r $i@zfs-base > fi > zfs send -RI $i@zfs-old $i@zfs-base | zfs > receive -Fudv $BACKUP > result=3D$? > if [ $result -eq 0 ]; > then > zfs destroy -r $i@zfs-old > zfs destroy -r $BACKUP/$FILESYS@zfs-old= > else > echo "STOP - - ERROR RUNNING COPY on $i= !!!!" > exit 1 > fi > else > echo "Base-only backup configured for $i" > fi > fi > echo > done > > And the output on the console when it happened: > > Begin local ZFS backup by SEND > Run backups of default [zsr/R/10.2-STABLE zsr/ssd-txlog zs/archive > zs/colo-archive zs/disk zs/pgsql zs/pics dbms/ticker] > Tue Oct 20 10:14:09 CDT 2015 > > Import backup pool > Imported; ready to proceed > Processing zsr/R/10.2-STABLE > Bring incremental backup up to date > _*Previous incremental sync was interrupted; resume*_ > attempting destroy backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-13-= 00h07 > success > attempting destroy > backup/R/10.2-STABLE@zfs-auto-snap_hourly-2015-10-18-12h04 > success > *local fs backup/R/10.2-STABLE does not have fromsnap (zfs-old in strea= m);** > **must have been deleted locally; ignoring* > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 into > backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 > snap backup/R/10.2-STABLE@zfs-auto-snap_daily-2015-10-18-00h07 already > exists; ignoring > received 0B stream in 1 seconds (0B/sec) > receiving incremental stream of > zsr/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 into > backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 > snap backup/R/10.2-STABLE@zfs-auto-snap_weekly-2015-10-18-00h14 already= > exists; ignoring > received 0B stream in 1 seconds (0B/sec) > receiving incremental stream of zsr/R/10.2-STABLE@zfs-base into > backup/R/10.2-STABLE@zfs-base > snap backup/R/10.2-STABLE@zfs-base already exists; ignoring > > That bolded pair of lines should _*not*_ be there. It means that the > snapshot "zfs-old" is on the source volume, but it shouldn't be there > because after the copy is complete we destroy it on both the source and= > destination. Further, when it is attempted to be sent while the > snapshot name (zfs-old) was found in a zfs list /*the data allegedly > associated with the phantom snapshot that shouldn't exist was not there= > */(second bolded line) > > zfs send -RI $i@zfs-old $i@zfs-base | zfs > receive -Fudv $BACKUP > result=3D$? > if [ $result -eq 0 ]; > then > *zfs destroy -r $i@zfs-old** > ** zfs destroy -r $BACKUP/$FILESYS@zfs-o= ld* > else > echo "STOP - - ERROR RUNNING COPY on $i= !!!!" > exit 1 > fi > > I don't have the trace from the last run (I didn't save it) *but there > were no errors returned by it *as it was run by hand (from the console)= > while I was watching it. > > This STRONGLY implies that the zfs destroy /allegedly /succeeded (it ra= n > without an error and actually removed the snapshot) but left the > snapshot _*name*_ on the volume as it was able to be found by the > script, and then when that was referenced by the backup script in the > incremental send the data set was invalid producing the resulting panic= =2E > > The bad news is that the pool shows no faults and a scrub which took > place a few days ago says the pool is fine. If I re-run the backup I > suspect it will properly complete (it always has in the past as a resum= e > from the interrupted one) but clearly, whatever is going on here looks > like some sort of race in the destroy commands (which _*are*_ being run= > recursively) that leaves the snapshot name on the volume but releases > the data stored in it, thus the panic when it is attempted to be > referenced. > > I have NOT run a scrub on the pool in an attempt to preserve whatever > evidence may be there; if there is something that I can look at with zd= b > or similar I'll leave this alone for a bit -- the machine came back up > and is running fine. > > This is a production system but I can take it down off-hours for a shor= t > time if there is a need to run something in single-user mode using zdb > to try to figure out what's going on. There have been no known hardwar= e > issues with it and all the other pools (including the ones on the same > host adapter) are and have been running without incident. > > Ideas? > > > Fatal trap 12: page fault while in kernel mode > cpuid =3D 9; apic id =3D 21 > fault virtual address =3D 0x378 > fault code =3D supervisor read data, page not present > instruction pointer =3D 0x20:0xffffffff80940ae0 > stack pointer =3D 0x28:0xfffffe0668018680 > frame pointer =3D 0x28:0xfffffe0668018700 > code segment =3D base 0x0, limit 0xfffff, type 0x1b > =3D DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags =3D interrupt enabled, resume, IOPL =3D 0 > current process =3D 70921 (zfs) > trap number =3D 12 > panic: page fault > cpuid =3D 9 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe0668018160 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0668018210 > vpanic() at vpanic+0x126/frame 0xfffffe0668018250 > panic() at panic+0x43/frame 0xfffffe06680182b0 > trap_fatal() at trap_fatal+0x36b/frame 0xfffffe0668018310 > trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe06680183b0 > trap() at trap+0x47a/frame 0xfffffe06680185c0 > calltrap() at calltrap+0x8/frame 0xfffffe06680185c0 > --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe0668018680, r= bp =3D > 0xfffffe0668018700 --- > *__mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe0668018700*= * > **dounmount() at dounmount+0x58/frame 0xfffffe0668018780** > **zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe06680187a0= ** > **dsl_dataset_user_release_impl() at > dsl_dataset_user_release_impl+0x103/frame 0xfffffe0668018920** > **dsl_dataset_user_release_onexit() at > dsl_dataset_user_release_onexit+0x5c/frame 0xfffffe0668018940** > **zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe0668018= 970** > **zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe0668018990* > devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe06680189b0 > devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe06680189e0 > _fdrop() at _fdrop+0x29/frame 0xfffffe0668018a00 > closef() at closef+0x21e/frame 0xfffffe0668018a90 > closefp() at closefp+0x98/frame 0xfffffe0668018ad0 > amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe0668018bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0668018bf0 > --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= > 0x7fffffffd1e8, rbp =3D 0x7fffffffd200 --- > Uptime: 5d0h58m0s > > > > On 10/15/2015 09:27, Karl Denninger wrote: >> Got another one of these this morning, after a long absence... >> >> Same symptom -- happened during a backup (send/receive) and it's in >> the same place -- when the snapshot is unmounted. I have a clean dump= >> and this is against a quite-recent checkout, so the >> most-currently-rolled forward ZFS changes are in this kernel. >> >> >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 14; apic id =3D 34 >> fault virtual address =3D 0x378 >> fault code =3D supervisor read data, page not present >> instruction pointer =3D 0x20:0xffffffff80940ae0 >> stack pointer =3D 0x28:0xfffffe066796c680 >> frame pointer =3D 0x28:0xfffffe066796c700 >> code segment =3D base 0x0, limit 0xfffff, type 0x1b >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags =3D interrupt enabled, resume, IOPL =3D 0 >> current process =3D 81187 (zfs) >> trap number =3D 12 >> panic: page fault >> cpuid =3D 14 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe066796c160 >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe066796c210 >> vpanic() at vpanic+0x126/frame 0xfffffe066796c250 >> panic() at panic+0x43/frame 0xfffffe066796c2b0 >> trap_fatal() at trap_fatal+0x36b/frame 0xfffffe066796c310 >> trap_pfault() at trap_pfault+0x2ed/frame 0xfffffe066796c3b0 >> trap() at trap+0x47a/frame 0xfffffe066796c5c0 >> calltrap() at calltrap+0x8/frame 0xfffffe066796c5c0 >> --- trap 0xc, rip =3D 0xffffffff80940ae0, rsp =3D 0xfffffe066796c680, = rbp >> =3D 0xfffffe >> 066796c700 --- >> __mtx_lock_sleep() at __mtx_lock_sleep+0x1b0/frame 0xfffffe066796c700 >> dounmount() at dounmount+0x58/frame 0xfffffe066796c780 >> zfs_unmount_snap() at zfs_unmount_snap+0x114/frame 0xfffffe066796c7a0 >> dsl_dataset_user_release_impl() at >> dsl_dataset_user_release_impl+0x103/frame 0xf >> ffffe066796c920 >> dsl_dataset_user_release_onexit() at >> dsl_dataset_user_release_onexit+0x5c/frame >> 0xfffffe066796c940 >> zfs_onexit_destroy() at zfs_onexit_destroy+0x56/frame 0xfffffe066796c9= 70 >> zfsdev_close() at zfsdev_close+0x52/frame 0xfffffe066796c990 >> devfs_fpdrop() at devfs_fpdrop+0xa9/frame 0xfffffe066796c9b0 >> devfs_close_f() at devfs_close_f+0x45/frame 0xfffffe066796c9e0 >> _fdrop() at _fdrop+0x29/frame 0xfffffe066796ca00 >> closef() at closef+0x21e/frame 0xfffffe066796ca90 >> closefp() at closefp+0x98/frame 0xfffffe066796cad0 >> amd64_syscall() at amd64_syscall+0x35d/frame 0xfffffe066796cbf0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe066796cbf0 >> --- syscall (6, FreeBSD ELF64, sys_close), rip =3D 0x801a01f5a, rsp =3D= >> 0x7fffffffd728, rbp =3D 0x7fffffffd740 --- >> Uptime: 3d15h37m26s >> >> >> On 8/27/2015 15:44, Karl Denninger wrote: >>> No, but that does sound like it might be involved..... >>> >>> And yeah, this did start when I moved the root pool to a mirrored pai= r >>> of Intel 530s off a pair of spinning-rust WD RE4s.... (The 530s are d= arn >>> nice performance-wise, reasonably inexpensive and thus very suitable = for >>> a root filesystem drive and they also pass the "pull the power cord" >>> test, incidentally.) >>> >>> You may be onto something -- I'll try shutting it off, but due to the= >>> fact that I can't make this happen and it's a "every week or two" pan= ic, >>> but ALWAYS when the zfs send | zfs recv is running AND it's always on= >>> the same filesystem it will be a fair while before I know if it's fix= ed >>> (like over a month, given the usual pattern here, as that would be 4 >>> "average" periods without a panic)..... >>> >>> I also wonder if I could tune this out with some of the other TRIM >>> parameters instead of losing it entirely. >>> >>> vfs.zfs.trim.max_interval: 1 >>> vfs.zfs.trim.timeout: 30 >>> vfs.zfs.trim.txg_delay: 32 >>> vfs.zfs.trim.enabled: 1 >>> vfs.zfs.vdev.trim_max_pending: 10000 >>> vfs.zfs.vdev.trim_max_active: 64 >>> vfs.zfs.vdev.trim_min_active: 1 >>> >>> That it's panic'ing on a mtx_lock_sleep might point this way.... the >>> trace shows it coming from a zfs_onexit_destroy, which ends up callin= g >>> zfs_unmount_snap() and then it blows in dounmount() while executing >>> mtx_lock_sleep(). >>> >>> I do wonder if I'm begging for new and innovative performance issues = if >>> I run with TRIM off for an extended period of time, however..... :-) >>> >>> >>> On 8/27/2015 15:30, Sean Chittenden wrote: >>>> Have you tried disabling TRIM? We recently ran in to an issue where= a `zfs delete` on a large dataset caused the host to panic because TRIM = was tripping over the ZFS deadman timer. Disabling TRIM worked as valid= workaround for us. ? You mentioned a recent move to SSDs, so this can = happen, esp after the drive has experienced a little bit of actual work. = ? -sc >>>> >>>> >>>> -- >>>> Sean Chittenden >>>> sean@chittenden.org >>>> >>>> >>>>> On Aug 27, 2015, at 13:22, Karl Denninger wrot= e: >>>>> >>>>> On 8/15/2015 12:38, Karl Denninger wrote: >>>>>> Update: >>>>>> >>>>>> This /appears /to be related to attempting to send or receive a >>>>>> /cloned /snapshot. >>>>>> >>>>>> I use /beadm /to manage boot environments and the crashes have all= >>>>>> come while send/recv-ing the root pool, which is the one where the= se >>>>>> clones get created. It is /not /consistent within a given snapsho= t >>>>>> when it crashes and a second attempt (which does a "recovery" >>>>>> send/receive) succeeds every time -- I've yet to have it panic twi= ce >>>>>> sequentially. >>>>>> >>>>>> I surmise that the problem comes about when a file in the cloned >>>>>> snapshot is modified, but this is a guess at this point. >>>>>> >>>>>> I'm going to try to force replication of the problem on my test sy= stem. >>>>>> >>>>>> On 7/31/2015 04:47, Karl Denninger wrote: >>>>>>> I have an automated script that runs zfs send/recv copies to brin= g a >>>>>>> backup data set into congruence with the running copies nightly. = The >>>>>>> source has automated snapshots running on a fairly frequent basis= >>>>>>> through zfs-auto-snapshot. >>>>>>> >>>>>>> Recently I have started having a panic show up about once a week = during >>>>>>> the backup run, but it's inconsistent. It is in the same place, = but I >>>>>>> cannot force it to repeat. >>>>>>> >>>>>>> The trap itself is a page fault in kernel mode in the zfs code at= >>>>>>> zfs_unmount_snap(); here's the traceback from the kvm (sorry for = the >>>>>>> image link but I don't have a better option right now.) >>>>>>> >>>>>>> I'll try to get a dump, this is a production machine with encrypt= ed swap >>>>>>> so it's not normally turned on. >>>>>>> >>>>>>> Note that the pool that appears to be involved (the backup pool) = has >>>>>>> passed a scrub and thus I would assume the on-disk structure is o= k..... >>>>>>> but that might be an unfair assumption. It is always occurring i= n the >>>>>>> same dataset although there are a half-dozen that are sync'd -- i= f this >>>>>>> one (the first one) successfully completes during the run then al= l the >>>>>>> rest will as well (that is, whenever I restart the process it has= always >>>>>>> failed here.) The source pool is also clean and passes a scrub. >>>>>>> >>>>>>> traceback is at http://www.denninger.net/kvmimage.png; apologies = for the >>>>>>> image traceback but this is coming from a remote KVM. >>>>>>> >>>>>>> I first saw this on 10.1-STABLE and it is still happening on Free= BSD >>>>>>> 10.2-PRERELEASE #9 r285890M, which I updated to in an attempt to = see if >>>>>>> the problem was something that had been addressed. >>>>>>> >>>>>>> >>>>>> --=20 >>>>>> Karl Denninger >>>>>> karl@denninger.net >>>>>> /The Market Ticker/ >>>>>> /[S/MIME encrypted email preferred]/ >>>>> Second update: I have now taken another panic on 10.2-Stable, same = deal, >>>>> but without any cloned snapshots in the source image. I had thought= that >>>>> removing cloned snapshots might eliminate the issue; that is now ou= t the >>>>> window. >>>>> >>>>> It ONLY happens on this one filesystem (the root one, incidentally)= >>>>> which is fairly-recently created as I moved this machine from spinn= ing >>>>> rust to SSDs for the OS and root pool -- and only when it is being >>>>> backed up by using zfs send | zfs recv (with the receive going to a= >>>>> different pool in the same machine.) I have yet to be able to prov= oke >>>>> it when using zfs send to copy to a different machine on the same L= AN, >>>>> but given that it is not able to be reproduced on demand I can't be= >>>>> certain it's timing related (e.g. performance between the two pools= in >>>>> question) or just that I haven't hit the unlucky combination. >>>>> >>>>> This looks like some sort of race condition and I will continue to = see >>>>> if I can craft a case to make it occur "on demand" >>>>> >>>>> --=20 >>>>> Karl Denninger >>>>> karl@denninger.net >>>>> /The Market Ticker/ >>>>> /[S/MIME encrypted email preferred]/ >>>> %SPAMBLOCK-SYS: Matched [+Sean Chittenden ], me= ssage ok >>>> >> --=20 >> Karl Denninger >> karl@denninger.net >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms080404050900010009090303 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTEwMjMxMjEwMjRaME8GCSqGSIb3DQEJBDFCBEDB q9IF7EElLPYVJGTl3P8KmY6HNtLNMvGCZasJ8f8Wi6OEBX6pqq7/Yax3Y/ZjD9Hv/NQv+sj/ Cz7dEK0U3VOUMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAZ6737DRw iijdu1KbVCl4guXF0ZQP8yd4uu01nAZtAJvO7NrS51Zye6BaBxIz69ZcfVMt+vnuvbDZUyoy Lm/sew94WqasUfnC8HCW2m9QlVjjOx/xJnjy2P0OqEty4XpKmxI6RMEStWLdC53Iw31lIJ2N E5NvO734tgZRaKc39QTqUZyEGLYQzHZ7YsAuG4kAMQONlDb+I7qL7naRvQTPB71n1/UcmtyF kIeTFoTEq2DNxAZV1Zt27t1K0W8955wTXWG04u+T6bGA6vMAqgg/EBg6exuGyYmBSVGvi1JE bkKTR4+nMuHqJZtVolQ+Lfk1cD24b7vFz7PJ3P/eOR9r775bjjZAAFgRXOJnSfkrlzHhnNAP JOA3UnZMeRH79C6k8r4WY2nU06myg0ywSAyI1R4f9XqdnPYAx6EmZy5+8ENQpZqJiimjrsgs LsYXgv7CC+eQICmJBUZblcYeEcheTa78hQ04NZTP1b8igv88bUYFXJ9u5W47XmIQQSP1Mv5E EADgqAN1/Vyg1E05TPGlByMAhr+MswMyIyfGNL9C2jKpq528+4oGjWWSyT+e/pyKzWV4n0wy KB7qtekIv6C042HvScWJaAhkxDRJ40dm2mRDUN8sk1f671AgFF1PYG0Le8oFSQVtlD68k96D SrS1zelyTh7b4JJqg9ZJ016uu+0AAAAAAAA= --------------ms080404050900010009090303-- From owner-freebsd-fs@freebsd.org Sat Oct 24 23:38:28 2015 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CACB7A1E9C7 for ; Sat, 24 Oct 2015 23:38:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6A95808 for ; Sat, 24 Oct 2015 23:38:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9ONcSZS018800 for ; Sat, 24 Oct 2015 23:38:28 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 162775] zpool(1): Document some undocumented zpool import options Date: Sat, 24 Oct 2015 23:38:28 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Documentation X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2015 23:38:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=162775 Garrett Cooper,425-314-3911 changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|mm@FreeBSD.org |freebsd-fs@FreeBSD.org Status|In Progress |Open --- Comment #4 from Garrett Cooper,425-314-3911 --- -T, -V, -X aren't documented in zpool(8); everything else seems to be accounted for now. Reassigning to freebsd-fs@ -- You are receiving this mail because: You are the assignee for the bug.