From owner-freebsd-fs@FreeBSD.ORG Sun Sep 21 14:10:36 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 882333C1 for ; Sun, 21 Sep 2014 14:10:36 +0000 (UTC) Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de [80.67.31.24]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B1F5DD7 for ; Sun, 21 Sep 2014 14:10:35 +0000 (UTC) Received: from [78.35.139.120] (helo=fabiankeil.de) by smtprelay01.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XVhqN-0000gX-AW for freebsd-fs@freebsd.org; Sun, 21 Sep 2014 16:10:27 +0200 Date: Sun, 21 Sep 2014 16:10:31 +0200 From: Fabian Keil To: Subject: panic: solaris assert: bpobj_iterate(&spa->spa_deferred_bpobj, spa_free_sync_cb, zio, tx) == 0 (0x6 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c, line: 6156 Message-ID: <4c86b205.1f11cf29@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/GIgwPXJOJWzwtyNRQYwxpqw"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 14:10:36 -0000 --Sig_/GIgwPXJOJWzwtyNRQYwxpqw Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Two days ago a power outage took out a zpool but not the laptop it was attached to. This resulted in: Sep 19 22:50:58 r500 kernel: [41317] ugen1.2: at usbus1 (disconne= cted) Sep 19 22:50:58 r500 kernel: [41317] umass0: at uhub1, port 2, addr 2 (disc= onnected) Sep 19 22:50:58 r500 kernel: [41317] da0 at umass-sim0 bus 0 scbus2 target = 0 lun 0 Sep 19 22:50:58 r500 kernel: [41317] da0: < > detached Sep 19 22:50:58 r500 kernel: [41317] pass2 at umass-sim0 bus 0 scbus2 targe= t 0 lun 0 Sep 19 22:50:58 r500 kernel: [41317] pass2: < > detached Sep 19 22:50:58 r500 kernel: [41317] (pass2:umass-sim0:0:0:0): Periph destr= oyed Sep 19 22:50:58 r500 kernel: [41317] GEOM_ELI: Device label/intenso1.eli de= stroyed. Sep 19 22:50:58 r500 kernel: [41317] GEOM_ELI: Detached label/intenso1.eli = on last close. Sep 19 22:50:58 r500 kernel: [41317] (da0:umass-sim0:0:0:0): Periph destroy= ed Sep 19 22:50:58 r500 ZFS: vdev is removed, pool_guid=3D13312956307733420090= vdev_guid=3D11021414854688829035 [...] Sep 19 22:50:58 r500 kernel: [41318] system power profile changed to 'econo= my' Sep 19 22:50:58 r500 kernel: [41318] acpi_acad0: Off Line Sep 19 22:50:59 r500 power_profile: changed to 'economy' Followed by a panic: (kgdb) where #0 doadump (textdump=3D0) at pcpu.h:219 #1 0xffffffff8030eeae in db_dump (dummy=3D, dummy2=3D= 0, dummy3=3D0, dummy4=3D0x0) at /usr/src/sys/ddb/db_command.c:543 #2 0xffffffff8030e98d in db_command (cmd_table=3D0x0) at /usr/src/sys/ddb/= db_command.c:449 #3 0xffffffff8030e704 in db_command_loop () at /usr/src/sys/ddb/db_command= .c:502 #4 0xffffffff80311160 in db_trap (type=3D, code=3D0) = at /usr/src/sys/ddb/db_main.c:231 #5 0xffffffff805d7bc1 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff8085ab67 in trap (frame=3D0xfffffe00955ed850) at /usr/src/sys= /amd64/amd64/trap.c:542 #7 0xffffffff8083eef2 in calltrap () at /usr/src/sys/amd64/amd64/exception= .S:231 #8 0xffffffff805d72be in kdb_enter (why=3D0xffffffff8095b0cd "panic", msg= =3D) at cpufunc.h:63 #9 0xffffffff80597d01 in panic (fmt=3D) at /usr/src/s= ys/kern/kern_shutdown.c:739 #10 0xffffffff8133d22f in assfail3 (a=3D, lv=3D, op=3D, rv=3D, f= =3D, l=3D) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91 #11 0xffffffff811477f8 in spa_sync (spa=3D0xfffff8005b727000, txg=3D69362) = at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6155 #12 0xffffffff81150ed6 in txg_sync_thread (arg=3D0xfffff8000291a000) at /us= r/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:517 #13 0xffffffff8055e4fa in fork_exit (callout=3D0xffffffff81150b30 , arg=3D0xfffff8000291a000, frame=3D0xfffffe00955edc00) at /usr/src= /sys/kern/kern_fork.c:977 #14 0xffffffff8083f42e in fork_trampoline () at /usr/src/sys/amd64/amd64/ex= ception.S:605 #15 0x0000000000000000 in ?? () The kernel is based on FreeBSD 11.0-CURRENT r271788. Later on another power outage took out the pool again, but this time it was just faulted as expected. The pool is: fk@r500 ~ $zpool status intenso1 pool: intenso1 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub in progress since Fri Sep 19 23:07:49 2014 400G scanned out of 941G at 29.3M/s, 5h15m to go 0 repaired, 42.48% done config: NAME STATE READ WRITE CKSUM intenso1 ONLINE 0 0 0 label/intenso1.eli ONLINE 0 0 0 errors: 8 data errors, use '-v' for a list Once the scrub is complete, I expect the "data errors" to be gone as they are merely the result of temporary read errors after the second outage. Apparently those aren't properly handled with the given pool layout, but that's another issue. Fabian --Sig_/GIgwPXJOJWzwtyNRQYwxpqw Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQe3FYACgkQBYqIVf93VJ227QCgvYZlnbLgM9ovd8h7BRplciWW s0MAoJu14e80wyEFnAusizsZfYjbaIpv =FqUi -----END PGP SIGNATURE----- --Sig_/GIgwPXJOJWzwtyNRQYwxpqw-- From owner-freebsd-fs@FreeBSD.ORG Sun Sep 21 18:27:40 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A44CC81 for ; Sun, 21 Sep 2014 18:27:40 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4054788E for ; Sun, 21 Sep 2014 18:27:40 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8LIReaM098902 for ; Sun, 21 Sep 2014 18:27:40 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 188328] [zfs] UPDATING should provide caveats for running `zpool upgrade` Date: Sun, 21 Sep 2014 18:27:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jan.kokemueller@gmail.com X-Bugzilla-Status: In Discussion X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-fs@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 18:27:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D188328 Jan Kokem=C3=BCller changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jan.kokemueller@gmail.com --- Comment #3 from Jan Kokem=C3=BCller --- Created attachment 147531 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D147531&action= =3Dedit Untested patch that adds the missing bootcode upgrade warning I've just had the same problem after upgrading my bootpool on 10-stable. I = used "zpool upgrade -a". I was already on pool version 5000; only some feature f= lags were added. The problematic feature flags not recognized by the bootloader = were "com.delphix:hole_birth" and "com.delphix:embedded_data". I was aware of the "gpart bootcode" dance but "zpool upgrade -a" definitely showed no message so I assumed it was safe to reboot without upgrading the bootloader. I've looked at the zpool code and it seems that in the case of "zpool upgrade -a" the bootcode upgrade warning may not be displayed. I've attached a patch that should fix it (untested, but it compiles). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@FreeBSD.ORG Sun Sep 21 19:26:15 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1B8FC5AD for ; Sun, 21 Sep 2014 19:26:15 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 01F41E02 for ; Sun, 21 Sep 2014 19:26:15 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8LJQEQD053090 for ; Sun, 21 Sep 2014 19:26:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [Bug 188328] [zfs] UPDATING should provide caveats for running `zpool upgrade` Date: Sun, 21 Sep 2014 19:26:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: smh@FreeBSD.org X-Bugzilla-Status: In Discussion X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: smh@FreeBSD.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 19:26:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=188328 Steven Hartland changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |smh@FreeBSD.org Assignee|freebsd-fs@FreeBSD.org |smh@FreeBSD.org -- You are receiving this mail because: You are the assignee for the bug. From owner-freebsd-fs@FreeBSD.ORG Sun Sep 21 21:00:07 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6BDFBD2 for ; Sun, 21 Sep 2014 21:00:07 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A56FC861 for ; Sun, 21 Sep 2014 21:00:07 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8LL07Sw014328 for ; Sun, 21 Sep 2014 21:00:07 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201409212100.s8LL07Sw014328@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-fs@FreeBSD.org Subject: Problem reports for freebsd-fs@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 21 Sep 2014 21:00:07 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Sep 2014 21:00:07 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ----------------+-----------+------------------------------------------------- Needs MFC | 133174 | [msdosfs] [patch] msdosfs must support multibyt Needs MFC | 136470 | [nfs] Cannot mount / in read-only, over NFS Needs MFC | 139651 | [nfs] mount(8): read-only remount of NFS volume Needs MFC | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non 4 problems total for which you should take action. From owner-freebsd-fs@FreeBSD.ORG Mon Sep 22 08:00:14 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4C7EA599 for ; Mon, 22 Sep 2014 08:00:14 +0000 (UTC) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2ED24CBD for ; Mon, 22 Sep 2014 08:00:14 +0000 (UTC) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.14.9/8.14.9) with ESMTP id s8M80EWh003303 for ; Mon, 22 Sep 2014 08:00:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Message-Id: <201409220800.s8M80EWh003303@kenobi.freebsd.org> From: bugzilla-noreply@freebsd.org To: freebsd-fs@FreeBSD.org Subject: [FreeBSD Bugzilla] Commit Needs MFC MIME-Version: 1.0 X-Bugzilla-Type: whine X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated Date: Mon, 22 Sep 2014 08:00:14 +0000 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 08:00:14 -0000 Hi, You have a bug in the "Needs MFC" state which has not been touched in 7 or more days. This email serves as a reminder that you may want to MFC this bug or marked it as completed. In the event you have a longer MFC timeout you may update this bug with a comment and I won't remind you again for 7 days. This reminder is only sent on Mondays. Please file a bug about concerns you may have. This search was scheduled by eadler@FreeBSD.org. (4 bugs) Bug 133174: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=133174 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [msdosfs] [patch] msdosfs must support multibyte international characters in file names Bug 136470: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=136470 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] Cannot mount / in read-only, over NFS Bug 139651: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=139651 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [nfs] mount(8): read-only remount of NFS volume does not work Bug 144447: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=144447 Severity: Affects Only Me Priority: Normal Hardware: Any Assignee: freebsd-fs@FreeBSD.org Status: Needs MFC Resolution: Summary: [zfs] sharenfs fsunshare() & fsshare_main() non functional From owner-freebsd-fs@FreeBSD.ORG Mon Sep 22 16:14:58 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5BA7EAF1 for ; Mon, 22 Sep 2014 16:14:58 +0000 (UTC) Received: from mail-ig0-x22d.google.com (mail-ig0-x22d.google.com [IPv6:2607:f8b0:4001:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 29021F0B for ; Mon, 22 Sep 2014 16:14:58 +0000 (UTC) Received: by mail-ig0-f173.google.com with SMTP id l13so3002014iga.12 for ; Mon, 22 Sep 2014 09:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:date:message-id:subject:from:to:content-type; bh=0Gg5UOsXpDsrKscYfVZ1C86ukUgBK1eQyTPi6/FP/2I=; b=HaOlnTQ/gWBfEY9/uUh1I1OELNfQRKWB6buzQHPJOHRUl6MnVXY3Kx8ZdItkI5POC1 v3GwmVKgqUjXJ/bKigO6688tP0WdyJIaZRNPnoyKFhMS8D+ol9H/izo1wCm8UnCdrWpJ KB47jmfDF4uiEZeGB9Tb/nOx6UYJPEFNBfClI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=0Gg5UOsXpDsrKscYfVZ1C86ukUgBK1eQyTPi6/FP/2I=; b=deKTVRHXgSiAQKt/tLSTJff/zl4tknlsxtNl9yhRs3wmd3fFdjAeKMMf23ZWLEFNC0 ChPwEpswKUrSqmI+M7Q7Of2SIcva0XBnp3mVFOw6JL2A/mUfLLklLjylTcswCeV781ds 5bYHd54WpA/yYMVfqt1LLU6yqsZBU0EXQpIO9cMnnIVTDamzd02AWE2wn14CN86al+xT bJHRk2kvhKng+bWxAlxpNS5/zApN0JgVefBhps1eD/ZyaC3tAOZGtSPEp3lwY1NB26yM PP7lL/u3sF2s6IR/Nrg9iE8DoQbPnXu5AkFpeUzOqccYzsHJQamitgKxSouGshEfatGh 81uQ== X-Gm-Message-State: ALoCoQldrjRxAM6/eE4HFSBtSpubAgqadPc4vWM31+yOkMVGkA/nIAhpIJkjVJG+gcUTR3lFdWH8 MIME-Version: 1.0 X-Received: by 10.50.170.196 with SMTP id ao4mr15475653igc.46.1411402497378; Mon, 22 Sep 2014 09:14:57 -0700 (PDT) Received: by 10.43.158.200 with HTTP; Mon, 22 Sep 2014 09:14:57 -0700 (PDT) Date: Mon, 22 Sep 2014 09:14:57 -0700 Message-ID: Subject: ZFS Warning Since Upgrade to 10.0 From: Tim Gustafson To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 16:14:58 -0000 I recently upgraded a ZFS file server from 9.2 to 10.0 and then started getting this warning when I run zpool status: status: One or more devices are configured to use a non-native block size. Expect reduced performance. action: Replace affected devices with devices that support the configured block size, or migrate data to a properly configured pool. I Googled around a bit, and understand the warning, but I have a problem: that zpool is 135TB and I don't have 135TB of disks laying around, nor the controllers necessary to support an additional 135TB of disks, to migrate this zpool to a properly configured one, nor could I easily have the server off-line for the requisite time that would be required to transfer 100+ TB of data from one set of hard drives to another. So my questions are: How much will this sub-optimal configuration affect performance? Does the upgrade to 10.0 represent a reduction in performance, or was the reduction in performance always there and just not reported? This server is used to store genome data, so performance is pretty important, but the users were happy with the performance when it was a 9.2 server. If I convince the users to go through an upgrade process to fix this issue, how much of a boost in performance can they expect to see? If it's a 2% boost, I don't think I can get them to invest in the upgrade, but it it's a 100% boost, I'm pretty sure I can. -- Tim Gustafson tjg@ucsc.edu 831-459-5354 Baskin Engineering, Room 313A From owner-freebsd-fs@FreeBSD.ORG Mon Sep 22 18:29:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3762EE83 for ; Mon, 22 Sep 2014 18:29:27 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id EF9B6167 for ; Mon, 22 Sep 2014 18:29:26 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id DB4C820E7088E; Mon, 22 Sep 2014 18:29:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1,RDNS_DYNAMIC,STOX_REPLY_TYPE autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 7227320E7088A; Mon, 22 Sep 2014 18:29:23 +0000 (UTC) Message-ID: <6EF866714C19452C86C1BC0C99539D00@multiplay.co.uk> From: "Steven Hartland" To: "Tim Gustafson" , References: Subject: Re: ZFS Warning Since Upgrade to 10.0 Date: Mon, 22 Sep 2014 19:29:19 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Sep 2014 18:29:27 -0000 ----- Original Message ----- From: "Tim Gustafson" To: Sent: Monday, September 22, 2014 5:14 PM Subject: ZFS Warning Since Upgrade to 10.0 >I recently upgraded a ZFS file server from 9.2 to 10.0 and then > started getting this warning when I run zpool status: > > status: One or more devices are configured to use a non-native block > size. Expect reduced performance. > action: Replace affected devices with devices that support the > configured block size, or migrate data to a properly configured pool. > > I Googled around a bit, and understand the warning, but I have a > problem: that zpool is 135TB and I don't have 135TB of disks laying > around, nor the controllers necessary to support an additional 135TB > of disks, to migrate this zpool to a properly configured one, nor > could I easily have the server off-line for the requisite time that > would be required to transfer 100+ TB of data from one set of hard > drives to another. > > So my questions are: > > How much will this sub-optimal configuration affect performance? That depends on your disks, as native 4k drivers when you send a 512 write it has to perform a COW operation. The only real way to tell is to compare the two in a test with your setup. > Does the upgrade to 10.0 represent a reduction in performance, or was > the reduction in performance always there and just not reported? This > server is used to store genome data, so performance is pretty > important, but the users were happy with the performance when it was a > 9.2 server. The issue was always there, its just ZFS now reports the issue. > If I convince the users to go through an upgrade process to fix this > issue, how much of a boost in performance can they expect to see? If > it's a 2% boost, I don't think I can get them to invest in the > upgrade, but it it's a 100% boost, I'm pretty sure I can. Impossible to say, you could test on a smaller install if you want to be sure. Regards Steve From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 01:33:52 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07BDE1A4 for ; Tue, 23 Sep 2014 01:33:52 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D11FC65E for ; Tue, 23 Sep 2014 01:33:51 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id kx10so5601660pab.30 for ; Mon, 22 Sep 2014 18:33:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:message-id:date:to:mime-version; bh=UM6sluS4+NYbdSYeCG2tyZ080yLZt11PxUim8YoxiWI=; b=xLUYlzfK24yVmBb9DJYIZ4srV+6+mgAHFJdoomj1pSlJ+PjqDd93miUQjiuG6NyJ5A 2qwtArM9knmqiZFUCp0OAGhNGm3YUU1a3q4Swj6eP2uar+63sRMHi7Q5CZd/svWZE677 Mcimd/5NmFVWAT5J36p3ppxYxNQIlK1840qHPKojLpKAPeNyMP8/uWfhmK7kuNIgHUdk nEoOeulBDymQJ0zymGPW9GeKLhinCXZrNn8SqoVitYFW1kwmWMTNjRIXpNkfvN7fbfuQ EUYJSpjF+0DsQ71AB+LRQ7ArAN+WqR1h7ylzjoGXdYL4wh1Q8cSk1S2N4uFeTN0B8wD5 zHQQ== X-Received: by 10.66.136.48 with SMTP id px16mr30780125pab.10.1411436031377; Mon, 22 Sep 2014 18:33:51 -0700 (PDT) Received: from macmini.internal.coolpeople.us ([2001:470:b:2d8::8001]) by mx.google.com with ESMTPSA id oq6sm992889pdb.45.2014.09.22.18.33.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 22 Sep 2014 18:33:50 -0700 (PDT) From: John Terrell Subject: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. Message-Id: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> Date: Mon, 22 Sep 2014 18:33:48 -0700 To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 01:33:52 -0000 Hello all, I'm trying to replicate one of the ZFS filesystems that lives = on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The transfer = dies at about the 1.7TB (of almost 4TB) mark indicating: "cannot receive incremental stream: invalid backup stream"=20 The stream being sent is not incremental so I'm not sure what the = receiver is trying to do. Here's the command I'm using to transfer = (executed on the SmartOS machine): ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v tank I've also tried to use mbuffer as the transport interface (removing SSH = from the equation) and it has the same result. Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS = implementations?= From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 02:08:02 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B20F84C9 for ; Tue, 23 Sep 2014 02:08:02 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9178F93B for ; Tue, 23 Sep 2014 02:08:02 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id s8N282qb031628 for ; Tue, 23 Sep 2014 02:08:02 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id s8N2823Q031623 for freebsd-fs@FreeBSD.org; Tue, 23 Sep 2014 02:08:02 GMT (envelope-from bdrewery) Received: (qmail 36358 invoked from network); 22 Sep 2014 21:08:00 -0500 Received: from unknown (HELO ?10.10.0.24?) (freebsd@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 22 Sep 2014 21:08:00 -0500 Message-ID: <5420D5FC.4030600@FreeBSD.org> Date: Mon, 22 Sep 2014 21:07:56 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: FreeBSD FS , Konstantin Belousov Subject: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Hb4K9m0hacv3IKadGr1XVwrOU5suiheIf" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 02:08:02 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Hb4K9m0hacv3IKadGr1XVwrOU5suiheIf Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable This is slightly different than my last report (involving zfs_lookup/zfs_dirent). I still need to gather information from that previous one. Here is what I have on this one. Quoted so Thunderbird doesn't trash it: > 1. Poudriere build startup, run for N=3D01,...,12 in parallel: > a. mount tmpfs on /poudriere/data/.m/exp-10amd64-commit-test/N > /poudriere/data/.m is a ZFS mount. > b. cpdup /poudriere/data/.m/exp-10amd64-commit-test/ref to /poudri= ere/data/.m/exp-10amd64-commit-test/N > [1] c. mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/N/.p > 2. SIGINT during this phase: > [2] a. umount -f all tmpfs > b. cpdup processes are *not* killed. They continue running. Most o= f them fail due to their dst tmpfs being gone. > [3] c. A cpdup process managed to continue running. Since the tmpfs wa= s gone it started trying to write to the ZFS mount under it. (/poudriere/= data/.m) >=20 > # procstat -kka|egrep "(lockmgr|tmpfs)" > [3] 76137 100919 cpdup - mi_switch+0x179 slee= pq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0x862 vop_s= tdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa vget+0x67 cache_lookup+0x5b2= vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xf1 lookup+0x5ad namei+0x4e4 kern_= linkat+0x13d sys_link+0x28 filemon_wrapper_link+0x19 amd64_syscall+0x25a > [1] 76346 102346 mkdir - mi_switch+0x179 slee= pq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0xea1 vop_s= tdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa lookup+0x496 namei+0x4e4 ker= n_mkdirat+0xcb amd64_syscall+0x25a Xfast_syscall+0xfb > [2] 76286 102085 umount - mi_switch+0x179 slee= pq_switch+0x152 sleepq_wait+0x43 _sleep+0x366 vfs_write_suspend+0xf0 vfs_= write_suspend_umnt+0x38 tmpfs_unmount+0x6d dounmount+0x424 sys_unmount+0x= 2ec amd64_syscall+0x25a Xfast_syscall+0xfb >=20 > [3] root 76137 0.7 0.0 14772 2844 3 D+ 3:12PM 0:05.2= 8 cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/= data/.m/exp-10amd64-commit-test/02 > [1] root 76346 0.0 0.0 8200 1916 3 D+ 3:12PM 0:00.0= 0 mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p > [2] root 76286 [command taken from gdb] = umount -f /poudriere/data/.m/exp-10amd64-commit-test/02 >=20 > Locked vnodes: >=20 > 0xfffff80fda0bcce8: tag zfs, type VDIR > usecount 2, writecount 0, refcount 3 mountedhere 0x0 > flags () > v_object 0xfffff805fc81a300 ref 0 pages 0 > lock type zfs: SHARED (count 2) with exclusive waiters pending > mount /poudriere/data/.m from zroot/poudriere/data/.m > path /poudriere/data/.m/exp-10am64-commit-test/02 > 0xfffff8065de471d8: tag zfs, type VDIR > usecount 1, writecount 0, refcount 3 mountedhere 0xfffff8058f582330= > flags () > lock type zfs: EXCL by thread 0xfffff8047c0eb920 (pid 76286) with e= xclusive and shared waiters pending > mount /poudriere/data/.m from zroot/poudriere/data/.m > path /poudriere/data/.m >=20 >=20 > Processes: >=20 > PID 76286 (umount -f /poudriere/data/.m/exp-10amd64-commit-test/02): > Waiting in vfs_write_suspend(mp=3D0xfffff8058f582330) for mn_writeopc= ount=3D=3D0 >=20 >=20 > (kgdb) thread 521 > [Switching to thread 521 (Thread 102085)]#0 sched_switch (td=3D0xfffff= 8047c0eb920, newtd=3D, flags=3D= ) at /usr/src/sys/kern/sched_ule.c:1932 > 1932 cpuid =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffff8047c0eb920, newtd=3D, flags=3D) at /usr/src/sys/kern/sched_ule.c:1932 > #1 0xffffffff80929379 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/= src/sys/kern/kern_synch.c:493 > #2 0xffffffff80966af2 in sleepq_switch (wchan=3D,= pri=3D) at /usr/src/sys/kern/subr_sleepqueue.c:552 > #3 0xffffffff80966953 in sleepq_wait (wchan=3D0xfffff8058f5823bc, pri=3D= 119) at /usr/src/sys/kern/subr_sleepqueue.c:631 > #4 0xffffffff80928c86 in _sleep (ident=3D, lock=3D= , priority=3D, wmesg=3D, sbt=3D, > pr=3D) at /usr/src/sys/kern/kern_synch.c:255 > #5 0xffffffff809df0b0 in vfs_write_suspend (mp=3D0xfffff8058f582330, f= lags=3D) at /usr/src/sys/kern/vfs_vnops.c:1793 > #6 0xffffffff809df308 in vfs_write_suspend_umnt (mp=3D0xfffff8058f5823= 30) at /usr/src/sys/kern/vfs_vnops.c:1848 > #7 0xffffffff822ad9ed in tmpfs_unmount (mp=3D0xfffff8058f582330, mntfl= ags=3D) at /usr/src/sys/modules/tmpfs/../../fs/tmpfs= /tmpfs_vfsops.c:280 > #8 0xffffffff809c7664 in dounmount (mp=3D0xfffff8058f582330, flags=3D1= 34742016, td=3D0xfffff8047c0eb920) at /usr/src/sys/kern/vfs_mount.c:1310 > #9 0xffffffff809c721c in sys_unmount (td=3D0xfffff8047c0eb920, uap=3D0= xfffffe1248712b80) at /usr/src/sys/kern/vfs_mount.c:1201 > #10 0xffffffff80d38b6a in amd64_syscall (td=3D0xfffff8047c0eb920, trace= d=3D0) at subr_syscall.c:133 > #11 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/= exception.S:390 >=20 > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_writeopcount > $12 =3D 1 > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_activevnodelistsize > $13 =3D 1 > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_vnodecovered > $25 =3D (struct vnode *) 0xfffff8065de471d8 > (kgdb) p *(struct mount*)0xfffff8058f582330 > $5 =3D {mnt_mtx =3D {lock_object =3D {lo_name =3D 0xffffffff80fc4c20 "s= truct mount mtx", lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0x= fffffe0000b1ea00}, mtx_lock =3D 4}, mnt_gen =3D 23, > mnt_list =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffff8117c2abce8}, mnt= _op =3D 0xffffffff822b3e40, mnt_vfc =3D 0xffffffff822b3f60, mnt_vnodecove= red =3D 0xfffff8065de471d8, mnt_syncer =3D 0x0, > mnt_ref =3D 7357, mnt_nvnodelist =3D {tqh_first =3D 0xfffff803f6a691d= 8, tqh_last =3D 0xfffff8065dc871f8}, mnt_nvnodelistsize =3D 7356, mnt_act= ivevnodelist =3D {tqh_first =3D 0xfffff806dff493b0, > tqh_last =3D 0xfffff806dff49470}, mnt_activevnodelistsize =3D 1, mn= t_writeopcount =3D 1, mnt_kern_flag =3D 150994953, mnt_flag =3D 4096, mnt= _opt =3D 0xfffff804e377eaf0, mnt_optnew =3D 0x0, > mnt_maxsymlinklen =3D 0, mnt_stat =3D {f_version =3D 537068824, f_typ= e =3D 135, f_flags =3D 4096, f_bsize =3D 4096, f_iosize =3D 4096, f_block= s =3D 1835008, f_bfree =3D 1798116, f_bavail =3D 1798116, > f_files =3D 25690112, f_ffree =3D 25682842, f_syncwrites =3D 0, f_a= syncwrites =3D 0, f_syncreads =3D 0, f_asyncreads =3D 0, f_spare =3D {0, = 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax =3D 255, f_owner =3D 0, > f_fsid =3D {val =3D {-2027880658, 135}}, f_charspare =3D '\0' , f_fstypename =3D "tmpfs\000\000\000\000\000\000\000\000\00= 0\000", > f_mntfromname =3D "tmpfs", '\0' , f_mntonname =3D= "/poudriere/data/.m/exp-10amd64-commit-test/02", '\0' = }, mnt_cred =3D 0xfffff8047c8a4a00, > mnt_data =3D 0xfffff8000bd2b800, mnt_time =3D 0, mnt_iosize_max =3D 6= 5536, mnt_export =3D 0x0, mnt_label =3D 0x0, mnt_hashseed =3D 3766194835,= mnt_lockref =3D 0, mnt_secondary_writes =3D 0, > mnt_secondary_accwrites =3D 0, mnt_susp_owner =3D 0xfffff8047c0eb920,= mnt_gjprovider =3D 0x0, mnt_explock =3D {lock_object =3D {lo_name =3D 0x= ffffffff80fa4b29 "explock", lo_flags =3D 108199936, > lo_data =3D 0, lo_witness =3D 0xfffffe0000b26c80}, lk_lock =3D 1,= lk_exslpfail =3D 0, lk_timo =3D 0, lk_pri =3D 96}, mnt_upper_link =3D {t= qe_next =3D 0x0, tqe_prev =3D 0xfffff8005c842ca0}, > mnt_uppers =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff8058f582650}} > (kgdb) p (((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh= _first) > $30 =3D (struct vnode *) 0xfffff806dff493b0 > (kgdb) p *(((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tq= h_first) > $24 =3D {v_tag =3D 0xffffffff822b2f76 "tmpfs", v_op =3D 0xffffffff822b3= 918, v_data =3D 0xfffff80ae07caae0, v_mount =3D 0xfffff8058f582330, v_nmn= tvnodes =3D {tqe_next =3D 0xfffff806071d11d8, > tqe_prev =3D 0xfffff804d63963d0}, v_un =3D {vu_mount =3D 0x0, vu_so= cket =3D 0x0, vu_cdev =3D 0x0, vu_fifoinfo =3D 0x0}, v_hashlist =3D {le_n= ext =3D 0x0, le_prev =3D 0x0}, v_cache_src =3D { > lh_first =3D 0x0}, v_cache_dst =3D {tqh_first =3D 0x0, tqh_last =3D= 0xfffff806dff49400}, v_cache_dd =3D 0x0, v_lock =3D {lock_object =3D {lo= _name =3D 0xffffffff822b2f76 "tmpfs", lo_flags =3D 116588544, > lo_data =3D 0, lo_witness =3D 0xfffffe0000b29a00}, lk_lock =3D 1,= lk_exslpfail =3D 0, lk_timo =3D 51, lk_pri =3D 96}, v_interlock =3D {loc= k_object =3D { > lo_name =3D 0xffffffff80fc4bc6 "vnode interlock", lo_flags =3D 16= 973824, lo_data =3D 0, lo_witness =3D 0xfffffe0000b1e500}, mtx_lock =3D 4= }, v_vnlock =3D 0xfffff806dff49418, v_actfreelist =3D { > tqe_next =3D 0x0, tqe_prev =3D 0xfffff8058f5823a8}, v_bufobj =3D {b= o_lock =3D {lock_object =3D {lo_name =3D 0xffffffff80fcd144 "bufobj inter= lock", lo_flags =3D 86179840, lo_data =3D 0, > lo_witness =3D 0xfffffe0000b26580}, rw_lock =3D 1}, bo_ops =3D = 0xffffffff814ab900, bo_object =3D 0xfffff809dc520a00, bo_synclist =3D {le= _next =3D 0x0, le_prev =3D 0x0}, > bo_private =3D 0xfffff806dff493b0, __bo_vnode =3D 0xfffff806dff493b= 0, bo_clean =3D {bv_hd =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff806dff= 494d0}, bv_root =3D {pt_root =3D 0}, bv_cnt =3D 0}, > bo_dirty =3D {bv_hd =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff806= dff494f0}, bv_root =3D {pt_root =3D 0}, bv_cnt =3D 0}, bo_numoutput =3D 0= , bo_flag =3D 0, bo_bsize =3D 4096}, v_pollinfo =3D 0x0, > v_label =3D 0x0, v_lockf =3D 0x0, v_rl =3D {rl_waiters =3D {tqh_first= =3D 0x0, tqh_last =3D 0xfffff806dff49538}, rl_currdep =3D 0x0}, v_cstart= =3D 0, v_lasta =3D 0, v_lastw =3D 0, v_clen =3D 0, > v_holdcnt =3D 1, v_usecount =3D 1, v_iflag =3D 512, v_vflag =3D 0, v_= writecount =3D 0, v_hash =3D 115340435, v_type =3D VREG} >=20 >=20 > PID 76137 (cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref = /poudriere/data/.m/exp-10amd64-commit-test/02): > Waiting for vget(vp=3D0xfffff8065de471d8) [which is mnt_vnodecovered = from umount -f] >=20 > (kgdb) thread 520 > [Switching to thread 520 (Thread 100919)]#0 sched_switch (td=3D0xfffff= 8005cd2a490, newtd=3D, flags=3D= ) at /usr/src/sys/kern/sched_ule.c:1932 > 1932 cpuid =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffff8005cd2a490, newtd=3D, flags=3D) at /usr/src/sys/kern/sched_ule.c:1932 > #1 0xffffffff80929379 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/= src/sys/kern/kern_synch.c:493 > #2 0xffffffff80966af2 in sleepq_switch (wchan=3D,= pri=3D) at /usr/src/sys/kern/subr_sleepqueue.c:552 > #3 0xffffffff80966953 in sleepq_wait (wchan=3D0xfffff8065de47240, pri=3D= 96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > #4 0xffffffff808fdf3a in sleeplk (lk=3D, flags=3D= , ilk=3D, wmesg=3D, pri=3D, > timo=3D) at /usr/src/sys/kern/kern_lock.c:225 > #5 0xffffffff808fcfc2 in __lockmgr_args (lk=3D0xfffff8065de47240, flag= s=3D, ilk=3D0xfffff8065de47270, wmesg=3D, pri=3D, > timo=3D, file=3D0xffffffff80fcd0e8 "/usr/src/s= ys/kern/vfs_subr.c", line=3D2137) at /usr/src/sys/kern/kern_lock.c:680 > #6 0xffffffff809be27c in vop_stdlock (ap=3D) at l= ockmgr.h:98 > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=3D, a= =3D) at vnode_if.c:2082 > #8 0xffffffff809ddbca in _vn_lock (vp=3D0xfffff8065de471d8, flags=3D, file=3D0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_sub= r.c", line=3D2137) at vnode_if.h:859 > #9 0xffffffff809cde87 in vget (vp=3D0xfffff8065de471d8, flags=3D209740= 8, td=3D0xfffff8005cd2a490) at /usr/src/sys/kern/vfs_subr.c:2137 > #10 0xffffffff809ba132 in cache_lookup (dvp=3D0xfffff80fda0bcce8, vpp=3D= 0xfffffe1247de78b8, cnp=3D0xfffffe1247de78e0, tsp=3D0x0, ticksp=3D0x0) at= /usr/src/sys/kern/vfs_cache.c:673 > #11 0xffffffff809bb2dc in vfs_cache_lookup (ap=3D)= at /usr/src/sys/kern/vfs_cache.c:1038 > #12 0xffffffff80e58e41 in VOP_LOOKUP_APV (vop=3D, = a=3D) at vnode_if.c:127 > #13 0xffffffff809c364d in lookup (ndp=3D0xfffffe1247de7858) at vnode_if= =2Eh:54 > #14 0xffffffff809c2d84 in namei (ndp=3D0xfffffe1247de7858) at /usr/src/= sys/kern/vfs_lookup.c:300 > #15 0xffffffff809d76dd in kern_linkat (td=3D0xfffff8005cd2a490, fd1=3D<= value optimized out>, fd2=3D-100, path1=3D, path2=3D= 0x80141c600
, > segflg=3DUIO_USERSPACE, follow=3D64) at /usr/src/sys/kern/vfs_sysca= lls.c:1572 > #16 0xffffffff809d7518 in sys_link (td=3D0x0, uap=3D) at /usr/src/sys/kern/vfs_syscalls.c:1541 > #17 0xffffffff820c64e9 in filemon_wrapper_link (td=3D0x0, uap=3D0xfffff= e1247de7b80) at filemon_wrapper.c:364 > #18 0xffffffff80d38b6a in amd64_syscall (td=3D0xfffff8005cd2a490, trace= d=3D0) at subr_syscall.c:133 > #19 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/= exception.S:390 >=20 > (kgdb) p *(struct vnode*)0xfffff8065de471d8 > $34 =3D {v_tag =3D 0xffffffff81f0f8ff "zfs", v_op =3D 0xffffffff81f24b5= 0, v_data =3D 0xfffff80cb12f22e0, v_mount =3D 0xfffff8005c8a8660, v_nmntv= nodes =3D {tqe_next =3D 0xfffff80528f4d938, > tqe_prev =3D 0xfffff804d659cb30}, v_un =3D {vu_mount =3D 0xfffff805= 8f582330, vu_socket =3D 0xfffff8058f582330, vu_cdev =3D 0xfffff8058f58233= 0, vu_fifoinfo =3D 0xfffff8058f582330}, v_hashlist =3D { > le_next =3D 0x0, le_prev =3D 0x0}, v_cache_src =3D {lh_first =3D 0x= 0}, v_cache_dst =3D {tqh_first =3D 0xfffff805d7724a80, tqh_last =3D 0xfff= ff805d7724aa0}, v_cache_dd =3D 0xfffff805d7724a80, > v_lock =3D {lock_object =3D {lo_name =3D 0xffffffff81f0f8ff "zfs", lo= _flags =3D 117112832, lo_data =3D 0, lo_witness =3D 0xfffffe0000b28780}, = lk_lock =3D 18446735296877738278, lk_exslpfail =3D 0, > lk_timo =3D 51, lk_pri =3D 96}, v_interlock =3D {lock_object =3D {l= o_name =3D 0xffffffff80fc4bc6 "vnode interlock", lo_flags =3D 16973824, l= o_data =3D 0, lo_witness =3D 0xfffffe0000b1e500}, > mtx_lock =3D 4}, v_vnlock =3D 0xfffff8065de47240, v_actfreelist =3D= {tqe_next =3D 0xfffff80528f4d938, tqe_prev =3D 0xfffff804d659cbd0}, v_bu= fobj =3D {bo_lock =3D {lock_object =3D { > lo_name =3D 0xffffffff80fcd144 "bufobj interlock", lo_flags =3D= 86179840, lo_data =3D 0, lo_witness =3D 0xfffffe0000b26580}, rw_lock =3D= 1}, bo_ops =3D 0xffffffff814ab900, bo_object =3D 0x0, > bo_synclist =3D {le_next =3D 0x0, le_prev =3D 0x0}, bo_private =3D = 0xfffff8065de471d8, __bo_vnode =3D 0xfffff8065de471d8, bo_clean =3D {bv_h= d =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff8065de472f8}, > bv_root =3D {pt_root =3D 0}, bv_cnt =3D 0}, bo_dirty =3D {bv_hd =3D= {tqh_first =3D 0x0, tqh_last =3D 0xfffff8065de47318}, bv_root =3D {pt_ro= ot =3D 0}, bv_cnt =3D 0}, bo_numoutput =3D 0, bo_flag =3D 0, > bo_bsize =3D 131072}, v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf = =3D 0x0, v_rl =3D {rl_waiters =3D {tqh_first =3D 0x0, tqh_last =3D 0xffff= f8065de47360}, rl_currdep =3D 0x0}, v_cstart =3D 0, > v_lasta =3D 0, v_lastw =3D 0, v_clen =3D 0, v_holdcnt =3D 3, v_usecou= nt =3D 1, v_iflag =3D 512, v_vflag =3D 0, v_writecount =3D 0, v_hash =3D = 106816625, v_type =3D VDIR} >=20 > (kgdb) frame 15 > #15 0xffffffff809d76dd in kern_linkat (td=3D0xfffff8005cd2a490, fd1=3D<= value optimized out>, fd2=3D-100, path1=3D, path2=3D= 0x80141c600
, > segflg=3DUIO_USERSPACE, follow=3D64) at /usr/src/sys/kern/vfs_sysca= lls.c:1572 > 1572 if ((error =3D namei(&nd)) =3D=3D 0) { > (kgdb) info locals > mp =3D (struct mount *) 0xfffff8058f582330 > nd =3D {ni_dirp =3D 0x80141c600
, ni= _segflg =3D UIO_USERSPACE, ni_rightsneeded =3D {cr_rights =3D {1441151880= 80051200, 288230376151711744}}, ni_startdir =3D 0x0, > ni_rootdir =3D 0xfffff8005c0ea1d8, ni_topdir =3D 0x0, ni_dirfd =3D -1= 00, ni_strictrelative =3D 0, ni_filecaps =3D {fc_rights =3D {cr_rights =3D= {0, 0}}, fc_ioctls =3D 0x0, fc_nioctls =3D -1, > fc_fcntls =3D 0}, ni_vp =3D 0xfffff8065de471d8, ni_dvp =3D 0xfffff8= 0fda0bcce8, ni_pathlen =3D 63, > ni_next =3D 0xfffff80e918bb02d "/usr/share/openssl/man/man3/CMS_Signe= rInfo_get0_signer_id.3.gz", ni_loopcnt =3D 0, ni_cnd =3D {cn_nameiop =3D = 1, cn_flags =3D 134236168, > cn_thread =3D 0xfffff8005cd2a490, cn_cred =3D 0xfffff8005c22a700, c= n_lkflags =3D 2097152, > cn_pnbuf =3D 0xfffff80e918bb000 "/poudriere/data/.m/exp-10amd64-com= mit-test/02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz= ", > cn_nameptr =3D 0xfffff80e918bb02b "02/usr/share/openssl/man/man3/CM= S_SignerInfo_get0_signer_id.3.gz", cn_namelen =3D 2, cn_consume =3D 0}} > rights =3D {cr_rights =3D {144115188080051200, 288230376151711744}} > mp =3D (struct mount *) 0xfffff8058f582330 > vp =3D (struct vnode *) 0xfffff806dff493b0 > error =3D > kern_linkat > mp =3D (struct mount *) 0xfffff8058f582330 > vp =3D (struct vnode *) 0xfffff806dff493b0 >=20 >=20 > PID 76346 (mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../0= 3/.p) > Waiting on vn_lock(0xfffff80fda0bcce8) >=20 > (kgdb) thread 522 > [Switching to thread 522 (Thread 102346)]#0 sched_switch (td=3D0xfffff= 80419f86920, newtd=3D, flags=3D= ) at /usr/src/sys/kern/sched_ule.c:1932 > 1932 cpuid =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffff80419f86920, newtd=3D, flags=3D) at /usr/src/sys/kern/sched_ule.c:1932 > #1 0xffffffff80929379 in mi_switch (flags=3D260, newtd=3D0x0) at /usr/= src/sys/kern/kern_synch.c:493 > #2 0xffffffff80966af2 in sleepq_switch (wchan=3D,= pri=3D) at /usr/src/sys/kern/subr_sleepqueue.c:552 > #3 0xffffffff80966953 in sleepq_wait (wchan=3D0xfffff80fda0bcd50, pri=3D= 96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > #4 0xffffffff808fdf3a in sleeplk (lk=3D, flags=3D= , ilk=3D, wmesg=3D, pri=3D, > timo=3D) at /usr/src/sys/kern/kern_lock.c:225 > #5 0xffffffff808fd601 in __lockmgr_args (lk=3D0xfffff80fda0bcd50, flag= s=3D, ilk=3D0xfffff80fda0bcd80, wmesg=3D, pri=3D, > timo=3D, file=3D0xffffffff80fcc4f0 "/usr/src/s= ys/kern/vfs_lookup.c", line=3D691) at /usr/src/sys/kern/kern_lock.c:939 > #6 0xffffffff809be27c in vop_stdlock (ap=3D) at l= ockmgr.h:98 > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=3D, a= =3D) at vnode_if.c:2082 > #8 0xffffffff809ddbca in _vn_lock (vp=3D0xfffff80fda0bcce8, flags=3D, file=3D0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_loo= kup.c", line=3D691) at vnode_if.h:859 > #9 0xffffffff809c3536 in lookup (ndp=3D0xfffffe1248b7b910) at /usr/src= /sys/kern/vfs_lookup.c:691 > #10 0xffffffff809c2d84 in namei (ndp=3D0xfffffe1248b7b910) at /usr/src/= sys/kern/vfs_lookup.c:300 > #11 0xffffffff809daf2b in kern_mkdirat (td=3D0xfffff80419f86920, fd=3D-= 100, path=3D0x7fffffffeca1
, segflg= =3DUIO_USERSPACE, mode=3D511) > at /usr/src/sys/kern/vfs_syscalls.c:3687 > #12 0xffffffff80d38b6a in amd64_syscall (td=3D0xfffff80419f86920, trace= d=3D0) at subr_syscall.c:133 > #13 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/= exception.S:390 > #14 0x0000000800953faa in ?? () >=20 > (kgdb) p *(struct vnode*) 0xfffff80fda0bcce8 > $35 =3D {v_tag =3D 0xffffffff81f0f8ff "zfs", v_op =3D 0xffffffff81f24b5= 0, v_data =3D 0xfffff80c56bf8b80, v_mount =3D 0xfffff8005c8a8660, v_nmntv= nodes =3D {tqe_next =3D 0xfffff806d8e1d1d8, > tqe_prev =3D 0xfffff80df75d51f8}, v_un =3D {vu_mount =3D 0x0, vu_so= cket =3D 0x0, vu_cdev =3D 0x0, vu_fifoinfo =3D 0x0}, v_hashlist =3D {le_n= ext =3D 0x0, le_prev =3D 0x0}, v_cache_src =3D { > lh_first =3D 0xfffff805d7724a80}, v_cache_dst =3D {tqh_first =3D 0x= fffff805df597690, tqh_last =3D 0xfffff805df5976b0}, v_cache_dd =3D 0xffff= f805df597690, v_lock =3D {lock_object =3D { > lo_name =3D 0xffffffff81f0f8ff "zfs", lo_flags =3D 117112832, lo_= data =3D 0, lo_witness =3D 0xfffffe0000b28780}, lk_lock =3D 21, lk_exslpf= ail =3D 0, lk_timo =3D 51, lk_pri =3D 96}, v_interlock =3D { > lock_object =3D {lo_name =3D 0xffffffff80fc4bc6 "vnode interlock", = lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0xfffffe0000b1e500},= mtx_lock =3D 4}, v_vnlock =3D 0xfffff80fda0bcd50, > v_actfreelist =3D {tqe_next =3D 0xfffff806d8e1d1d8, tqe_prev =3D 0xff= fff80528f4d9f8}, v_bufobj =3D {bo_lock =3D {lock_object =3D {lo_name =3D = 0xffffffff80fcd144 "bufobj interlock", > lo_flags =3D 86179840, lo_data =3D 0, lo_witness =3D 0xfffffe00= 00b26580}, rw_lock =3D 1}, bo_ops =3D 0xffffffff814ab900, bo_object =3D 0= xfffff805fc81a300, bo_synclist =3D {le_next =3D 0x0, > le_prev =3D 0x0}, bo_private =3D 0xfffff80fda0bcce8, __bo_vnode =3D= 0xfffff80fda0bcce8, bo_clean =3D {bv_hd =3D {tqh_first =3D 0x0, tqh_last= =3D 0xfffff80fda0bce08}, bv_root =3D {pt_root =3D 0}, > bv_cnt =3D 0}, bo_dirty =3D {bv_hd =3D {tqh_first =3D 0x0, tqh_la= st =3D 0xfffff80fda0bce28}, bv_root =3D {pt_root =3D 0}, bv_cnt =3D 0}, b= o_numoutput =3D 0, bo_flag =3D 0, bo_bsize =3D 131072}, > v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf =3D 0x0, v_rl =3D {rl_wa= iters =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff80fda0bce70}, rl_currde= p =3D 0x0}, v_cstart =3D 0, v_lasta =3D 0, v_lastw =3D 0, > v_clen =3D 0, v_holdcnt =3D 3, v_usecount =3D 2, v_iflag =3D 512, v_v= flag =3D 0, v_writecount =3D 0, v_hash =3D 18, v_type =3D VDIR} >=20 Let me know if I can provide any further information. --=20 Regards, Bryan Drewery --Hb4K9m0hacv3IKadGr1XVwrOU5suiheIf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) iQEcBAEBAgAGBQJUINX8AAoJEDXXcbtuRpfPVgcH/RqaU6geH8W5vcudmtaxxQXp HDQ5OJDIoE1GjcN8VgYSjlqSSIdyQXYoGblQRuguibAII2u/d+/9kpNqI/LUhP4+ irPwezLbVJrxOJySsPy9Ib619Au6Lk8DNCME7LJvI0rhIa0NsENRBMIEnBtY1vEq UbqMzlSuunYryv3FofNXOc+79oUN6fdLoKbU1Tk63/T90yGLGHEM6xY6GZyzaYBU EjU6BWoP8OxIVCkeM9MkBkA4Ij2usp/ylj0mR5e3RUG7p7SJaY6jzMz7rGOfvme1 g8fejvLP896G1E5dVrTk/i4gDZ/UB6nBocVA+66094kLjKkxQZS7XoQE/0GZoOw= =+qmG -----END PGP SIGNATURE----- --Hb4K9m0hacv3IKadGr1XVwrOU5suiheIf-- From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 13:12:58 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E803C69; Tue, 23 Sep 2014 13:12:58 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF5D4BB7; Tue, 23 Sep 2014 13:12:57 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8NDCj41010238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Sep 2014 16:12:45 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8NDCj41010238 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8NDCiRf010237; Tue, 23 Sep 2014 16:12:44 +0300 (EEST) (envelope-from kostik) Date: Tue, 23 Sep 2014 16:12:44 +0300 From: Konstantin Belousov To: Bryan Drewery Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140923131244.GC8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5420D5FC.4030600@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 13:12:58 -0000 On Mon, Sep 22, 2014 at 09:07:56PM -0500, Bryan Drewery wrote: > This is slightly different than my last report (involving > zfs_lookup/zfs_dirent). I still need to gather information from that > previous one. > > Here is what I have on this one. Quoted so Thunderbird doesn't trash it: > > > 1. Poudriere build startup, run for N=01,...,12 in parallel: > > a. mount tmpfs on /poudriere/data/.m/exp-10amd64-commit-test/N > > /poudriere/data/.m is a ZFS mount. > > b. cpdup /poudriere/data/.m/exp-10amd64-commit-test/ref to /poudriere/data/.m/exp-10amd64-commit-test/N > > [1] c. mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/N/.p > > 2. SIGINT during this phase: > > [2] a. umount -f all tmpfs > > b. cpdup processes are *not* killed. They continue running. Most of them fail due to their dst tmpfs being gone. > > [3] c. A cpdup process managed to continue running. Since the tmpfs was gone it started trying to write to the ZFS mount under it. (/poudriere/data/.m) > > > > # procstat -kka|egrep "(lockmgr|tmpfs)" > > [3] 76137 100919 cpdup - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0x862 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa vget+0x67 cache_lookup+0x5b2 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xf1 lookup+0x5ad namei+0x4e4 kern_linkat+0x13d sys_link+0x28 filemon_wrapper_link+0x19 amd64_syscall+0x25a > > [1] 76346 102346 mkdir - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0xea1 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa lookup+0x496 namei+0x4e4 kern_mkdirat+0xcb amd64_syscall+0x25a Xfast_syscall+0xfb > > [2] 76286 102085 umount - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 _sleep+0x366 vfs_write_suspend+0xf0 vfs_write_suspend_umnt+0x38 tmpfs_unmount+0x6d dounmount+0x424 sys_unmount+0x2ec amd64_syscall+0x25a Xfast_syscall+0xfb > > > > [3] root 76137 0.7 0.0 14772 2844 3 D+ 3:12PM 0:05.28 cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02 > > [1] root 76346 0.0 0.0 8200 1916 3 D+ 3:12PM 0:00.00 mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p > > [2] root 76286 [command taken from gdb] umount -f /poudriere/data/.m/exp-10amd64-commit-test/02 > > > > Locked vnodes: > > > > 0xfffff80fda0bcce8: tag zfs, type VDIR > > usecount 2, writecount 0, refcount 3 mountedhere 0x0 > > flags () > > v_object 0xfffff805fc81a300 ref 0 pages 0 > > lock type zfs: SHARED (count 2) with exclusive waiters pending > > mount /poudriere/data/.m from zroot/poudriere/data/.m > > path /poudriere/data/.m/exp-10am64-commit-test/02 > > 0xfffff8065de471d8: tag zfs, type VDIR > > usecount 1, writecount 0, refcount 3 mountedhere 0xfffff8058f582330 > > flags () > > lock type zfs: EXCL by thread 0xfffff8047c0eb920 (pid 76286) with exclusive and shared waiters pending > > mount /poudriere/data/.m from zroot/poudriere/data/.m > > path /poudriere/data/.m > > > > > > Processes: > > > > PID 76286 (umount -f /poudriere/data/.m/exp-10amd64-commit-test/02): > > Waiting in vfs_write_suspend(mp=0xfffff8058f582330) for mn_writeopcount==0 > > > > > > (kgdb) thread 521 > > [Switching to thread 521 (Thread 102085)]#0 sched_switch (td=0xfffff8047c0eb920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff8047c0eb920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8058f5823bc, pri=119) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff80928c86 in _sleep (ident=, lock=, priority=, wmesg=, sbt=, > > pr=) at /usr/src/sys/kern/kern_synch.c:255 > > #5 0xffffffff809df0b0 in vfs_write_suspend (mp=0xfffff8058f582330, flags=) at /usr/src/sys/kern/vfs_vnops.c:1793 > > #6 0xffffffff809df308 in vfs_write_suspend_umnt (mp=0xfffff8058f582330) at /usr/src/sys/kern/vfs_vnops.c:1848 > > #7 0xffffffff822ad9ed in tmpfs_unmount (mp=0xfffff8058f582330, mntflags=) at /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vfsops.c:280 > > #8 0xffffffff809c7664 in dounmount (mp=0xfffff8058f582330, flags=134742016, td=0xfffff8047c0eb920) at /usr/src/sys/kern/vfs_mount.c:1310 > > #9 0xffffffff809c721c in sys_unmount (td=0xfffff8047c0eb920, uap=0xfffffe1248712b80) at /usr/src/sys/kern/vfs_mount.c:1201 > > #10 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8047c0eb920, traced=0) at subr_syscall.c:133 > > #11 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_writeopcount > > $12 = 1 > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_activevnodelistsize > > $13 = 1 > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_vnodecovered > > $25 = (struct vnode *) 0xfffff8065de471d8 > > (kgdb) p *(struct mount*)0xfffff8058f582330 > > $5 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff80fc4c20 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1ea00}, mtx_lock = 4}, mnt_gen = 23, > > mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffff8117c2abce8}, mnt_op = 0xffffffff822b3e40, mnt_vfc = 0xffffffff822b3f60, mnt_vnodecovered = 0xfffff8065de471d8, mnt_syncer = 0x0, > > mnt_ref = 7357, mnt_nvnodelist = {tqh_first = 0xfffff803f6a691d8, tqh_last = 0xfffff8065dc871f8}, mnt_nvnodelistsize = 7356, mnt_activevnodelist = {tqh_first = 0xfffff806dff493b0, > > tqh_last = 0xfffff806dff49470}, mnt_activevnodelistsize = 1, mnt_writeopcount = 1, mnt_kern_flag = 150994953, mnt_flag = 4096, mnt_opt = 0xfffff804e377eaf0, mnt_optnew = 0x0, > > mnt_maxsymlinklen = 0, mnt_stat = {f_version = 537068824, f_type = 135, f_flags = 4096, f_bsize = 4096, f_iosize = 4096, f_blocks = 1835008, f_bfree = 1798116, f_bavail = 1798116, > > f_files = 25690112, f_ffree = 25682842, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, > > f_fsid = {val = {-2027880658, 135}}, f_charspare = '\0' , f_fstypename = "tmpfs\000\000\000\000\000\000\000\000\000\000", > > f_mntfromname = "tmpfs", '\0' , f_mntonname = "/poudriere/data/.m/exp-10amd64-commit-test/02", '\0' }, mnt_cred = 0xfffff8047c8a4a00, > > mnt_data = 0xfffff8000bd2b800, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 3766194835, mnt_lockref = 0, mnt_secondary_writes = 0, > > mnt_secondary_accwrites = 0, mnt_susp_owner = 0xfffff8047c0eb920, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xffffffff80fa4b29 "explock", lo_flags = 108199936, > > lo_data = 0, lo_witness = 0xfffffe0000b26c80}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 0, lk_pri = 96}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0xfffff8005c842ca0}, > > mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffff8058f582650}} > > (kgdb) p (((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first) > > $30 = (struct vnode *) 0xfffff806dff493b0 > > (kgdb) p *(((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first) > > $24 = {v_tag = 0xffffffff822b2f76 "tmpfs", v_op = 0xffffffff822b3918, v_data = 0xfffff80ae07caae0, v_mount = 0xfffff8058f582330, v_nmntvnodes = {tqe_next = 0xfffff806071d11d8, > > tqe_prev = 0xfffff804d63963d0}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = { > > lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffff806dff49400}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff822b2f76 "tmpfs", lo_flags = 116588544, > > lo_data = 0, lo_witness = 0xfffffe0000b29a00}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = { > > lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff806dff49418, v_actfreelist = { > > tqe_next = 0x0, tqe_prev = 0xfffff8058f5823a8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0, > > lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff809dc520a00, bo_synclist = {le_next = 0x0, le_prev = 0x0}, > > bo_private = 0xfffff806dff493b0, __bo_vnode = 0xfffff806dff493b0, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494d0}, bv_root = {pt_root = 0}, bv_cnt = 0}, > > bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494f0}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 4096}, v_pollinfo = 0x0, > > v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff806dff49538}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, > > v_holdcnt = 1, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 115340435, v_type = VREG} > > > > > > PID 76137 (cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02): > > Waiting for vget(vp=0xfffff8065de471d8) [which is mnt_vnodecovered from umount -f] > > > > (kgdb) thread 520 > > [Switching to thread 520 (Thread 100919)]#0 sched_switch (td=0xfffff8005cd2a490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff8005cd2a490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8065de47240, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff808fdf3a in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, > > timo=) at /usr/src/sys/kern/kern_lock.c:225 > > #5 0xffffffff808fcfc2 in __lockmgr_args (lk=0xfffff8065de47240, flags=, ilk=0xfffff8065de47270, wmesg=, pri=, > > timo=, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at /usr/src/sys/kern/kern_lock.c:680 > > #6 0xffffffff809be27c in vop_stdlock (ap=) at lockmgr.h:98 > > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 > > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff8065de471d8, flags=, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 > > #9 0xffffffff809cde87 in vget (vp=0xfffff8065de471d8, flags=2097408, td=0xfffff8005cd2a490) at /usr/src/sys/kern/vfs_subr.c:2137 > > #10 0xffffffff809ba132 in cache_lookup (dvp=0xfffff80fda0bcce8, vpp=0xfffffe1247de78b8, cnp=0xfffffe1247de78e0, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:673 > > #11 0xffffffff809bb2dc in vfs_cache_lookup (ap=) at /usr/src/sys/kern/vfs_cache.c:1038 > > #12 0xffffffff80e58e41 in VOP_LOOKUP_APV (vop=, a=) at vnode_if.c:127 > > #13 0xffffffff809c364d in lookup (ndp=0xfffffe1247de7858) at vnode_if.h:54 > > #14 0xffffffff809c2d84 in namei (ndp=0xfffffe1247de7858) at /usr/src/sys/kern/vfs_lookup.c:300 > > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=, fd2=-100, path1=, path2=0x80141c600
, > > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572 > > #16 0xffffffff809d7518 in sys_link (td=0x0, uap=) at /usr/src/sys/kern/vfs_syscalls.c:1541 > > #17 0xffffffff820c64e9 in filemon_wrapper_link (td=0x0, uap=0xfffffe1247de7b80) at filemon_wrapper.c:364 > > #18 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8005cd2a490, traced=0) at subr_syscall.c:133 > > #19 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > > > (kgdb) p *(struct vnode*)0xfffff8065de471d8 > > $34 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80cb12f22e0, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff80528f4d938, > > tqe_prev = 0xfffff804d659cb30}, v_un = {vu_mount = 0xfffff8058f582330, vu_socket = 0xfffff8058f582330, vu_cdev = 0xfffff8058f582330, vu_fifoinfo = 0xfffff8058f582330}, v_hashlist = { > > le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0xfffff805d7724a80, tqh_last = 0xfffff805d7724aa0}, v_cache_dd = 0xfffff805d7724a80, > > v_lock = {lock_object = {lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 18446735296877738278, lk_exslpfail = 0, > > lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, > > mtx_lock = 4}, v_vnlock = 0xfffff8065de47240, v_actfreelist = {tqe_next = 0xfffff80528f4d938, tqe_prev = 0xfffff804d659cbd0}, v_bufobj = {bo_lock = {lock_object = { > > lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0x0, > > bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff8065de471d8, __bo_vnode = 0xfffff8065de471d8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de472f8}, > > bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de47318}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, > > bo_bsize = 131072}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff8065de47360}, rl_currdep = 0x0}, v_cstart = 0, > > v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 3, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 106816625, v_type = VDIR} > > > > (kgdb) frame 15 > > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=, fd2=-100, path1=, path2=0x80141c600
, > > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572 > > 1572 if ((error = namei(&nd)) == 0) { > > (kgdb) info locals > > mp = (struct mount *) 0xfffff8058f582330 > > nd = {ni_dirp = 0x80141c600
, ni_segflg = UIO_USERSPACE, ni_rightsneeded = {cr_rights = {144115188080051200, 288230376151711744}}, ni_startdir = 0x0, > > ni_rootdir = 0xfffff8005c0ea1d8, ni_topdir = 0x0, ni_dirfd = -100, ni_strictrelative = 0, ni_filecaps = {fc_rights = {cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, > > fc_fcntls = 0}, ni_vp = 0xfffff8065de471d8, ni_dvp = 0xfffff80fda0bcce8, ni_pathlen = 63, > > ni_next = 0xfffff80e918bb02d "/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 1, cn_flags = 134236168, > > cn_thread = 0xfffff8005cd2a490, cn_cred = 0xfffff8005c22a700, cn_lkflags = 2097152, > > cn_pnbuf = 0xfffff80e918bb000 "/poudriere/data/.m/exp-10amd64-commit-test/02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", > > cn_nameptr = 0xfffff80e918bb02b "02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", cn_namelen = 2, cn_consume = 0}} > > rights = {cr_rights = {144115188080051200, 288230376151711744}} > > mp = (struct mount *) 0xfffff8058f582330 > > vp = (struct vnode *) 0xfffff806dff493b0 > > error = > > kern_linkat > > mp = (struct mount *) 0xfffff8058f582330 > > vp = (struct vnode *) 0xfffff806dff493b0 > > > > > > PID 76346 (mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p) > > Waiting on vn_lock(0xfffff80fda0bcce8) > > > > (kgdb) thread 522 > > [Switching to thread 522 (Thread 102346)]#0 sched_switch (td=0xfffff80419f86920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff80419f86920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff80fda0bcd50, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff808fdf3a in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, > > timo=) at /usr/src/sys/kern/kern_lock.c:225 > > #5 0xffffffff808fd601 in __lockmgr_args (lk=0xfffff80fda0bcd50, flags=, ilk=0xfffff80fda0bcd80, wmesg=, pri=, > > timo=, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at /usr/src/sys/kern/kern_lock.c:939 > > #6 0xffffffff809be27c in vop_stdlock (ap=) at lockmgr.h:98 > > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 > > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff80fda0bcce8, flags=, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at vnode_if.h:859 > > #9 0xffffffff809c3536 in lookup (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:691 > > #10 0xffffffff809c2d84 in namei (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:300 > > #11 0xffffffff809daf2b in kern_mkdirat (td=0xfffff80419f86920, fd=-100, path=0x7fffffffeca1
, segflg=UIO_USERSPACE, mode=511) > > at /usr/src/sys/kern/vfs_syscalls.c:3687 > > #12 0xffffffff80d38b6a in amd64_syscall (td=0xfffff80419f86920, traced=0) at subr_syscall.c:133 > > #13 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > #14 0x0000000800953faa in ?? () > > > > (kgdb) p *(struct vnode*) 0xfffff80fda0bcce8 > > $35 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80c56bf8b80, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff806d8e1d1d8, > > tqe_prev = 0xfffff80df75d51f8}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = { > > lh_first = 0xfffff805d7724a80}, v_cache_dst = {tqh_first = 0xfffff805df597690, tqh_last = 0xfffff805df5976b0}, v_cache_dd = 0xfffff805df597690, v_lock = {lock_object = { > > lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 21, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = { > > lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff80fda0bcd50, > > v_actfreelist = {tqe_next = 0xfffff806d8e1d1d8, tqe_prev = 0xfffff80528f4d9f8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock", > > lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff805fc81a300, bo_synclist = {le_next = 0x0, > > le_prev = 0x0}, bo_private = 0xfffff80fda0bcce8, __bo_vnode = 0xfffff80fda0bcce8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce08}, bv_root = {pt_root = 0}, > > bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce28}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072}, > > v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce70}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, > > v_clen = 0, v_holdcnt = 3, v_usecount = 2, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 18, v_type = VDIR} > > > > Let me know if I can provide any further information. There is indeed some write op in flight on the tmpfs mount, which blocks unmount suspension. The 'bt all' from ddb would give some overview of the threads states. Follow https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html to gather required information. From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 20:20:30 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 95481F7F for ; Tue, 23 Sep 2014 20:20:30 +0000 (UTC) Received: from mail.lifanov.com (mail.lifanov.com [206.125.175.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7EE1A1F4 for ; Tue, 23 Sep 2014 20:20:30 +0000 (UTC) Received: by mail.lifanov.com (Postfix, from userid 58) id 97A321B38FA; Tue, 23 Sep 2014 16:20:24 -0400 (EDT) Received: from [127.0.0.1] (vnat004.nandomedia.com [166.108.31.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.lifanov.com (Postfix) with ESMTPSA id 41B7D1B2AD9 for ; Tue, 23 Sep 2014 16:20:22 -0400 (EDT) Message-ID: <5421D604.9050009@mail.lifanov.com> Date: Tue, 23 Sep 2014 16:20:20 -0400 From: Nikolai Lifanov User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: zfs bookmark [ -r ] question Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 20:20:30 -0000 I noticed that recursive bookmarks were not made available with the import of https://www.illumos.org/issues/4369 Is there a technical reason for this? - Nikolai Lifanov From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 21:20:49 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F3133E8 for ; Tue, 23 Sep 2014 21:20:49 +0000 (UTC) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 38ACAB51 for ; Tue, 23 Sep 2014 21:20:48 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA22647; Wed, 24 Sep 2014 00:20:40 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1XWXVn-000B1E-Tw; Wed, 24 Sep 2014 00:20:39 +0300 Message-ID: <5421E3EF.4080704@FreeBSD.org> Date: Wed, 24 Sep 2014 00:19:43 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: Nikolai Lifanov , freebsd-fs@FreeBSD.org Subject: Re: zfs bookmark [ -r ] question References: <5421D604.9050009@mail.lifanov.com> In-Reply-To: <5421D604.9050009@mail.lifanov.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 21:20:50 -0000 On 23/09/2014 23:20, Nikolai Lifanov wrote: > I noticed that recursive bookmarks were not made available with the > import of https://www.illumos.org/issues/4369 > > Is there a technical reason for this? As far as I can see that functionality (-r) was not implemented upstream, so it was impossible to import it. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 01:26:44 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DA6F5686; Wed, 24 Sep 2014 01:26:44 +0000 (UTC) Received: from mail.lifanov.com (mail.lifanov.com [206.125.175.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C588F7AF; Wed, 24 Sep 2014 01:26:44 +0000 (UTC) Received: by mail.lifanov.com (Postfix, from userid 58) id A83F01B3905; Tue, 23 Sep 2014 21:26:43 -0400 (EDT) Received: from app.lifanov.com (chat.lifanov.com [206.125.175.13]) by mail.lifanov.com (Postfix) with ESMTPA id 029B61B2AD9; Tue, 23 Sep 2014 21:26:41 -0400 (EDT) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 23 Sep 2014 21:26:41 -0400 From: Nikolai Lifanov To: Andriy Gapon Subject: Re: zfs bookmark [ -r ] question In-Reply-To: <5421E3EF.4080704@FreeBSD.org> References: <5421D604.9050009@mail.lifanov.com> <5421E3EF.4080704@FreeBSD.org> Message-ID: <68e59516484bc319ca4f2c773db1a61d@mail.lifanov.com> X-Sender: lifanov@mail.lifanov.com User-Agent: Roundcube Webmail/1.0.2 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 01:26:44 -0000 On 2014-09-23 17:19, Andriy Gapon wrote: > On 23/09/2014 23:20, Nikolai Lifanov wrote: >> I noticed that recursive bookmarks were not made available with the >> import of https://www.illumos.org/issues/4369 >> >> Is there a technical reason for this? > > As far as I can see that functionality (-r) was not implemented > upstream, so it > was impossible to import it. You are right. I'm looking at Illumos git repo and it's not supported. I guess the issue description at the above link is misleading. - Nikolai Lifanov From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 01:53:29 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B02A1443 for ; Wed, 24 Sep 2014 01:53:29 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 798C0A28 for ; Wed, 24 Sep 2014 01:53:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id s8O1rThM019269 for ; Wed, 24 Sep 2014 01:53:29 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id s8O1rTWT019268 for freebsd-fs@FreeBSD.org; Wed, 24 Sep 2014 01:53:29 GMT (envelope-from bdrewery) Received: (qmail 67545 invoked from network); 23 Sep 2014 20:53:25 -0500 Received: from unknown (HELO ?10.10.0.24?) (freebsd@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 23 Sep 2014 20:53:25 -0500 Message-ID: <5422240F.4080003@FreeBSD.org> Date: Tue, 23 Sep 2014 20:53:19 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> In-Reply-To: <20140923131244.GC8870@kib.kiev.ua> OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6Pp002pEspcDm8b2dMRaurCDNkosnUEng" Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 01:53:29 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6Pp002pEspcDm8b2dMRaurCDNkosnUEng Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 9/23/2014 8:12 AM, Konstantin Belousov wrote: > On Mon, Sep 22, 2014 at 09:07:56PM -0500, Bryan Drewery wrote: [...] > There is indeed some write op in flight on the tmpfs mount, which > blocks unmount suspension. The 'bt all' from ddb would give some > overview of the threads states. >=20 > Follow > https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/k= erneldebug-deadlocks.html > to gather required information. >=20 I tried your patch and still ran into the deadlock. Here is the debug information: https://people.freebsd.org/~bdrewery/vfs_deadlock.2.txt --=20 Regards, Bryan Drewery --6Pp002pEspcDm8b2dMRaurCDNkosnUEng Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) iQEcBAEBAgAGBQJUIiQPAAoJEDXXcbtuRpfPh7YH/jiK15pFgZ6JO4HRQjvUOvZA qGJoIDcRHyE3dONvuXG87fuEpmX91NjQkgwhhFryyVUvX7Zvh6l9KsHFkHvTK7If 5IMCS0CV+jFOsS0LIR6+AL9m5gkpioaBiu1gVcOXCW99xPBlfhT501PULmkciAl3 PGB4fvCMeGvCKSFJybiJIMVnVMe1KoEs5ebskl8ixof/kPvAcvikBqEHkf+UC9VG pcMyodZ7dmMDGDHlbZeikEoGwbpQwvQLVwe/AB+s62ZqBt0qw2asz1LS8vgTC0j7 kvBglUNVcRac68CwoLqp8DZsNdolC89aljUiXexmkwEfPtT4vi096+LOSER8PYY= =C0Wq -----END PGP SIGNATURE----- --6Pp002pEspcDm8b2dMRaurCDNkosnUEng-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 02:40:22 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E289775; Wed, 24 Sep 2014 02:40:22 +0000 (UTC) Received: from odin.blazingdot.com (odin.blazingdot.com [204.109.60.170]) by mx1.freebsd.org (Postfix) with ESMTP id 50D99D53; Wed, 24 Sep 2014 02:40:22 +0000 (UTC) Received: by odin.blazingdot.com (Postfix, from userid 1001) id 19549131944; Tue, 23 Sep 2014 22:40:15 -0400 (EDT) Date: Tue, 23 Sep 2014 22:40:15 -0400 From: Marcus Reid To: Nikolai Lifanov Subject: Re: zfs bookmark [ -r ] question Message-ID: <20140924024014.GA49256@blazingdot.com> References: <5421D604.9050009@mail.lifanov.com> <5421E3EF.4080704@FreeBSD.org> <68e59516484bc319ca4f2c773db1a61d@mail.lifanov.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <68e59516484bc319ca4f2c773db1a61d@mail.lifanov.com> X-Coffee-Level: nearly-fatal User-Agent: Mutt/1.5.23 (2014-03-12) Cc: freebsd-fs@freebsd.org, Andriy Gapon X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 02:40:22 -0000 On Tue, Sep 23, 2014 at 09:26:41PM -0400, Nikolai Lifanov wrote: > On 2014-09-23 17:19, Andriy Gapon wrote: > > On 23/09/2014 23:20, Nikolai Lifanov wrote: > >> I noticed that recursive bookmarks were not made available with the > >> import of https://www.illumos.org/issues/4369 > >> > >> Is there a technical reason for this? > > > > As far as I can see that functionality (-r) was not implemented > > upstream, so it > > was impossible to import it. > > You are right. I'm looking at Illumos git repo and it's not supported. I > guess the issue description at the above link is misleading. I looked into this as well. As I recall, the stated reason for not supporting -r is that they had no need for it, and since bookmarks use a different interface than snapshots it would have been a lot of work. It would definitely be a nice feature to have though. Marcus From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 10:28:06 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BCAEACEF; Wed, 24 Sep 2014 10:28:06 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 420E4238; Wed, 24 Sep 2014 10:28:05 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OARx4U018719 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 13:27:59 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OARx4U018719 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OARwaQ018717; Wed, 24 Sep 2014 13:27:58 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 13:27:58 +0300 From: Konstantin Belousov To: Bryan Drewery Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924102758.GH8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5422240F.4080003@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 10:28:06 -0000 On Tue, Sep 23, 2014 at 08:53:19PM -0500, Bryan Drewery wrote: > I tried your patch and still ran into the deadlock. Here is the debug > information: https://people.freebsd.org/~bdrewery/vfs_deadlock.2.txt I see what is going on. Previous patch cannot help there. The problem is that kern_linkat() starts write with vn_start_write(9), and then tries to execute namei(9) with a path potentially crossing the mount point. If the mount point is being unmounted, unmount cannot lay suspension on the filesystem due to writer, but it owns the covered vnode lock. On the other hand, namei(9) is unable to cross the mount point since covered vnode is locked, thus writer does not make progress. The similar issue exists in rename code. The deadlock is only possible for filesystems which suspend writes on unmount; such fs are UFS and tmpfs. Try this. diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c index b3b7ed5..9775480 100644 --- a/sys/kern/vfs_syscalls.c +++ b/sys/kern/vfs_syscalls.c @@ -1551,10 +1551,10 @@ kern_linkat(struct thread *td, int fd1, int fd2, char *path1, char *path2, cap_rights_t rights; int error; +again: bwillwrite(); NDINIT_AT(&nd, LOOKUP, follow | AUDITVNODE1, segflg, path1, fd1, td); -again: if ((error = namei(&nd)) != 0) return (error); NDFREE(&nd, NDF_ONLY_PNBUF); @@ -1563,14 +1563,11 @@ again: vrele(vp); return (EPERM); /* POSIX */ } - if ((error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) { - vrele(vp); - return (error); - } NDINIT_ATRIGHTS(&nd, CREATE, LOCKPARENT | SAVENAME | AUDITVNODE2, segflg, path2, fd2, cap_rights_init(&rights, CAP_LINKAT), td); if ((error = namei(&nd)) == 0) { if (nd.ni_vp != NULL) { + NDFREE(&nd, NDF_ONLY_PNBUF); if (nd.ni_dvp == nd.ni_vp) vrele(nd.ni_dvp); else @@ -1587,26 +1584,41 @@ again: error = EXDEV; else error = can_hardlink(vp, td->td_ucred); - if (error == 0) #ifdef MAC + if (error == 0) error = mac_vnode_check_link(td->td_ucred, nd.ni_dvp, vp, &nd.ni_cnd); - if (error == 0) #endif - error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + return (error); + } + error = vn_start_write(vp, &mp, V_NOWAIT); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + error = vn_start_write(NULL, &mp, + V_XSLEEP | PCATCH); + if (error != 0) + return (error); + goto again; + } + error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); VOP_UNLOCK(vp, 0); vput(nd.ni_dvp); + vn_finished_write(mp); + NDFREE(&nd, NDF_ONLY_PNBUF); } else { vput(nd.ni_dvp); NDFREE(&nd, NDF_ONLY_PNBUF); vrele(vp); - vn_finished_write(mp); goto again; } - NDFREE(&nd, NDF_ONLY_PNBUF); } vrele(vp); - vn_finished_write(mp); return (error); } @@ -3519,6 +3531,7 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, cap_rights_t rights; int error; +again: bwillwrite(); #ifdef MAC NDINIT_ATRIGHTS(&fromnd, DELETE, LOCKPARENT | LOCKLEAF | SAVESTART | @@ -3539,14 +3552,6 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, VOP_UNLOCK(fromnd.ni_vp, 0); #endif fvp = fromnd.ni_vp; - if (error == 0) - error = vn_start_write(fvp, &mp, V_WAIT | PCATCH); - if (error != 0) { - NDFREE(&fromnd, NDF_ONLY_PNBUF); - vrele(fromnd.ni_dvp); - vrele(fvp); - goto out1; - } NDINIT_ATRIGHTS(&tond, RENAME, LOCKPARENT | LOCKLEAF | NOCACHE | SAVESTART | AUDITVNODE2, pathseg, new, newfd, cap_rights_init(&rights, CAP_LINKAT), td); @@ -3559,11 +3564,28 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, NDFREE(&fromnd, NDF_ONLY_PNBUF); vrele(fromnd.ni_dvp); vrele(fvp); - vn_finished_write(mp); goto out1; } tdvp = tond.ni_dvp; tvp = tond.ni_vp; + error = vn_start_write(fvp, &mp, V_NOWAIT); + if (error != 0) { + NDFREE(&fromnd, NDF_ONLY_PNBUF); + NDFREE(&tond, NDF_ONLY_PNBUF); + if (tvp != NULL) + vput(tvp); + if (tdvp == tvp) + vrele(tdvp); + else + vput(tdvp); + vrele(fromnd.ni_dvp); + vrele(fvp); + vrele(tond.ni_startdir); + error = vn_start_write(NULL, &mp, V_WAIT | PCATCH); + if (error != 0) + return (error); + goto again; + } if (tvp != NULL) { if (fvp->v_type == VDIR && tvp->v_type != VDIR) { error = ENOTDIR; From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 13:32:48 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C9F77B76 for ; Wed, 24 Sep 2014 13:32:48 +0000 (UTC) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) by mx1.freebsd.org (Postfix) with SMTP id 7E84BC64 for ; Wed, 24 Sep 2014 13:32:47 +0000 (UTC) Received: (qmail 69732 invoked from network); 24 Sep 2014 13:26:07 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay01.pair.com with SMTP; 24 Sep 2014 13:26:07 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8ODQ649012026; Wed, 24 Sep 2014 15:26:06 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8ODQ6T9012025; Wed, 24 Sep 2014 15:26:06 +0200 (CEST) (envelope-from pho) Date: Wed, 24 Sep 2014 15:26:05 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924132605.GA11772@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924102758.GH8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 13:32:48 -0000 On Wed, Sep 24, 2014 at 01:27:58PM +0300, Konstantin Belousov wrote: > On Tue, Sep 23, 2014 at 08:53:19PM -0500, Bryan Drewery wrote: > > I tried your patch and still ran into the deadlock. Here is the debug > > information: https://people.freebsd.org/~bdrewery/vfs_deadlock.2.txt > > I see what is going on. Previous patch cannot help there. > > The problem is that kern_linkat() starts write with vn_start_write(9), > and then tries to execute namei(9) with a path potentially crossing > the mount point. If the mount point is being unmounted, unmount cannot > lay suspension on the filesystem due to writer, but it owns the covered > vnode lock. On the other hand, namei(9) is unable to cross the mount > point since covered vnode is locked, thus writer does not make progress. > > The similar issue exists in rename code. The deadlock is only possible > for filesystems which suspend writes on unmount; such fs are UFS and tmpfs. > > Try this. > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > index b3b7ed5..9775480 100644 I had been trying different scenarios without much luck, but with your description I got the problem instantly. :) The patch is an improvement, but: http://people.freebsd.org/~pho/stress/log/kostik718.txt - Peter From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 13:47:31 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CEB67EA; Wed, 24 Sep 2014 13:47:31 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 55888DC0; Wed, 24 Sep 2014 13:47:30 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8ODlPdu021559 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 16:47:25 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8ODlPdu021559 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8ODlPYp021558; Wed, 24 Sep 2014 16:47:25 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 16:47:25 +0300 From: Konstantin Belousov To: Peter Holm Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924134725.GI8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924132605.GA11772@x2.osted.lan> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 13:47:31 -0000 On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > The patch is an improvement, but: > > http://people.freebsd.org/~pho/stress/log/kostik718.txt Does you load included both rename and link, or only one of those syscalls ? I see a bug in the rename part of the patch, below is the update. If the mntref issue is not solved, please try to isolate which one of rename or link syscalls cause it. diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c index b3b7ed5..f2669ca 100644 --- a/sys/kern/vfs_syscalls.c +++ b/sys/kern/vfs_syscalls.c @@ -1551,10 +1551,10 @@ kern_linkat(struct thread *td, int fd1, int fd2, char *path1, char *path2, cap_rights_t rights; int error; +again: bwillwrite(); NDINIT_AT(&nd, LOOKUP, follow | AUDITVNODE1, segflg, path1, fd1, td); -again: if ((error = namei(&nd)) != 0) return (error); NDFREE(&nd, NDF_ONLY_PNBUF); @@ -1563,14 +1563,11 @@ again: vrele(vp); return (EPERM); /* POSIX */ } - if ((error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) { - vrele(vp); - return (error); - } NDINIT_ATRIGHTS(&nd, CREATE, LOCKPARENT | SAVENAME | AUDITVNODE2, segflg, path2, fd2, cap_rights_init(&rights, CAP_LINKAT), td); if ((error = namei(&nd)) == 0) { if (nd.ni_vp != NULL) { + NDFREE(&nd, NDF_ONLY_PNBUF); if (nd.ni_dvp == nd.ni_vp) vrele(nd.ni_dvp); else @@ -1587,26 +1584,41 @@ again: error = EXDEV; else error = can_hardlink(vp, td->td_ucred); - if (error == 0) #ifdef MAC + if (error == 0) error = mac_vnode_check_link(td->td_ucred, nd.ni_dvp, vp, &nd.ni_cnd); - if (error == 0) #endif - error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + return (error); + } + error = vn_start_write(vp, &mp, V_NOWAIT); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + error = vn_start_write(NULL, &mp, + V_XSLEEP | PCATCH); + if (error != 0) + return (error); + goto again; + } + error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); VOP_UNLOCK(vp, 0); vput(nd.ni_dvp); + vn_finished_write(mp); + NDFREE(&nd, NDF_ONLY_PNBUF); } else { vput(nd.ni_dvp); NDFREE(&nd, NDF_ONLY_PNBUF); vrele(vp); - vn_finished_write(mp); goto again; } - NDFREE(&nd, NDF_ONLY_PNBUF); } vrele(vp); - vn_finished_write(mp); return (error); } @@ -3519,6 +3531,7 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, cap_rights_t rights; int error; +again: bwillwrite(); #ifdef MAC NDINIT_ATRIGHTS(&fromnd, DELETE, LOCKPARENT | LOCKLEAF | SAVESTART | @@ -3539,14 +3552,6 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, VOP_UNLOCK(fromnd.ni_vp, 0); #endif fvp = fromnd.ni_vp; - if (error == 0) - error = vn_start_write(fvp, &mp, V_WAIT | PCATCH); - if (error != 0) { - NDFREE(&fromnd, NDF_ONLY_PNBUF); - vrele(fromnd.ni_dvp); - vrele(fvp); - goto out1; - } NDINIT_ATRIGHTS(&tond, RENAME, LOCKPARENT | LOCKLEAF | NOCACHE | SAVESTART | AUDITVNODE2, pathseg, new, newfd, cap_rights_init(&rights, CAP_LINKAT), td); @@ -3559,11 +3564,28 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, NDFREE(&fromnd, NDF_ONLY_PNBUF); vrele(fromnd.ni_dvp); vrele(fvp); - vn_finished_write(mp); goto out1; } tdvp = tond.ni_dvp; tvp = tond.ni_vp; + error = vn_start_write(fvp, &mp, V_NOWAIT); + if (error != 0) { + NDFREE(&fromnd, NDF_ONLY_PNBUF); + NDFREE(&tond, NDF_ONLY_PNBUF); + if (tvp != NULL) + vput(tvp); + if (tdvp == tvp) + vrele(tdvp); + else + vput(tdvp); + vrele(fromnd.ni_dvp); + vrele(fvp); + vrele(tond.ni_startdir); + error = vn_start_write(NULL, &mp, V_XSLEEP | PCATCH); + if (error != 0) + return (error); + goto again; + } if (tvp != NULL) { if (fvp->v_type == VDIR && tvp->v_type != VDIR) { error = ENOTDIR; From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 15:21:18 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6F001E1 for ; Wed, 24 Sep 2014 15:21:18 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 95E06BB8 for ; Wed, 24 Sep 2014 15:21:18 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id s8OFLI0r091611 for ; Wed, 24 Sep 2014 15:21:18 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id s8OFLIve091610 for freebsd-fs@FreeBSD.org; Wed, 24 Sep 2014 15:21:18 GMT (envelope-from bdrewery) Received: (qmail 20704 invoked from network); 24 Sep 2014 10:21:13 -0500 Received: from unknown (HELO ?10.10.0.24?) (freebsd@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 24 Sep 2014 10:21:13 -0500 Message-ID: <5422E15F.8050702@FreeBSD.org> Date: Wed, 24 Sep 2014 10:21:03 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Peter Holm , Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> In-Reply-To: <20140924132605.GA11772@x2.osted.lan> OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6L95Hio9mHWwXDUslpWojPu5fbsVNbV1C" Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:21:18 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --6L95Hio9mHWwXDUslpWojPu5fbsVNbV1C Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 9/24/2014 8:26 AM, Peter Holm wrote: > On Wed, Sep 24, 2014 at 01:27:58PM +0300, Konstantin Belousov wrote: >> On Tue, Sep 23, 2014 at 08:53:19PM -0500, Bryan Drewery wrote: >>> I tried your patch and still ran into the deadlock. Here is the debug= >>> information: https://people.freebsd.org/~bdrewery/vfs_deadlock.2.txt >> >> I see what is going on. Previous patch cannot help there. >> >> The problem is that kern_linkat() starts write with vn_start_write(9),= >> and then tries to execute namei(9) with a path potentially crossing >> the mount point. If the mount point is being unmounted, unmount canno= t >> lay suspension on the filesystem due to writer, but it owns the covere= d >> vnode lock. On the other hand, namei(9) is unable to cross the mount >> point since covered vnode is locked, thus writer does not make progres= s. >> >> The similar issue exists in rename code. The deadlock is only possible= >> for filesystems which suspend writes on unmount; such fs are UFS and t= mpfs. >> >> Try this. >> >> diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c >> index b3b7ed5..9775480 100644 >=20 > I had been trying different scenarios without much luck, but with > your description I got the problem instantly. :) >=20 > The patch is an improvement, but: >=20 > http://people.freebsd.org/~pho/stress/log/kostik718.txt >=20 > - Peter >=20 By the way this was my test script: http://dpaste.com/35AYZZ7 It could be simplified a lot. --=20 Regards, Bryan Drewery --6L95Hio9mHWwXDUslpWojPu5fbsVNbV1C Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) iQEcBAEBAgAGBQJUIuFiAAoJEDXXcbtuRpfP7AgIAJfRpz4TgSHCmhkV5dWjuROZ S78/01sl2VNzhK/O1dNSdVEkPmoKbTRGO9nbecLytaVqjUZk8EbbL0DEG07srDoB znXg1b8ASglw2lxqNfQ/849FvwHtHb3aDSNIALBiOctrIPngm4PgnIm5jKTaaAQJ 4j9DwTJ93d5mOw2JzaKZ0Ro7WBmCpx+D9hvfVYPo1sPbvtp5ujkRG+I92iJswq8U +3iiJYBjhFCYZmD3HBV4O7Bbvqa1w3Ip/0UPdsYhwNSlPT6wEL5ujdIcCRFanegy u7zJ+of/dS9mqq/cfc6CIyOJ1uERs9pF5fcLuRntFY//upwVmaZn58ct2uyc7rg= =FsIY -----END PGP SIGNATURE----- --6L95Hio9mHWwXDUslpWojPu5fbsVNbV1C-- From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 15:34:17 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E40D47B1 for ; Wed, 24 Sep 2014 15:34:17 +0000 (UTC) Received: from relay03.pair.com (relay03.pair.com [209.68.5.17]) by mx1.freebsd.org (Postfix) with SMTP id 990A8DFD for ; Wed, 24 Sep 2014 15:34:17 +0000 (UTC) Received: (qmail 24600 invoked from network); 24 Sep 2014 15:34:15 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay03.pair.com with SMTP; 24 Sep 2014 15:34:15 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8OFYEUP016055; Wed, 24 Sep 2014 17:34:15 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8OFYE9e016054; Wed, 24 Sep 2014 17:34:14 +0200 (CEST) (envelope-from pho) Date: Wed, 24 Sep 2014 17:34:14 +0200 From: Peter Holm To: Bryan Drewery Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924153414.GA15739@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <5422E15F.8050702@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5422E15F.8050702@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:34:18 -0000 On Wed, Sep 24, 2014 at 10:21:03AM -0500, Bryan Drewery wrote: > On 9/24/2014 8:26 AM, Peter Holm wrote: > > On Wed, Sep 24, 2014 at 01:27:58PM +0300, Konstantin Belousov wrote: > >> On Tue, Sep 23, 2014 at 08:53:19PM -0500, Bryan Drewery wrote: > >>> I tried your patch and still ran into the deadlock. Here is the debug > >>> information: https://people.freebsd.org/~bdrewery/vfs_deadlock.2.txt > >> > >> I see what is going on. Previous patch cannot help there. > >> > >> The problem is that kern_linkat() starts write with vn_start_write(9), > >> and then tries to execute namei(9) with a path potentially crossing > >> the mount point. If the mount point is being unmounted, unmount cannot > >> lay suspension on the filesystem due to writer, but it owns the covered > >> vnode lock. On the other hand, namei(9) is unable to cross the mount > >> point since covered vnode is locked, thus writer does not make progress. > >> > >> The similar issue exists in rename code. The deadlock is only possible > >> for filesystems which suspend writes on unmount; such fs are UFS and tmpfs. > >> > >> Try this. > >> > >> diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > >> index b3b7ed5..9775480 100644 > > > > I had been trying different scenarios without much luck, but with > > your description I got the problem instantly. :) > > > > The patch is an improvement, but: > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > - Peter > > > > By the way this was my test script: > > http://dpaste.com/35AYZZ7 > > It could be simplified a lot. > Thank you. Here's what I came up with: http://people.freebsd.org/~pho/link.sh - Peter From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 15:37:28 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A510089B for ; Wed, 24 Sep 2014 15:37:28 +0000 (UTC) Received: from relay03.pair.com (relay03.pair.com [209.68.5.17]) by mx1.freebsd.org (Postfix) with SMTP id 40944E30 for ; Wed, 24 Sep 2014 15:37:27 +0000 (UTC) Received: (qmail 23631 invoked from network); 24 Sep 2014 15:30:46 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay03.pair.com with SMTP; 24 Sep 2014 15:30:46 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8OFUjJ6015924; Wed, 24 Sep 2014 17:30:46 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8OFUjFS015920; Wed, 24 Sep 2014 17:30:45 +0200 (CEST) (envelope-from pho) Date: Wed, 24 Sep 2014 17:30:45 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924153045.GA15685@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924134725.GI8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:37:28 -0000 On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > The patch is an improvement, but: > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > Does you load included both rename and link, or only one of those > syscalls ? I see a bug in the rename part of the patch, below is > the update. > Both. I have split the tests in two now. Uptime is by now one hour. I'll let that run for a few hours more, before switching to random tests. I did get this page fault once: http://people.freebsd.org/~pho/stress/log/kostik719.txt but I guess it's unrelated? I have recompiled uma_core.c and vm_pageout with "-O0" in case it shows up again. - Peter From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 15:57:34 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35CA4D19; Wed, 24 Sep 2014 15:57:34 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CA4E112B; Wed, 24 Sep 2014 15:57:33 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OFvSqJ023445 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 18:57:28 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OFvSqJ023445 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OFvS6L023444; Wed, 24 Sep 2014 18:57:28 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 18:57:28 +0300 From: Konstantin Belousov To: Peter Holm Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924155728.GN8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924153045.GA15685@x2.osted.lan> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 15:57:34 -0000 On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > The patch is an improvement, but: > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > Does you load included both rename and link, or only one of those > > syscalls ? I see a bug in the rename part of the patch, below is > > the update. > > > > Both. I have split the tests in two now. Uptime is by now one hour. > I'll let that run for a few hours more, before switching to random > tests. > > I did get this page fault once: > http://people.freebsd.org/~pho/stress/log/kostik719.txt > but I guess it's unrelated? I have recompiled uma_core.c and > vm_pageout with "-O0" in case it shows up again. This looks unrelated. But, in the log, I see user-controllable LOR caused by my patch. Please use the following update instead. diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c index b3b7ed5..a4aa19e 100644 --- a/sys/kern/vfs_syscalls.c +++ b/sys/kern/vfs_syscalls.c @@ -1551,10 +1551,10 @@ kern_linkat(struct thread *td, int fd1, int fd2, char *path1, char *path2, cap_rights_t rights; int error; +again: bwillwrite(); NDINIT_AT(&nd, LOOKUP, follow | AUDITVNODE1, segflg, path1, fd1, td); -again: if ((error = namei(&nd)) != 0) return (error); NDFREE(&nd, NDF_ONLY_PNBUF); @@ -1563,50 +1563,65 @@ again: vrele(vp); return (EPERM); /* POSIX */ } - if ((error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) { - vrele(vp); - return (error); - } NDINIT_ATRIGHTS(&nd, CREATE, LOCKPARENT | SAVENAME | AUDITVNODE2, segflg, path2, fd2, cap_rights_init(&rights, CAP_LINKAT), td); if ((error = namei(&nd)) == 0) { if (nd.ni_vp != NULL) { + NDFREE(&nd, NDF_ONLY_PNBUF); if (nd.ni_dvp == nd.ni_vp) vrele(nd.ni_dvp); else vput(nd.ni_dvp); vrele(nd.ni_vp); - error = EEXIST; - } else if ((error = vn_lock(vp, LK_EXCLUSIVE)) == 0) { + vrele(vp); + return (EEXIST); + } else if (nd.ni_dvp->v_mount != vp->v_mount) { /* - * Check for cross-device links. No need to - * recheck vp->v_type, since it cannot change - * for non-doomed vnode. + * Cross-device link. No need to recheck + * vp->v_type, since it cannot change, except + * to VBAD. */ - if (nd.ni_dvp->v_mount != vp->v_mount) - error = EXDEV; - else - error = can_hardlink(vp, td->td_ucred); - if (error == 0) + NDFREE(&nd, NDF_ONLY_PNBUF); + vput(nd.ni_dvp); + vrele(vp); + return (EXDEV); + } else if ((error = vn_lock(vp, LK_EXCLUSIVE)) == 0) { + error = can_hardlink(vp, td->td_ucred); #ifdef MAC + if (error == 0) error = mac_vnode_check_link(td->td_ucred, nd.ni_dvp, vp, &nd.ni_cnd); - if (error == 0) #endif - error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + return (error); + } + error = vn_start_write(vp, &mp, V_NOWAIT); + if (error != 0) { + vput(vp); + vput(nd.ni_dvp); + NDFREE(&nd, NDF_ONLY_PNBUF); + error = vn_start_write(NULL, &mp, + V_XSLEEP | PCATCH); + if (error != 0) + return (error); + goto again; + } + error = VOP_LINK(nd.ni_dvp, vp, &nd.ni_cnd); VOP_UNLOCK(vp, 0); vput(nd.ni_dvp); + vn_finished_write(mp); + NDFREE(&nd, NDF_ONLY_PNBUF); } else { vput(nd.ni_dvp); NDFREE(&nd, NDF_ONLY_PNBUF); vrele(vp); - vn_finished_write(mp); goto again; } - NDFREE(&nd, NDF_ONLY_PNBUF); } vrele(vp); - vn_finished_write(mp); return (error); } @@ -3519,6 +3534,7 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, cap_rights_t rights; int error; +again: bwillwrite(); #ifdef MAC NDINIT_ATRIGHTS(&fromnd, DELETE, LOCKPARENT | LOCKLEAF | SAVESTART | @@ -3539,14 +3555,6 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, VOP_UNLOCK(fromnd.ni_vp, 0); #endif fvp = fromnd.ni_vp; - if (error == 0) - error = vn_start_write(fvp, &mp, V_WAIT | PCATCH); - if (error != 0) { - NDFREE(&fromnd, NDF_ONLY_PNBUF); - vrele(fromnd.ni_dvp); - vrele(fvp); - goto out1; - } NDINIT_ATRIGHTS(&tond, RENAME, LOCKPARENT | LOCKLEAF | NOCACHE | SAVESTART | AUDITVNODE2, pathseg, new, newfd, cap_rights_init(&rights, CAP_LINKAT), td); @@ -3559,11 +3567,28 @@ kern_renameat(struct thread *td, int oldfd, char *old, int newfd, char *new, NDFREE(&fromnd, NDF_ONLY_PNBUF); vrele(fromnd.ni_dvp); vrele(fvp); - vn_finished_write(mp); goto out1; } tdvp = tond.ni_dvp; tvp = tond.ni_vp; + error = vn_start_write(fvp, &mp, V_NOWAIT); + if (error != 0) { + NDFREE(&fromnd, NDF_ONLY_PNBUF); + NDFREE(&tond, NDF_ONLY_PNBUF); + if (tvp != NULL) + vput(tvp); + if (tdvp == tvp) + vrele(tdvp); + else + vput(tdvp); + vrele(fromnd.ni_dvp); + vrele(fvp); + vrele(tond.ni_startdir); + error = vn_start_write(NULL, &mp, V_XSLEEP | PCATCH); + if (error != 0) + return (error); + goto again; + } if (tvp != NULL) { if (fvp->v_type == VDIR && tvp->v_type != VDIR) { error = ENOTDIR; From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 17:15:12 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6E7CEEBA for ; Wed, 24 Sep 2014 17:15:12 +0000 (UTC) Received: from relay03.pair.com (relay03.pair.com [209.68.5.17]) by mx1.freebsd.org (Postfix) with SMTP id 31CD9D2C for ; Wed, 24 Sep 2014 17:15:11 +0000 (UTC) Received: (qmail 47440 invoked from network); 24 Sep 2014 17:15:10 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay03.pair.com with SMTP; 24 Sep 2014 17:15:10 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8OHF9lR019172; Wed, 24 Sep 2014 19:15:09 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8OHF9aZ019171; Wed, 24 Sep 2014 19:15:09 +0200 (CEST) (envelope-from pho) Date: Wed, 24 Sep 2014 19:15:09 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924171509.GA18965@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924155728.GN8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 17:15:12 -0000 On Wed, Sep 24, 2014 at 06:57:28PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > > The patch is an improvement, but: > > > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > > > Does you load included both rename and link, or only one of those > > > syscalls ? I see a bug in the rename part of the patch, below is > > > the update. > > > > > > > Both. I have split the tests in two now. Uptime is by now one hour. > > I'll let that run for a few hours more, before switching to random > > tests. > > > > I did get this page fault once: > > http://people.freebsd.org/~pho/stress/log/kostik719.txt > > but I guess it's unrelated? I have recompiled uma_core.c and > > vm_pageout with "-O0" in case it shows up again. > > This looks unrelated. But, in the log, I see user-controllable LOR > caused by my patch. Please use the following update instead. > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > index b3b7ed5..a4aa19e 100644 > --- a/sys/kern/vfs_syscalls.c Seems unchanged to me? 20140924 19:08:31 all (1/2): link.sh lock order reversal: 1st 0xfffff800b06ce068 ufs (ufs) @ kern/vfs_subr.c:2137 2nd 0xfffffe0785edfeb8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 3rd 0xfffff800b06a6548 ufs (ufs) @ kern/vfs_subr.c:2137 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19150 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19200 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db19290 __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db193c0 ffs_lock() at ffs_lock+0x92/frame 0xfffffe081db19410 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19440 _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db194b0 vget() at vget+0x67/frame 0xfffffe081db194f0 vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe081db19540 ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe081db195d0 softdep_sync_buf() at softdep_sync_buf+0xac0/frame 0xfffffe081db196b0 ffs_syncvnode() at ffs_syncvnode+0x286/frame 0xfffffe081db19730 ffs_sync() at ffs_sync+0x20f/frame 0xfffffe081db197f0 dounmount() at dounmount+0x3da/frame 0xfffffe081db19870 sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- lock order reversal: 1st 0xfffff800290fa068 ufs (ufs) @ kern/vfs_mount.c:1223 2nd 0xfffff800b0214068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1375 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19370 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19420 witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db194b0 __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db195e0 vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081db19600 VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19630 _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db196a0 ffs_flushfiles() at ffs_flushfiles+0x120/frame 0xfffffe081db19710 softdep_flushfiles() at softdep_flushfiles+0x232/frame 0xfffffe081db19780 ffs_unmount() at ffs_unmount+0xe5/frame 0xfffffe081db197f0 dounmount() at dounmount+0x424/frame 0xfffffe081db19870 sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- 20140924 19:10:34 all (2/2): link2.sh with FreeBSD 11.0-CURRENT (PHO) #0 r272060M: Wed Sep 24 19:00:20 CEST 2014 - Peter From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 17:43:22 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1B707716; Wed, 24 Sep 2014 17:43:22 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B0A50BA; Wed, 24 Sep 2014 17:43:21 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8OHhFD5024502 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Sep 2014 20:43:15 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8OHhFD5024502 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8OHhFve024501; Wed, 24 Sep 2014 20:43:15 +0300 (EEST) (envelope-from kostik) Date: Wed, 24 Sep 2014 20:43:15 +0300 From: Konstantin Belousov To: Peter Holm Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140924174315.GO8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> <20140924171509.GA18965@x2.osted.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924171509.GA18965@x2.osted.lan> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 17:43:22 -0000 On Wed, Sep 24, 2014 at 07:15:09PM +0200, Peter Holm wrote: > On Wed, Sep 24, 2014 at 06:57:28PM +0300, Konstantin Belousov wrote: > > On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > > > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > > > The patch is an improvement, but: > > > > > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > > > > > Does you load included both rename and link, or only one of those > > > > syscalls ? I see a bug in the rename part of the patch, below is > > > > the update. > > > > > > > > > > Both. I have split the tests in two now. Uptime is by now one hour. > > > I'll let that run for a few hours more, before switching to random > > > tests. > > > > > > I did get this page fault once: > > > http://people.freebsd.org/~pho/stress/log/kostik719.txt > > > but I guess it's unrelated? I have recompiled uma_core.c and > > > vm_pageout with "-O0" in case it shows up again. > > > > This looks unrelated. But, in the log, I see user-controllable LOR > > caused by my patch. Please use the following update instead. > > > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > > index b3b7ed5..a4aa19e 100644 > > --- a/sys/kern/vfs_syscalls.c > > Seems unchanged to me? > > 20140924 19:08:31 all (1/2): link.sh > lock order reversal: > 1st 0xfffff800b06ce068 ufs (ufs) @ kern/vfs_subr.c:2137 > 2nd 0xfffffe0785edfeb8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 > 3rd 0xfffff800b06a6548 ufs (ufs) @ kern/vfs_subr.c:2137 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19150 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19200 > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db19290 > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db193c0 > ffs_lock() at ffs_lock+0x92/frame 0xfffffe081db19410 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19440 > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db194b0 > vget() at vget+0x67/frame 0xfffffe081db194f0 > vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe081db19540 > ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe081db195d0 > softdep_sync_buf() at softdep_sync_buf+0xac0/frame 0xfffffe081db196b0 > ffs_syncvnode() at ffs_syncvnode+0x286/frame 0xfffffe081db19730 > ffs_sync() at ffs_sync+0x20f/frame 0xfffffe081db197f0 > dounmount() at dounmount+0x3da/frame 0xfffffe081db19870 > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > lock order reversal: > 1st 0xfffff800290fa068 ufs (ufs) @ kern/vfs_mount.c:1223 > 2nd 0xfffff800b0214068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1375 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19370 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19420 > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db194b0 > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db195e0 > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081db19600 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19630 > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db196a0 > ffs_flushfiles() at ffs_flushfiles+0x120/frame 0xfffffe081db19710 > softdep_flushfiles() at softdep_flushfiles+0x232/frame 0xfffffe081db19780 > ffs_unmount() at ffs_unmount+0xe5/frame 0xfffffe081db197f0 > dounmount() at dounmount+0x424/frame 0xfffffe081db19870 > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > 20140924 19:10:34 all (2/2): link2.sh > > with > FreeBSD 11.0-CURRENT (PHO) #0 r272060M: Wed Sep 24 19:00:20 CEST 2014 No, these two are known and harmless. The patch added new LOR, where you link between two different mount points. The code first locked vnodes, and only then checked for EXDEV. The log show some like ufs/tmpfs etc. From owner-freebsd-fs@FreeBSD.ORG Wed Sep 24 19:08:25 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C0C22416 for ; Wed, 24 Sep 2014 19:08:25 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 842C0CBD for ; Wed, 24 Sep 2014 19:08:25 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id s8OJ8Opx064749 for ; Wed, 24 Sep 2014 19:08:24 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id s8OJ8O5v064748 for freebsd-fs@freebsd.org; Wed, 24 Sep 2014 19:08:24 GMT (envelope-from bdrewery) Received: (qmail 51959 invoked from network); 24 Sep 2014 14:08:22 -0500 Received: from unknown (HELO ?10.10.0.24?) (freebsd@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 24 Sep 2014 14:08:22 -0500 Message-ID: <5423169F.9040503@FreeBSD.org> Date: Wed, 24 Sep 2014 14:08:15 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: Konstantin Belousov , Peter Holm Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> In-Reply-To: <20140924155728.GN8870@kib.kiev.ua> OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="paTVKq18cpIHSMIHCp1BG8KVFPpb9tCgU" Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Sep 2014 19:08:25 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --paTVKq18cpIHSMIHCp1BG8KVFPpb9tCgU Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 9/24/2014 10:57 AM, Konstantin Belousov wrote: > This looks unrelated. But, in the log, I see user-controllable LOR > caused by my patch. Please use the following update instead. This patch has been working well for me too. --=20 Regards, Bryan Drewery --paTVKq18cpIHSMIHCp1BG8KVFPpb9tCgU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) iQEcBAEBAgAGBQJUIxafAAoJEDXXcbtuRpfPhf8H/iz2yICysC1qmP7EdXKQgOJ9 z7ZhINGh06SZpdHMJ9umP4qdgbwnjf0MQjCr2nHlT2UAlprGGUGF5EVHHqQkpqBU 8iuhJp9VMCxVgXjP43xS0xZnIVYStkjAdDK2Zq22IJoyb15bSPHujVGuayADl8X7 FVInLfg2zqSa+dhTCeiAF3eLq3aoTOde2dp6erw+8U1pWwRNQqcrTKWH069v3H0p l1fg4oFepAmu/xL4fH2COGTh1KNt/bRQp/rrDa3MMrLKcWUaywGW/bNNFOPNs0WK 2joBZEENuIcUqOUkTtverghKi4AcuJjkKLJOqRnE9gmoXTmJQDtEqLV3N0rMkHA= =i5TG -----END PGP SIGNATURE----- --paTVKq18cpIHSMIHCp1BG8KVFPpb9tCgU-- From owner-freebsd-fs@FreeBSD.ORG Thu Sep 25 07:05:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50FF869 for ; Thu, 25 Sep 2014 07:05:00 +0000 (UTC) Received: from relay01.pair.com (relay01.pair.com [209.68.5.15]) by mx1.freebsd.org (Postfix) with SMTP id 11E91CBF for ; Thu, 25 Sep 2014 07:04:59 +0000 (UTC) Received: (qmail 46722 invoked from network); 25 Sep 2014 07:04:58 -0000 Received: from 87.58.146.155 (HELO x2.osted.lan) (87.58.146.155) by relay01.pair.com with SMTP; 25 Sep 2014 07:04:58 -0000 X-pair-Authenticated: 87.58.146.155 Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.14.5/8.14.5) with ESMTP id s8P74uWO045229; Thu, 25 Sep 2014 09:04:56 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.14.5/8.14.5/Submit) id s8P74u90045222; Thu, 25 Sep 2014 09:04:56 +0200 (CEST) (envelope-from pho) Date: Thu, 25 Sep 2014 09:04:56 +0200 From: Peter Holm To: Konstantin Belousov Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140925070456.GA43144@x2.osted.lan> References: <5420D5FC.4030600@FreeBSD.org> <20140923131244.GC8870@kib.kiev.ua> <5422240F.4080003@FreeBSD.org> <20140924102758.GH8870@kib.kiev.ua> <20140924132605.GA11772@x2.osted.lan> <20140924134725.GI8870@kib.kiev.ua> <20140924153045.GA15685@x2.osted.lan> <20140924155728.GN8870@kib.kiev.ua> <20140924171509.GA18965@x2.osted.lan> <20140924174315.GO8870@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140924174315.GO8870@kib.kiev.ua> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Sep 2014 07:05:00 -0000 On Wed, Sep 24, 2014 at 08:43:15PM +0300, Konstantin Belousov wrote: > On Wed, Sep 24, 2014 at 07:15:09PM +0200, Peter Holm wrote: > > On Wed, Sep 24, 2014 at 06:57:28PM +0300, Konstantin Belousov wrote: > > > On Wed, Sep 24, 2014 at 05:30:45PM +0200, Peter Holm wrote: > > > > On Wed, Sep 24, 2014 at 04:47:25PM +0300, Konstantin Belousov wrote: > > > > > On Wed, Sep 24, 2014 at 03:26:05PM +0200, Peter Holm wrote: > > > > > > The patch is an improvement, but: > > > > > > > > > > > > http://people.freebsd.org/~pho/stress/log/kostik718.txt > > > > > > > > > > Does you load included both rename and link, or only one of those > > > > > syscalls ? I see a bug in the rename part of the patch, below is > > > > > the update. > > > > > > > > > > > > > Both. I have split the tests in two now. Uptime is by now one hour. > > > > I'll let that run for a few hours more, before switching to random > > > > tests. > > > > > > > > I did get this page fault once: > > > > http://people.freebsd.org/~pho/stress/log/kostik719.txt > > > > but I guess it's unrelated? I have recompiled uma_core.c and > > > > vm_pageout with "-O0" in case it shows up again. > > > > > > This looks unrelated. But, in the log, I see user-controllable LOR > > > caused by my patch. Please use the following update instead. > > > > > > diff --git a/sys/kern/vfs_syscalls.c b/sys/kern/vfs_syscalls.c > > > index b3b7ed5..a4aa19e 100644 > > > --- a/sys/kern/vfs_syscalls.c > > > > Seems unchanged to me? > > > > 20140924 19:08:31 all (1/2): link.sh > > lock order reversal: > > 1st 0xfffff800b06ce068 ufs (ufs) @ kern/vfs_subr.c:2137 > > 2nd 0xfffffe0785edfeb8 bufwait (bufwait) @ ufs/ffs/ffs_vnops.c:261 > > 3rd 0xfffff800b06a6548 ufs (ufs) @ kern/vfs_subr.c:2137 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19150 > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19200 > > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db19290 > > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db193c0 > > ffs_lock() at ffs_lock+0x92/frame 0xfffffe081db19410 > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19440 > > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db194b0 > > vget() at vget+0x67/frame 0xfffffe081db194f0 > > vfs_hash_get() at vfs_hash_get+0xe1/frame 0xfffffe081db19540 > > ffs_vgetf() at ffs_vgetf+0x40/frame 0xfffffe081db195d0 > > softdep_sync_buf() at softdep_sync_buf+0xac0/frame 0xfffffe081db196b0 > > ffs_syncvnode() at ffs_syncvnode+0x286/frame 0xfffffe081db19730 > > ffs_sync() at ffs_sync+0x20f/frame 0xfffffe081db197f0 > > dounmount() at dounmount+0x3da/frame 0xfffffe081db19870 > > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > > lock order reversal: > > 1st 0xfffff800290fa068 ufs (ufs) @ kern/vfs_mount.c:1223 > > 2nd 0xfffff800b0214068 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1375 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe081db19370 > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe081db19420 > > witness_checkorder() at witness_checkorder+0xdc2/frame 0xfffffe081db194b0 > > __lockmgr_args() at __lockmgr_args+0x9d2/frame 0xfffffe081db195e0 > > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe081db19600 > > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xfc/frame 0xfffffe081db19630 > > _vn_lock() at _vn_lock+0xd2/frame 0xfffffe081db196a0 > > ffs_flushfiles() at ffs_flushfiles+0x120/frame 0xfffffe081db19710 > > softdep_flushfiles() at softdep_flushfiles+0x232/frame 0xfffffe081db19780 > > ffs_unmount() at ffs_unmount+0xe5/frame 0xfffffe081db197f0 > > dounmount() at dounmount+0x424/frame 0xfffffe081db19870 > > sys_unmount() at sys_unmount+0x2ec/frame 0xfffffe081db199a0 > > amd64_syscall() at amd64_syscall+0x278/frame 0xfffffe081db19ab0 > > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe081db19ab0 > > --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800891bca, rsp = 0x7fffffffdf08, rbp = 0x7fffffffe020 --- > > 20140924 19:10:34 all (2/2): link2.sh > > > > with > > FreeBSD 11.0-CURRENT (PHO) #0 r272060M: Wed Sep 24 19:00:20 CEST 2014 > > No, these two are known and harmless. The patch added new LOR, > where you link between two different mount points. The code first > locked vnodes, and only then checked for EXDEV. The log show some > like ufs/tmpfs etc. Yes thank you, I see. There seems to be a problem with the patch, causing a umount to fail: # umount /mnt umount: unmount of /mnt failed: Device busy # http://people.freebsd.org/~pho/stress/log/kostik720.txt -- Peter From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 07:02:36 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C11391AA for ; Fri, 26 Sep 2014 07:02:36 +0000 (UTC) Received: from mail.jrv.org (rrcs-24-73-246-106.sw.biz.rr.com [24.73.246.106]) by mx1.freebsd.org (Postfix) with ESMTP id 906517A9 for ; Fri, 26 Sep 2014 07:02:36 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id DA33F1790C8 for ; Fri, 26 Sep 2014 01:42:57 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id eNQKoWMu6GBG for ; Fri, 26 Sep 2014 01:42:48 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id EC4E21790C3 for ; Fri, 26 Sep 2014 01:42:47 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id t4hO_jjT1JfY for ; Fri, 26 Sep 2014 01:42:47 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id C02AC1790C0 for ; Fri, 26 Sep 2014 01:42:47 -0500 (CDT) Message-ID: <54250AE9.6070609@jrv.org> Date: Fri, 26 Sep 2014 01:42:49 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: zfs recv hangs in kmem arena on STABLE10 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 07:02:36 -0000 FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 r272070M: Wed Sep 24 17:36:56 CDT 2014 james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 With current STABLE10 I am unable to replicate a ZFS pool using zfs send/recv without zfs hanging in state "kmem arena", within the first 4TB or so (of a 23TB Pool). The most recent attempt used this command line SUPERTEX:/root# zfs send -R BIGTEX/UNIX@syssnap | ssh BLACKIE zfs recv -duvF BIGTOX though local replications fail in kmem arena too. The two machines I've been attempting this on have 16BG and 32GB of RAM each and are otherwise idle. Any suggestions on how to get around, or investigate, "kmem arena"? # top last pid: 3272; load averages: 0.22, 0.22, 0.23 up 0+08:25:02 01:32:07 34 processes: 1 running, 33 sleeping CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% idle Mem: 21M Active, 82M Inact, 15G Wired, 28M Cache, 450M Free ARC: 12G Total, 24M MFU, 12G MRU, 23M Anon, 216M Header, 47M Other Swap: 16G Total, 16G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1173 root 1 52 0 86476K 7780K select 0 124:33 0.00% sshd 1176 root 1 46 0 87276K 47732K kmem a 3 48:36 0.00% zfs 968 root 32 20 0 12344K 1888K rpcsvc 0 0:13 0.00% nfsd 1009 root 1 20 0 25452K 2864K select 3 0:01 0.00% ntpd ... From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 10:10:38 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8248761A; Fri, 26 Sep 2014 10:10:38 +0000 (UTC) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 48731F30; Fri, 26 Sep 2014 10:10:38 +0000 (UTC) Received: from [127.0.0.1] (nat.in.devexperts.com [89.113.128.63]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPSA id 166B656403; Fri, 26 Sep 2014 14:10:28 +0400 (MSK) Message-ID: <54253B93.1020906@FreeBSD.org> Date: Fri, 26 Sep 2014 14:10:27 +0400 From: Lev Serebryakov Reply-To: lev@FreeBSD.org Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org, freebsd-emulation@freebsd.org Subject: Special bhyve-oriented =?UTF-8?B?wqtuZXR3b3JrwrsgZmlsZXN5c3RlbT8=?= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 10:10:38 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 It is only idea to discuss: maybe, when bhyve is not «experimental» anymore we need special filesystem to «tunnel» host filesystems to guest OS (FreeBSD, at least)? Now it could be accomplished with NFS, of course, but, maybe, it is worth to think about special FS which cross VM boundary not via UDP/TCP? What do you think? - -- // Lev Serebryakov -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQJ8BAEBCgBmBQJUJTuTXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRGOTZEMUNBMEI1RjQzMThCNjc0QjMzMEFF QUIwM0M1OEJGREM0NzhGAAoJEOqwPFi/3EeP1dsP/i1e8qy5+kMyrDSiGw/3Gt4R abhaJlfxSecadnw58+AX1mjJmZWJILhTUh2mduKxIAxtsOginDfMhsftiix8oh9d unNzdLWXUcRvObBZzvMu32c01gu7nGM44szAGCwa4vRx+DvW9JGehIl4PQLOQUsh S6SsRZwqC71NxVVbWiHK60BKK7QW7M3AaMGwHXkZVUE1Ibji1xQOxiCMnb6BiQJs f+xHcS85gVxHGX6ct7Tm0mTXbbKmRKt+KrIgZOs3ct3J+rb/fHKXgHtssC7z+kHr MfsWHQnTw6/bLOkob7/wAd6nlHI21rskpUee4oAa/xIDrBpBM+NaV/9sjZOMhMg7 HM1Zd0fPIVFBYOr7GTru5e5yxbJIJ4qXuuELaqmqxjUG0AaxNo+VPMG9eE8/tTFw BVTL3aJ3j5RUcZ3PnZTU7CFzdHaa7KkcbInEAvkf7yPf2GE8gfOlssIHUPCU6aEA xZJP1cis36JBArwzTJOJPmhF6S+H/e0ijBWey9CSfEHL/yq8A+8u5IuWf2DLK+36 UFgs1uyzGvX7G/jIuBcU3SuU/joKD8sQvoiqNQpJshyGJcmH6YtuiVG1Y21jjLTr CJQLDPNgyVk8v1rfj4vQG4YX6sceHSgNlLuLHZU46FOYunRuXtRJYazmYI5Gb8bt Zd2vTGZtOv7mQrODdLYu =8g8T -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 12:49:23 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 283C2AF2 for ; Fri, 26 Sep 2014 12:49:23 +0000 (UTC) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "people.fsn.hu", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CFA9D69E for ; Fri, 26 Sep 2014 12:49:22 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id 8F4C21504086; Fri, 26 Sep 2014 14:49:12 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 5.7265] X-CRM114-CacheID: sfid-20140926_14491_FDE4533F X-CRM114-Status: Good ( pR: 5.7265 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Fri Sep 26 14:49:12 2014 X-DSPAM-Confidence: 0.8513 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 542560c812461657611632 X-DSPAM-Factors: 27, From*"Nagy, Attila" , 0.00012, boot, 0.00311, cache, 0.00311, Received*(japan.t+online.co.hu, 0.00466, Received*(japan.t, 0.00466, Received*online.co.hu, 0.00466, Received*[195.228.243.99]), 0.00466, Received*online.co.hu+[195.228.243.99]), 0.00466, byte, 0.00569, byte, 0.00569, compression, 0.00851, is+enabled, 0.00851, enabled, 0.00927, zpool, 0.01000, egrep, 0.01000, From*Attila"+; Fri, 26 Sep 2014 14:49:10 +0200 (CEST) Message-ID: <542560C1.9070207@fsn.hu> Date: Fri, 26 Sep 2014 14:49:05 +0200 From: "Nagy, Attila" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-fs Subject: 16 exabytes of L2ARC? Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 12:49:23 -0000 Hi, Running stable/10@r271944: # zpool iostat -v capacity operations bandwidth pool alloc free read write read write ---------- ----- ----- ----- ----- ----- ----- data 17.3T 40.7T 165 1.24K 1.63M 90.8M da0 4.31T 10.2T 41 318 418K 22.7M da1 4.32T 10.2T 41 317 416K 22.7M da2 4.32T 10.2T 41 317 416K 22.7M da3 4.31T 10.2T 41 317 418K 22.7M cache - - - - - - ada0 513G 16.0E 222 179 1.05M 2.79M ada1 511G 16.0E 222 180 1.05M 2.80M ---------- ----- ----- ----- ----- ----- ----- # egrep 'ada.*MB' /var/run/dmesg.boot ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes) ada0: 381554MB (781422768 512 byte sectors: 16H 63S/T 16383C) ada1: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes) ada1: 381554MB (781422768 512 byte sectors: 16H 63S/T 16383C) What did go wrong and what should I expect? lz4 compression is enabled on the pool and it has ashift=12. From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 21:48:00 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2175072C; Fri, 26 Sep 2014 21:48:00 +0000 (UTC) Received: from caida.org (rommie.caida.org [192.172.226.78]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09D61CD4; Fri, 26 Sep 2014 21:47:59 +0000 (UTC) Message-ID: <5425DF0E.4040406@caida.org> Date: Fri, 26 Sep 2014 14:47:58 -0700 From: Daniel Andersen User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: Process enters unkillable state and somewhat wedges zfs References: <53F25402.1020907@caida.org> <201408271639.09352.jhb@freebsd.org> <53FE4C9F.7030406@caida.org> <5842681.mjgMD2kESs@ralph.baldwin.cx> In-Reply-To: <5842681.mjgMD2kESs@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 21:48:00 -0000 Okay, we were finally able to collect a trace on this. I took two, just in case. basically did a tr , continue, go back into ddb and did another. Tracing pid 89483 tid 101168 td 0xfffff8048e301920 cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 trap() at trap+0x42/frame 0xfffffe2f395e1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 --- trap 0x13, rip = 0xffffffff808aada0, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bf60 --- __lockmgr_args() at __lockmgr_args+0x30/frame 0xfffffe2ffca2bf60 vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe2ffca2bf80 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9d/frame 0xfffffe2ffca2bfb0 _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 namei() at namei+0x504/frame 0xfffffe2ffca2c480 vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- Tracing pid 89483 tid 101168 td 0xfffff8048e301920 cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 trap() at trap+0x42/frame 0xfffffe2f395e1f20 nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 --- trap 0x13, rip = 0xffffffff80e6110e, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bfb0 --- VOP_LOCK1_APV() at VOP_LOCK1_APV+0xae/frame 0xfffffe2ffca2bfb0 _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 namei() at namei+0x504/frame 0xfffffe2ffca2c480 vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- On 08/29/2014 11:24 AM, John Baldwin wrote: > On Wednesday, August 27, 2014 02:24:47 PM Daniel Andersen wrote: >> On 08/27/2014 01:39 PM, John Baldwin wrote: >>> These are all blocked in "zfs" then. (For future reference, the 'mwchan' >>> field that you see as 'STATE' in top or via 'ps O mwchan' is more detailed >>> than the 'D' state.) >>> >>> To diagnose this further, you would need to see which thread holds the >>> ZFS vnode lock these threads need. I have some gdb scripts you can use to >>> do that at www.freebsd.org/~jhb/gdb/. You would want to download 'gdb6*' >>> files from there and then do this as root: >>> >>> # cd /path/to/gdb/files >>> # kgdb >>> (kgdb) source gdb6 >>> (kgdb) sleepchain 42335 >>> >>> Where '42335' is the pid of some process stuck in "zfs". >> >> I will keep this in mind the next time the machine wedges. Another data >> point: the second procstat output I sent was the most recent. All the >> processes listed there were after the fact. The process that started the >> entire problem ( this time ) was sudo, and it only has this one entry in >> procstat: >> >> 38003 102797 sudo - >> >> Of note, this does not appear to be blocked on zfs in anyway. 'ps' showed >> it in 'R' state instead of 'D' ( I will be sure to use mwchan in the >> future. ) It appeared to be pegging an entire CPU core at 100% usage, as >> well, and was only single threaded. > > Well, if it is spinning in some sort of loop in the kernel while holding a > ZFS vnode lock that could be blocking all the other threads. In that case, > you don't need to do what I asked for above. Instead, we need to find out > what that thread is doing. There are two ways of doing this. One is to > force a panic via 'sysctl debug.kdb.panic=1' and then use kgdb on the > crashdump to determine what the running thread is doing. Another option > is to break into the DDB debugger on the console (note that you will need > to build a custom kernel with DDB if you are on stable) and request a > stack trace of the running process via 'tr '. Ideally you can do this > over a serial console so you can just cut and paste the output of the trace > into a mail. Over a video console you can either transcribe it by hand or > take photos. > From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 22:04:48 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E8EC78AF for ; Fri, 26 Sep 2014 22:04:47 +0000 (UTC) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com [209.85.192.45]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A82F3E70 for ; Fri, 26 Sep 2014 22:04:47 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id q108so9621451qgd.18 for ; Fri, 26 Sep 2014 15:04:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=eh9ps449MQgQrJg0fhteGI7h+WDTcVvDRV0zcyYAOcw=; b=WVWwq8JBwLWKsQTA5i2LRcBqHK+rPfWTdK/C81IivHB99XBAsiQy4G6N2muHYFNCEf T2ikSJ/KEMbC9TuvEjFosBnxmWDS+Ay9YNqZGhhjhNYkWWLFf552kVi+KChzLtTwsk34 HtDRhE1++MPNs5OnpCOMvc8SwDvtyQB/9rWu70DmQxl0x6ku4oShbTdRmrr3VQDQckXS 3yC9IMMYYfRX6zDxE1S9eqGApldWgIJG05XjzM64+WPP2KRn+naa5D9T8ZfcvIUFdViL yDaXe3YCqqdoq47ORVBjnOrowrk3qkMgTmfxoiNBUnzIGhTA6WpAw9lEPNh4Zo2HyKIH iPgw== X-Gm-Message-State: ALoCoQlFs/labrToJkp96/BafuC8sVjQKzm0DqP6qn1PuaqtaM3t9e4Yc+vjeVGYMwUOZqUkTzIT MIME-Version: 1.0 X-Received: by 10.140.92.98 with SMTP id a89mr33276145qge.85.1411769086042; Fri, 26 Sep 2014 15:04:46 -0700 (PDT) Received: by 10.140.16.183 with HTTP; Fri, 26 Sep 2014 15:04:45 -0700 (PDT) In-Reply-To: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> Date: Fri, 26 Sep 2014 16:04:45 -0600 Message-ID: Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. From: Will Andrews To: John Terrell Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 22:04:48 -0000 What do the zfs commands print? The -v option prints some metadata that might help diagnose the problem. --Will. On Monday, September 22, 2014, John Terrell wrote: > Hello all, I'm trying to replicate one of the ZFS filesystems that lives > on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. The transfer > dies at about the 1.7TB (of almost 4TB) mark indicating: > > "cannot receive incremental stream: invalid backup stream" > > The stream being sent is not incremental so I'm not sure what the receiver > is trying to do. Here's the command I'm using to transfer (executed on > the SmartOS machine): > > ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v tank > > I've also tried to use mbuffer as the transport interface (removing SSH > from the equation) and it has the same result. > > Is the possibly an incompatibility between FreeBSD10 and SmartOS ZFS > implementations? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > " > From owner-freebsd-fs@FreeBSD.ORG Fri Sep 26 22:07:20 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B682942 for ; Fri, 26 Sep 2014 22:07:20 +0000 (UTC) Received: from anubis.delphij.net (anubis.delphij.net [64.62.153.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anubis.delphij.net", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5167AE8E for ; Fri, 26 Sep 2014 22:07:19 +0000 (UTC) Received: from zeta.ixsystems.com (unknown [69.198.165.132]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by anubis.delphij.net (Postfix) with ESMTPSA id 503D21CD3E; Fri, 26 Sep 2014 15:07:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphij.net; s=anubis; t=1411769239; x=1411783639; bh=7MtMsuc6ILku58xIbTHWlFxHGapBSRtqWYa+EC0KUnc=; h=Date:From:Reply-To:To:Subject:References:In-Reply-To; b=SPBp7EtEAmT0LCD84z9hCYLvxTiqS4z3vyvAJBNEsQF+HbDnj3yflUfNn19xShnAg gs/h0sJo9Z4DhD2/H79o1sNZrHkh7osOXKBnC6axQY7VLU7FZHOLOS0eAMYAiHESz/ 7KQQL7GJqmlrbWU6Mk2fS2H1oN+/Y24TS1vftdZk= Message-ID: <5425E397.3020700@delphij.net> Date: Fri, 26 Sep 2014 15:07:19 -0700 From: Xin Li Reply-To: d@delphij.net Organization: The FreeBSD Project MIME-Version: 1.0 To: John Terrell , freebsd-fs@freebsd.org Subject: Re: zfs-send/receive between FreeBSD10 and SmartOS (Illumos) fails. References: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> In-Reply-To: <37BD986A-9B05-4CE8-9D55-CAB4B2713D9D@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Sep 2014 22:07:20 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 09/22/14 18:33, John Terrell wrote: > Hello all, I'm trying to replicate one of the ZFS filesystems that > lives on my FreeBSD 10 NAS to a SmartOS (Illumos based) server. > The transfer dies at about the 1.7TB (of almost 4TB) mark > indicating: > > "cannot receive incremental stream: invalid backup stream" > > The stream being sent is not incremental so I'm not sure what the > receiver is trying to do. Here's the command I'm using to > transfer (executed on the SmartOS machine): > > ssh root@fbsd10 zfs send -v -R tank@daily20140920 | zfs receive -v > tank > > I've also tried to use mbuffer as the transport interface (removing > SSH from the equation) and it has the same result. > > Is the possibly an incompatibility between FreeBSD10 and SmartOS > ZFS implementations? There shouldn't be any intentional incompatibility between FreeBSD and other OpenZFS implementations. Do you know the kernel version of SmartOS that corresponds to Illumos's revision? Cheers, - -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0 iQIcBAEBCgAGBQJUJeOXAAoJEJW2GBstM+nsqMIP/3/lYJ/JkweNl8kONXbQPu7p sy0Qnh34TRUPQRwo3mI4B/kpG5LR2yjsHrCHMsL0/17Mik1y/CDf2XkgLMn2y3gM c85Bu0qP+YIVQwCbJf2VcyCNlVnlH2gvMzeYTHAu7fZs9q5XiEWBSn2ICiG4J7ZR 8wTS9OzG/Zn3v2BRBRt/xw7m0q6m9f+OcnDj2uia72QI2gRjIMz0vJNKxTo2B78v 6znXrceMwOx9FqgPG+fm3bvYaZCHBQOMUs6MgJ2gR76FvTFRq8RawQj1E9K0eHmk q3GWMg97y03CsEBiyKrApLnthlPiwu6mfae6jIPyYVL//rXzBCFLWngdraR+E617 PFKOyhhJ5tFeYbc7x5HpKTKl+YouMomD5S2yoGm/3QMTMH50ZOheIJYaQG1G3op6 /WJcWBpdAF3HG+5QgCiRpt0/bdIznBWsP6EpSB6fVcjkEZ5C/OFCBaNTdaIsXg2A NraZH39EmKfDE9OkuwaC7JVRJKOfppio9yHB/Uy0sGretK2ai1hRpqqaXrh4YTtH ZOrfI7q9qxH8tdqkRBijKh0BSFtjWFEJdUhqmQ1voj6AbxpijKMVm26FwYi1jyr8 jTPTIL2GiaEPZ3uufpZus1lCj6aYi8FjohV5kLTcv4cSsbnTqhFnpHx0afTB8icC v2Qub7dPXXDy3gD5vl8i =Z3CR -----END PGP SIGNATURE----- From owner-freebsd-fs@FreeBSD.ORG Sat Sep 27 09:14:14 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EB45524 for ; Sat, 27 Sep 2014 09:14:14 +0000 (UTC) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "people.fsn.hu", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 021E0371 for ; Sat, 27 Sep 2014 09:14:12 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id F290C13E9667; Sat, 27 Sep 2014 11:14:07 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.3 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 9.2334] X-CRM114-CacheID: sfid-20140927_11140_31B99EA4 X-CRM114-Status: Good ( pR: 9.2334 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Sat Sep 27 11:14:07 2014 X-DSPAM-Confidence: 0.9950 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 54267fdf659991824840573 X-DSPAM-Factors: 27, From*"Nagy, Attila" , 0.00012, >+Hi, 0.00103, wrote+>, 0.00169, Hi+>, 0.00190, boot, 0.00311, cache, 0.00311, cache, 0.00311, >+>, 0.00406, >+>, 0.00406, Nagy+Attila, 0.00427, Received*(japan.t+online.co.hu, 0.00466, Received*(japan.t, 0.00466, Received*online.co.hu, 0.00466, Received*[195.228.243.99]), 0.00466, Received*online.co.hu+[195.228.243.99]), 0.00466, byte, 0.00569, byte, 0.00569, References*fsn.hu>, 0.00569, In-Reply-To*fsn.hu>, 0.00614, 14+49, 0.00639, wrote, 0.00646, v+>, 0.00730, the+cache, 0.00851, zpool, 0.01000, egrep, 0.01000, >+#, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from [192.168.0.22] (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 1571A13E9658 for ; Sat, 27 Sep 2014 11:14:02 +0200 (CEST) Message-ID: <54267FD3.2080603@fsn.hu> Date: Sat, 27 Sep 2014 11:13:55 +0200 From: "Nagy, Attila" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.1.1 MIME-Version: 1.0 To: freebsd-fs Subject: Re: 16 exabytes of L2ARC? References: <542560C1.9070207@fsn.hu> In-Reply-To: <542560C1.9070207@fsn.hu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 09:14:14 -0000 On 09/26/14 14:49, Nagy, Attila wrote: > Hi, > > Running stable/10@r271944: > # zpool iostat -v > capacity operations bandwidth > pool alloc free read write read write > ---------- ----- ----- ----- ----- ----- ----- > data 17.3T 40.7T 165 1.24K 1.63M 90.8M > da0 4.31T 10.2T 41 318 418K 22.7M > da1 4.32T 10.2T 41 317 416K 22.7M > da2 4.32T 10.2T 41 317 416K 22.7M > da3 4.31T 10.2T 41 317 418K 22.7M > cache - - - - - - > ada0 513G 16.0E 222 179 1.05M 2.79M > ada1 511G 16.0E 222 180 1.05M 2.80M > ---------- ----- ----- ----- ----- ----- ----- > > # egrep 'ada.*MB' /var/run/dmesg.boot > ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes) > ada0: 381554MB (781422768 512 byte sectors: 16H 63S/T 16383C) > ada1: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes) > ada1: 381554MB (781422768 512 byte sectors: 16H 63S/T 16383C) I've removed the cache devices and re-added them, now it's fine: cache - - - - - - ada0 355M 372G 24 0 151K 0 ada1 345M 372G 12 505 71.7K 2.79M From owner-freebsd-fs@FreeBSD.ORG Sat Sep 27 12:55:32 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A0AAAAE7; Sat, 27 Sep 2014 12:55:32 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5CD31B1E; Sat, 27 Sep 2014 12:55:32 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 356ADB946; Sat, 27 Sep 2014 08:55:31 -0400 (EDT) From: John Baldwin To: Daniel Andersen Subject: Re: Process enters unkillable state and somewhat wedges zfs Date: Sat, 27 Sep 2014 08:46:53 -0400 Message-ID: <6662003.L0FKeJoQHN@ralph.baldwin.cx> User-Agent: KMail/4.12.5 (FreeBSD/10.1-BETA2; KDE/4.12.5; amd64; ; ) In-Reply-To: <5425DF0E.4040406@caida.org> References: <53F25402.1020907@caida.org> <5842681.mjgMD2kESs@ralph.baldwin.cx> <5425DF0E.4040406@caida.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Sat, 27 Sep 2014 08:55:31 -0400 (EDT) Cc: freebsd-fs@freebsd.org, avg@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 12:55:32 -0000 On Friday, September 26, 2014 02:47:58 PM Daniel Andersen wrote: > Okay, we were finally able to collect a trace on this. I took two, just in > case. basically did a tr , continue, go back into ddb and did another. > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > --- trap 0x13, rip = 0xffffffff808aada0, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bf60 --- > __lockmgr_args() at __lockmgr_args+0x30/frame 0xfffffe2ffca2bf60 > vop_stdlock() at vop_stdlock+0x3c/frame 0xfffffe2ffca2bf80 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9d/frame 0xfffffe2ffca2bfb0 > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- > > > > Tracing pid 89483 tid 101168 td 0xfffff8048e301920 > cpustop_handler() at cpustop_handler+0x28/frame 0xfffffe2f395e1ce0 > ipi_nmi_handler() at ipi_nmi_handler+0x3f/frame 0xfffffe2f395e1d00 > trap() at trap+0x42/frame 0xfffffe2f395e1f20 > nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe2f395e1f20 > --- trap 0x13, rip = 0xffffffff80e6110e, rsp = 0xfffffe2f395e1fe0, rbp = 0xfffffe2ffca2bfb0 --- > VOP_LOCK1_APV() at VOP_LOCK1_APV+0xae/frame 0xfffffe2ffca2bfb0 > _vn_lock() at _vn_lock+0x43/frame 0xfffffe2ffca2c010 > zfs_lookup() at zfs_lookup+0x3a1/frame 0xfffffe2ffca2c0a0 > zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe2ffca2c1e0 > VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x92/frame 0xfffffe2ffca2c210 > vfs_cache_lookup() at vfs_cache_lookup+0xcf/frame 0xfffffe2ffca2c260 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c290 > null_lookup() at null_lookup+0x8b/frame 0xfffffe2ffca2c300 > VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x92/frame 0xfffffe2ffca2c330 > lookup() at lookup+0x58b/frame 0xfffffe2ffca2c3b0 > namei() at namei+0x504/frame 0xfffffe2ffca2c480 > vn_open_cred() at vn_open_cred+0x232/frame 0xfffffe2ffca2c5d0 > vop_stdvptocnp() at vop_stdvptocnp+0x17d/frame 0xfffffe2ffca2c910 > null_vptocnp() at null_vptocnp+0x2b/frame 0xfffffe2ffca2c970 > VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x98/frame 0xfffffe2ffca2c9a0 > vn_vptocnp_locked() at vn_vptocnp_locked+0x113/frame 0xfffffe2ffca2ca20 > vn_fullpath1() at vn_fullpath1+0x1b2/frame 0xfffffe2ffca2ca80 > kern___getcwd() at kern___getcwd+0x115/frame 0xfffffe2ffca2cae0 > amd64_syscall() at amd64_syscall+0x357/frame 0xfffffe2ffca2cbf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2ffca2cbf0 > --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x2000cd8cda, rsp = 0x7fffffffd358, rbp = 0x7fffffffd440 --- Hmm, if you are still in this state, can you do a ktrace of this running process and see if it is logging any syscall events (just want to make sure it is stuck in the kernel rather than constantly calling system calls due to a loop in userland)? Also, do you have a nullfs mount of a zfs path? (It looks that way from your stack trace). I did find this thread: https://www.mail-archive.com/freebsd-current@freebsd.org/msg147678.html Andriy (cc'd) probably has better insight into this than I. > On 08/29/2014 11:24 AM, John Baldwin wrote: > > On Wednesday, August 27, 2014 02:24:47 PM Daniel Andersen wrote: > >> On 08/27/2014 01:39 PM, John Baldwin wrote: > >>> These are all blocked in "zfs" then. (For future reference, the 'mwchan' > >>> field that you see as 'STATE' in top or via 'ps O mwchan' is more detailed > >>> than the 'D' state.) > >>> > >>> To diagnose this further, you would need to see which thread holds the > >>> ZFS vnode lock these threads need. I have some gdb scripts you can use to > >>> do that at www.freebsd.org/~jhb/gdb/. You would want to download 'gdb6*' > >>> files from there and then do this as root: > >>> > >>> # cd /path/to/gdb/files > >>> # kgdb > >>> (kgdb) source gdb6 > >>> (kgdb) sleepchain 42335 > >>> > >>> Where '42335' is the pid of some process stuck in "zfs". > >> > >> I will keep this in mind the next time the machine wedges. Another data > >> point: the second procstat output I sent was the most recent. All the > >> processes listed there were after the fact. The process that started the > >> entire problem ( this time ) was sudo, and it only has this one entry in > >> procstat: > >> > >> 38003 102797 sudo - > >> > >> Of note, this does not appear to be blocked on zfs in anyway. 'ps' showed > >> it in 'R' state instead of 'D' ( I will be sure to use mwchan in the > >> future. ) It appeared to be pegging an entire CPU core at 100% usage, as > >> well, and was only single threaded. > > > > Well, if it is spinning in some sort of loop in the kernel while holding a > > ZFS vnode lock that could be blocking all the other threads. In that case, > > you don't need to do what I asked for above. Instead, we need to find out > > what that thread is doing. There are two ways of doing this. One is to > > force a panic via 'sysctl debug.kdb.panic=1' and then use kgdb on the > > crashdump to determine what the running thread is doing. Another option > > is to break into the DDB debugger on the console (note that you will need > > to build a custom kernel with DDB if you are on stable) and request a > > stack trace of the running process via 'tr '. Ideally you can do this > > over a serial console so you can just cut and paste the output of the trace > > into a mail. Over a video console you can either transcribe it by hand or > > take photos. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Sat Sep 27 18:24:46 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A442A1E2 for ; Sat, 27 Sep 2014 18:24:46 +0000 (UTC) Received: from mail-vc0-x22c.google.com (mail-vc0-x22c.google.com [IPv6:2607:f8b0:400c:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 62AAFE2B for ; Sat, 27 Sep 2014 18:24:46 +0000 (UTC) Received: by mail-vc0-f172.google.com with SMTP id lf12so184999vcb.3 for ; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IFUrF0HdXyB6BJOc44SzX0RWgrMcP8FdfiRrGi9SWXc=; b=oFMP8N6gN/F3CA4ZBsoWfex3rVrtUNuNnRZ0qJ95nJhHs96lFGHnK66qLzO+4uImSg f0xtPxPgCwmLHO4hqCzyxMDi0mVs5WVt0N1loNH7gPG19W3a/0sX+mWt0bbCtfCqgiPZ s6IIpFr3lXaEVe/HzVShT73P3gRi7lhpjwgb70/frm9oK+tV3kymh0N3zYHytXy57LjP okjUyJDeB9FHbA7vWecpFgMcRcBLT8M79GWwzzbadcdIDCgIKXZtMCm8z7wT6aD6knEO G3fvavZ8s3rWQ15jpd/zWSNDkd1cf2fkRU+UQOXe4s1Zh/UMNUkwA+hXSITjcGKTpvLu 2wOQ== MIME-Version: 1.0 X-Received: by 10.220.20.5 with SMTP id d5mr21754922vcb.9.1411842285236; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) Received: by 10.31.160.7 with HTTP; Sat, 27 Sep 2014 11:24:45 -0700 (PDT) In-Reply-To: References: Date: Sat, 27 Sep 2014 14:24:45 -0400 Message-ID: Subject: Re: Unexpected zfs ARC behavior From: FF To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 18:24:46 -0000 Hi, forwarding from -questions. It looks like the image didn't make it across, so here it is: http://snag.gy/Ghb7X.jpg Thanks in advance for any pointers or suggestions, or confirmations that somehow this behavior is normal. -- So on a somewhat loaded ZFS file server (all NFS) serving mostly VMs with the load steadily increasing over the month (migrated from other servers)... the ARC size has unexpectedly dropped (please see attached graphic, if it makes through the mailing list server). The system peaks around 2,000 NFS IOPS and hasn't exhibited any slow downs. The L2ARC has a very low hit rate and prefetch has been turned off to increase the L2ARC efficiency... but none of that should really matter as far as I can tell since the L1ARC should try to use all the memory it can. > > Some tuning of zfs sysctls (mostly write_boost and write_max) to increase > them, but this has since been backed off to default. > > vmstat -m reports that 24G is dedicated to opensolaris: > solaris 1113433 24565820K - 6185559839 > 16,32,64,128,256,512,1024,2048,4096 > > And top reports memory free: > > last pid: 32075; load averages: 0.16, 0.13, > 0.14 up 36+21:44:59 09:16:49 > 24 processes: 1 running, 23 sleeping > CPU: 0.0% user, 0.0% nice, 0.0% system, 0.6% interrupt, 99.4% idle > Mem: 25M Active, 1187M Inact, 26G Wired, 2048K Cache, 1536M Buf, 4160M Free > ARC: 14G Total, 1679M MFU, 12G MRU, 28M Anon, 156M Header, 21M Other > > zfs-stats -a : > > ------------------------------------------------------------------------ > ZFS Subsystem Report Sat Sep 27 09:17:07 2014 > ------------------------------------------------------------------------ > > System Information: > > Kernel Version: 902001 (osreldate) > Hardware Platform: amd64 > Processor Architecture: amd64 > > ZFS Storage pool Version: 5000 > ZFS Filesystem Version: 5 > > FreeBSD 9.2-RELEASE-p10 #0 r270148M: Mon Aug 18 23:14:36 EDT 2014 root > 9:17AM up 36 days, 21:45, 2 users, load averages: 0.12, 0.12, 0.13 > > ------------------------------------------------------------------------ > > System Memory: > > 0.08% 26.22 MiB Active, 3.75% 1.16 GiB Inact > 82.36% 25.47 GiB Wired, 0.01% 2.00 MiB Cache > 13.80% 4.27 GiB Free, 0.00% 1.03 MiB Gap > > Real Installed: 32.00 GiB > Real Available: 99.63% 31.88 GiB > Real Managed: 97.01% 30.93 GiB > > Logical Total: 32.00 GiB > Logical Used: 83.03% 26.57 GiB > Logical Free: 16.97% 5.43 GiB > > Kernel Memory: 23.54 GiB > Data: 99.90% 23.52 GiB > Text: 0.10% 23.13 MiB > > Kernel Memory Map: 29.76 GiB > Size: 76.40% 22.74 GiB > Free: 23.60% 7.02 GiB > > ------------------------------------------------------------------------ > > ARC Summary: (HEALTHY) > Memory Throttle Count: 0 > > ARC Misc: > Deleted: 90.09m > Recycle Misses: 2.44m > Mutex Misses: 794.67k > Evict Skips: 17.90m > > ARC Size: 44.78% 13.40 GiB > Target Size: (Adaptive) 44.78% 13.40 GiB > Min Size (Hard Limit): 12.50% 3.74 GiB > Max Size (High Water): 8:1 29.93 GiB > > ARC Size Breakdown: > Recently Used Cache Size: 86.16% 11.55 GiB > Frequently Used Cache Size: 13.84% 1.85 GiB > > ARC Hash Breakdown: > Elements Max: 786.71k > Elements Current: 87.05% 684.85k > Collisions: 153.35m > Chain Max: 16 > Chains: 194.92k > > ------------------------------------------------------------------------ > > ARC Efficiency: 506.24m > Cache Hit Ratio: 87.56% 443.25m > Cache Miss Ratio: 12.44% 62.99m > Actual Hit Ratio: 80.06% 405.29m > > Data Demand Efficiency: 93.92% 372.74m > Data Prefetch Efficiency: 49.49% 69.76m > > CACHE HITS BY CACHE LIST: > Anonymously Used: 6.10% 27.05m > Most Recently Used: 31.37% 139.05m > Most Frequently Used: 60.07% 266.24m > Most Recently Used Ghost: 0.80% 3.56m > Most Frequently Used Ghost: 1.66% 7.35m > > CACHE HITS BY DATA TYPE: > Demand Data: 78.98% 350.09m > Prefetch Data: 7.79% 34.53m > Demand Metadata: 12.15% 53.86m > Prefetch Metadata: 1.08% 4.77m > > CACHE MISSES BY DATA TYPE: > Demand Data: 35.95% 22.65m > Prefetch Data: 55.93% 35.23m > Demand Metadata: 4.49% 2.83m > Prefetch Metadata: 3.63% 2.29m > > ------------------------------------------------------------------------ > > > Any suggestions? Is this expected or acceptable behavior? > > Thanks in advance, > > -- > FF > -- FF