From owner-freebsd-fs@freebsd.org Sun Sep 30 21:00:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7B8D10A7076 for ; Sun, 30 Sep 2018 21:00:43 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1951A72417 for ; Sun, 30 Sep 2018 21:00:43 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id D228D10A7070; Sun, 30 Sep 2018 21:00:42 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9833A10A706F for ; Sun, 30 Sep 2018 21:00:42 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 399117240F for ; Sun, 30 Sep 2018 21:00:42 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 6D74F24CC1 for ; Sun, 30 Sep 2018 21:00:41 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w8UL0fS1099441 for ; Sun, 30 Sep 2018 21:00:41 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w8UL0fi5099433 for fs@FreeBSD.org; Sun, 30 Sep 2018 21:00:41 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201809302100.w8UL0fi5099433@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 30 Sep 2018 21:00:41 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Sep 2018 21:00:44 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off New | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Tue Oct 2 21:32:34 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1964910ADBA4 for ; Tue, 2 Oct 2018 21:32:34 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-vs1-xe2f.google.com (mail-vs1-xe2f.google.com [IPv6:2607:f8b0:4864:20::e2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E9897D864 for ; Tue, 2 Oct 2018 21:32:33 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by mail-vs1-xe2f.google.com with SMTP id y11-v6so1996941vso.5 for ; Tue, 02 Oct 2018 14:32:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=PEaQqL2IG6c8uzfNBRXUOvLGpDAISvz3V4oQjHL8Qhs=; b=DzoNmwMcYh9HhLA/HMF7gE4scjBzzzib3bqXw49/mM8sP8EehIyAeEzzHKvopB4EyI /AlTksww47lodIZVTF7pu3FeU4UUhPE/hkYL53vX+5VetlKSEZIp30Nkj4EyrEjFjBSu Xda9K9czj40d5XgcELXDMol0RIiasUMA9TAvo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=PEaQqL2IG6c8uzfNBRXUOvLGpDAISvz3V4oQjHL8Qhs=; b=QKCs2JsItvoGbMnfCLoN42rWqFZIZdcLJd0+WCgf8RLK9s7iv15jLVv1a9WOzXaR3Y jlGggp0KZGYASXL9bkeQwqWZ8Rxo7eMylJqHIXlVk46CPjkMcV/fLkq+jQhqHTNc6tgl 7x/6DgDxPhznvxO0vac22I7YNqcp9dDAInxwn2AaeqVLVUil/gp8d18iq8VhbWEKz4HG FEjUqytGRzvvre7HF5usjhlEXBGNpGFcC8FGrr5Rx2q5UPDsnW0mEpwy7Zpj7rtLTx3X t6marfwji1EcwCKF+FVhaohfo+PoT/+1f/jvGcLCq27JLVzoTzI9j7ybJZAvl/CXtGJi H03g== X-Gm-Message-State: ABuFfojYpvccXr3bC0UVYdqk7u90SScqnCEal2pR6QjpAvxtQhvXLbYs 6gFu2OEp34w8pQqkxVoSS0MbzLsA8Df9FR4GeNz2P4/ijQE= X-Google-Smtp-Source: ACcGV628IVpZ9boYMLcklU/68Wb2Gl7bGUHSTe94Ytwh2qzJaKHVqYA5p9DMzvBTOls/fEWbJA5+P58LKbAbTtH5v88= X-Received: by 2002:a67:c982:: with SMTP id y2-v6mr6350959vsk.94.1538515952601; Tue, 02 Oct 2018 14:32:32 -0700 (PDT) MIME-Version: 1.0 From: Matthew Ahrens Date: Tue, 2 Oct 2018 14:32:21 -0700 Message-ID: Subject: First OpenZFS Leadership Meeting To: developer , illumos-zfs , zfs-devel@list.zfsonlinux.org, zfs-devel@freebsd.org, freebsd-fs , zfs-discuss Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Oct 2018 21:32:34 -0000 At this year's OpenZFS Developer Conference, we decided we should have more regular meetings to coordinate activities across all the platforms. The first meeting will be held *October 9th, 1:00-1:30pm Pacific time*, and we plan to hold this every 4 weeks thereafter. Everyone is welcome to attend and participate, and we will try to keep the meeting on agenda and on time. The meetings will be held online via Zoom , and recorded and posted to the website and YouTube after the meeting. The agenda for the first meeting will be a discussion of *feature availability on each platform*, with a goal of finding owners for everything in the "integrated to at least one platform" section. For more information and details on how to attend, please see the agenda document: https://docs.google.com/document/d/1w2jv2XVYFmBVvG1EGf-9A5HBVsjAYoLIFZAnWHhV-BM/edit --matt p.s. Videos from the OpenZFS Developer Summit are now available: http://www.open-zfs.org/wiki/OpenZFS_Developer_Summit_2018 From owner-freebsd-fs@freebsd.org Wed Oct 3 01:39:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5263510B3CCE for ; Wed, 3 Oct 2018 01:39:03 +0000 (UTC) (envelope-from jeffc@supranet.net) Received: from mailbox.supranet.net (mailbox.supranet.net [IPv6:2607:f4e0:100:111::9]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A79A785008 for ; Wed, 3 Oct 2018 01:39:02 +0000 (UTC) (envelope-from jeffc@supranet.net) Received: from [209.204.169.179] (helo=[192.168.1.205]) by mailbox.supranet.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.82 (FreeBSD)) (envelope-from ) id 1g7W7v-000GNd-HM for freebsd-fs@freebsd.org; Tue, 02 Oct 2018 20:38:59 -0500 Date: Tue, 2 Oct 2018 18:38:57 -0700 From: Jeff Chan Reply-To: Jeff Chan Message-ID: <609311573.20181002183857@supranet.net> To: freebsd-fs@freebsd.org Subject: "ZFS: can't find pool by guid" after reboot during 11.1 to 11.2 upgrade MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 01:39:03 -0000 Hi Folks, Apologies in advance if this isn't the right place to ask. Wasn't sure if it was perhaps related to the 11.2 ZFS-related boot loader changes though. During a FreeBSD upgrade from 11.1 to 11.2, after the reboot step between freebsd-update installs, got an error from the boot loader: ZFS: can't find pool by guid ZFS: can't find pool by guid ZFS: can't find pool by guid can't load 'kernel' Booting from memstick USB into liveCD to get some diagnostics: root@:~ # gpart show => 34 5860533101 da0 GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 diskid/DISK-Z1Z36VFN00009426QDUJ GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 da1 GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 da2 GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 da3 GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 diskid/DISK-Z1Z38VVZ000094274QNB GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 diskid/DISK-Z1Y084BY GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 34 5860533101 diskid/DISK-Z1Y2YWBN0000C524DDKE GPT (2.7T) 34 6 - free - (3.0K) 40 1024 1 freebsd-boot (512K) 1064 16777216 2 freebsd-swap (8.0G) 16778280 5843754848 3 freebsd-zfs (2.7T) 5860533128 7 - free - (3.5K) => 1 62652415 da4 MBR (30G) 1 1600 1 !239 (800K) 1601 1505616 2 freebsd [active] (735M) 1507217 61145199 - free - (29G) => 0 1505616 da4s2 BSD (735M) 0 16 - free - (8.0K) 16 1505600 1 freebsd-ufs (735M) => 1 62652415 diskid/DISK-0336216080014934 MBR (30G) 1 1600 1 !239 (800K) 1601 1505616 2 freebsd [active] (735M) 1507217 61145199 - free - (29G) => 0 1505616 diskid/DISK-0336216080014934s2 BSD (735M) 0 16 - free - (8.0K) 16 1505600 1 freebsd-ufs (735M) (da4 is the USB drive, obviously) root@:~ # zpool import ZFS filesystem version: 5 ZFS storage pool version: features support (5000) pool: zroot id: 12421595622411140545 state: ONLINE status: Some supported features are not enabled on the pool. action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: zroot ONLINE raidz1-0 ONLINE diskid/DISK-Z1Z36VFN00009426QDUJ ONLINE diskid/DISK-Z1Z38VVZ000094274QNB ONLINE diskid/DISK-Z1Y084BY ONLINE diskid/DISK-Z1Y2YWBN0000C524DDKE ONLINE pool: zroot id: 6087619269570541526 state: ONLINE status: Some supported features are not enabled on the pool. action: The pool can be imported using its name or numeric identifier, though some features will not be available without an explicit 'zpool upgrade'. config: zroot ONLINE raidz1-0 ONLINE da0p3 ONLINE da1p3 ONLINE da2p3 ONLINE da3p3 ONLINE Not able to import zroot by name: root@:~ # zpool import -fR /mnt zroot cannot import 'zroot': more than one matching pool import by numeric ID instead What's the best way to fix this? Also the zpool hasn't been upgraded in a long time, and I wonder if that interacted with the 11.2 boot loader code change: https://www.freebsd.org/releases/11.2R/relnotes.html#boot-loader > 9.1. Boot Loader Changes > > The boot code and loader(8) have been updated to check for > unsupported ZFS feature flags. If unsupported features are active, > the pool is not considered as a bootable pool, and a diagnostic > message is printed to the console. [r321519] and the related patch: https://svnweb.freebsd.org/base?view=revision&revision=321519 If we should do a zpool upgrade, can that be done from the liveCD safely? This is a hot spare of a production system, and it would be nice if it could be restored to operation with a few commands. There's (always) the chance that a hard drive is wearing out too, though recent zpool scrubs seem fine, and smartctl errors look ok. Much easier to fix from a running system if needed. Thanks in advance for any advice! Cheers, Jeff C. -- Jeff Chan mailto:jeffc@supranet.net From owner-freebsd-fs@freebsd.org Wed Oct 3 03:36:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42B4910B67CE for ; Wed, 3 Oct 2018 03:36:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D57388A0CB for ; Wed, 3 Oct 2018 03:36:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 9A30A10B67CC; Wed, 3 Oct 2018 03:36:41 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88F6710B67CB for ; Wed, 3 Oct 2018 03:36:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 289568A0C9 for ; Wed, 3 Oct 2018 03:36:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8830721D06 for ; Wed, 3 Oct 2018 03:36:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w933aeMN004052 for ; Wed, 3 Oct 2018 03:36:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w933aeLW004051 for fs@FreeBSD.org; Wed, 3 Oct 2018 03:36:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Wed, 03 Oct 2018 03:36:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vi@endrift.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 03:36:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 vi@endrift.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |vi@endrift.com --- Comment #18 from vi@endrift.com --- Been repeatedly hitting this over the past month. I noticed correlated issu= es with the CPU temperature monitoring, but I think those were a red herring. = One thing I can say for sure: turning off hyperthreading made this take a LOT longer to happen. My best guess is that it's a race condition with locking,= so the more cores you have, the more likely you are to hit it. Notably, this IS happening on baremetal for me, though it's in a jail. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Oct 3 06:02:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 060C010B930B for ; Wed, 3 Oct 2018 06:02:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9950B8DB09 for ; Wed, 3 Oct 2018 06:02:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5E25F10B9309; Wed, 3 Oct 2018 06:02:38 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4CFE710B9308 for ; Wed, 3 Oct 2018 06:02:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E49E98DB04 for ; Wed, 3 Oct 2018 06:02:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 25A20231C4 for ; Wed, 3 Oct 2018 06:02:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w9362b2G015962 for ; Wed, 3 Oct 2018 06:02:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w9362b4C015954 for fs@FreeBSD.org; Wed, 3 Oct 2018 06:02:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Wed, 03 Oct 2018 06:02:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: vi@endrift.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 06:02:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 --- Comment #19 from vi@endrift.com --- This is the line that the _sx_xlock_hard call that's hanging is on: https://github.com/freebsd/freebsd/blob/release/11.2.0/sys/cddl/contrib/ope= nsolaris/uts/common/fs/zfs/zil.c#L2190 (beginning of zil_commit_writer). That means that the issuer lock is being = held by something. Notably, the issuer lock can be held while waiting on a txg to sync in zil_commit_waiter_timeout (or more precisely, its call to zil_commit_writer_stall). I'm not entirely sure (since I'm unfamiliar with = this code), but I think this may be the source of the deadlock. Upon inspection, both the quiesce and sync threads appear to be idle, and I can't find any evidence that there's any other thread that's in that function, so I'm not = sure that's what's going on. Plus I have no idea what's going on with the un-ptraceable process as mentioned in comment 9, which I've observed too. It appears to be livelocked on my system (100% of one thread at all times), bu= t I can't inspect it at all--even kgdb can't touch it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Oct 3 08:08:40 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C34B510BBC10 for ; Wed, 3 Oct 2018 08:08:40 +0000 (UTC) (envelope-from Jean-Marc.Lacroix@unice.fr) Received: from ip-nice01.unice.fr (ip-nice01.unice.fr [134.59.1.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4B9EC70E0E for ; Wed, 3 Oct 2018 08:08:40 +0000 (UTC) (envelope-from Jean-Marc.Lacroix@unice.fr) X-IronPort-AV: E=Sophos;i="5.54,335,1534802400"; d="scan'208";a="234131365" Received: from naxos2.unice.fr ([134.59.1.112]) by ip-nice08.unice.fr with ESMTP; 03 Oct 2018 10:07:31 +0200 Received: from [134.59.10.29] (txmath29.unice.fr [134.59.10.29]) (authenticated bits=0) by naxos2.unice.fr (8.14.7/8.14.7) with ESMTP id w9387Vh7331062 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 3 Oct 2018 10:07:31 +0200 To: freebsd-fs@freebsd.org From: Jean-Marc LACROIX Subject: ZFS mix disks diffrent size and speed in a pool Message-ID: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> Date: Wed, 3 Oct 2018 10:07:31 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: fr X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 08:08:40 -0000 Hello,     we actually have a storage solution based on a DELL 630 + 3 JBODs DELL MD 1420 (HBA mode), The ZFS pool is made of 9 raidz2 vdevs + 1 mirror log + 2 spares     All disks are of the same type in capacity and speed. The problem is that DELL is unable to provide the SAME kind of disk (capacity AND speed) when a disk has to be replaced. So the question is: Is it possible to replace a 1TB 7200rpm by a 1.2TB 10000rpm disk without any consequences for the existing pool ? THanks in advance Best Regards JM -- LACROIX Jean-Marc office: W612 Administrateur Systèmes et Réseaux LJAD phone: 04.92.07.62.51 fax: 04.93.51.79.74 email: jml@unice.fr Address: Laboratoire J.A.Dieudonne - UMR CNRS 7351 Universite de Nice Sophia-Antipolis Parc Valrose - 06108 Nice Cedex 2 - France From owner-freebsd-fs@freebsd.org Wed Oct 3 08:12:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2E9610BBE93 for ; Wed, 3 Oct 2018 08:12:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41B6F71247 for ; Wed, 3 Oct 2018 08:12:52 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x531.google.com with SMTP id g32-v6so4457159edg.13 for ; Wed, 03 Oct 2018 01:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=GOrbfTr5niV92Xs3ZBdRXqGs26otbqNblZilWK7GVvg=; b=gFN06nkgJAOO7RwzfFGEyb+ufEvhzFO7malXC4c/LP3+4hde3w9I95kkGu7ZyEFLgm 07BWAmtaZds6cbvtcEgk1nEdRnfgI4vxv1afAr5I99PaZT7oDIwz3JTj3mKJI0SY9PuI izqTKZpPSqnDt16Yumks5jd9dP4Nzw0bHUG968w8cgvjRjm1fmljmmJzxrtT+TDnPxJu UgtAfRtBC6DD32lN88MhVpZGbVdUAD02OTvsx6zrhjUpJZtZaguZCam7glzdfyB9UbX2 4By7jtFV1Uw+527PL52kTzAFjiDW1TM/+bpY8I0skXzGTchyYWGu8U8ikYfRaC69g3Lz qBNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=GOrbfTr5niV92Xs3ZBdRXqGs26otbqNblZilWK7GVvg=; b=AzrvOUQEcdCbPaxpBKVNRGUhewsbr57dY1AGqbWdb9jiiPcEEMqkIpx+t7XvResMsl 93t/o6/4PgSwyYR+TIEJ6IsVShp1ZZpNqdOyFIrTHMmlod5tAYMPU1jv0zGxruOtsbqO ZtRY7NJYo4RxmnhqI46pT8+Zw++L4cN4wy4afYq5KKy50iAxqPbJTZwIcQKZbnc/LCVt eAQ6weenI9JTcFxbfBb8I8Jd0wgw4AZFaY722VFPcGe8NXOcXly09DRemy0zGHcWWIVq CbMfGk2P39PgLCNMqi1MIKgIWylSWhFCfb6tsZ2es9t2UYDpCdOvBRu4scYVHY9rqPQ4 W8cA== X-Gm-Message-State: ABuFfog1kdGY47vb9owGPsdrUrTVv9LR/JWGJHOL6GgFH5RrgAxoqtFl PFZLmewAKIFAyYyECtJQQrMkqASAuHHZPw== X-Google-Smtp-Source: ACcGV61XBgdWl6K4KnHafjjgLJ0gy0jovmDrDZ+xz/aDnEW7bOUUVL84taeXWD0elzsiNMMdWvCj9g== X-Received: by 2002:a17:906:2ce:: with SMTP id 14-v6mr778624ejk.66.1538554370458; Wed, 03 Oct 2018 01:12:50 -0700 (PDT) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id y14-v6sm161365ejq.21.2018.10.03.01.12.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Oct 2018 01:12:49 -0700 (PDT) Subject: Re: ZFS mix disks diffrent size and speed in a pool To: freebsd-fs@freebsd.org References: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> From: Steven Hartland Message-ID: <72e4915f-3d2f-3a3c-ada1-44f797b7244f@multiplay.co.uk> Date: Wed, 3 Oct 2018 09:12:50 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 08:12:53 -0000 That will be fine, you wont benefit from the extra space or speed but it will just work. On 03/10/2018 09:07, Jean-Marc LACROIX wrote: > Hello, > >     we actually have a storage solution based on a DELL 630 + 3 JBODs > DELL MD 1420 (HBA mode), > > The ZFS pool is made of 9 raidz2 vdevs + 1 mirror log + 2 spares > >     All disks are of the same type in capacity and speed. > > The problem is that DELL is unable to provide the SAME kind of disk > (capacity AND speed) > > when a disk has to be replaced. So the question is: > > Is it possible to replace a 1TB 7200rpm by a 1.2TB 10000rpm disk > without any consequences for the existing pool ? > > THanks in advance > > Best Regards > > JM > From owner-freebsd-fs@freebsd.org Wed Oct 3 08:49:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EA8E10BCED8 for ; Wed, 3 Oct 2018 08:49:53 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0281E72650 for ; Wed, 3 Oct 2018 08:49:52 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Date: Wed, 3 Oct 2018 10:49:44 +0200 (CEST) From: Ronald Klop To: freebsd-fs@freebsd.org Message-ID: <492380418.1.1538556584486@localhost> In-Reply-To: <72e4915f-3d2f-3a3c-ada1-44f797b7244f@multiplay.co.uk> References: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> <72e4915f-3d2f-3a3c-ada1-44f797b7244f@multiplay.co.uk> Subject: Re: ZFS mix disks diffrent size and speed in a pool MIME-Version: 1.0 X-Mailer: Realworks (427.25715-265900) Importance: Normal X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 08:49:53 -0000 After all the disks are replaced the extra space can be used. See man zpool: autoexpand=on | off Controls automatic pool expansion when the underlying LUN is grown. If set to "on", the pool will be resized according to the size of the expanded device. If the device is part of a mirror or raidz then all devices within that mirror/raidz group must be expanded before the new space is made available to the pool. The default behavior is "off". This property can also be referred to by its shortened column name, expand. So it can be nice to start using bigger disks (2+ TB) if you know you need more space in the future and the price is ok. Regards, Ronald. Van: Steven Hartland Datum: woensdag, 3 oktober 2018 10:12 Aan: freebsd-fs@freebsd.org Onderwerp: Re: ZFS mix disks diffrent size and speed in a pool > > That will be fine, you wont benefit from the extra space or speed but it will just work. > > On 03/10/2018 09:07, Jean-Marc LACROIX wrote: > > Hello, > > > > we actually have a storage solution based on a DELL 630 + 3 JBODs > DELL MD 1420 (HBA mode), > > > > The ZFS pool is made of 9 raidz2 vdevs + 1 mirror log + 2 spares > > > > All disks are of the same type in capacity and speed. > > > > The problem is that DELL is unable to provide the SAME kind of disk > (capacity AND speed) > > > > when a disk has to be replaced. So the question is: > > > > Is it possible to replace a 1TB 7200rpm by a 1.2TB 10000rpm disk > without any consequences for the existing pool ? > > > > THanks in advance > > > > Best Regards > > > > JM > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > From owner-freebsd-fs@freebsd.org Wed Oct 3 14:32:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB1B310C4ED7 for ; Wed, 3 Oct 2018 14:32:28 +0000 (UTC) (envelope-from jgitlin@goboomtown.com) Received: from mail-yw1-xc31.google.com (mail-yw1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 664AA7D307 for ; Wed, 3 Oct 2018 14:32:28 +0000 (UTC) (envelope-from jgitlin@goboomtown.com) Received: by mail-yw1-xc31.google.com with SMTP id d126-v6so2355983ywa.5 for ; Wed, 03 Oct 2018 07:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=goboomtown-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=fjnwbuzPql1MMeMXV4ao89ZK0ceN0WkrayQhvO4fmMI=; b=bL0zA3zxDJotUssjRYP0XH7LeVMlyj+pEjSZ8cIeBLof2iQIBM3doZc19g7nkmY6+Y oGYyGDY9ypjPZpZNkgIfzQUG/ckKDdhW8yDWulb7R36EJ/IZ+k9V8vLqMBczHUS9VOU/ gl7Ci/7nP5ODB56336rA3t/VWi9gj9RWGe2o9H7qRlUn4athUp4P8iGkd+u9bkQjrb77 ceSiuF5LzugNJCuKScPh7HB8bYAynUlDCEaovjrUw59li8sVerV1QXeWnawvG0vg/qoc UZ/jbeVMF+deH9OLtMabqrYFINOdjPTax1plyQzols+7bwZ/aJl+yR03AUZNer6bBEJR HkvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=fjnwbuzPql1MMeMXV4ao89ZK0ceN0WkrayQhvO4fmMI=; b=JFzYJjFe3ptwmQ03lip4uEheFv+vxN2J6k3j1XHSIAX2Dlu3ReHio3FuIGctLiKjtA e5BZQIWt02G3Qhr/21Eghlt9Itmx2P1nhzLT9EhW3GkqJEwOAYaUYRO/K8o3P0uW+LFz Edv3JkS3NfGgh9Jo6pMhkHHa1ZH/NaTbKacS9eBQovV31EcV1b6CTkVW3EvwOHxuSBvc lE0uXaZu/KHi0EvzgqFj3ooomU0aEBxm0hJfU8tliQ0PdE9sRDfh0D0ywANkxPUKceKa fE+TAveg+bTqsxR8BdPPJSl+2RcHJUWgHtM99RLlanNwPiPTQtyPxG2YpHp8mWnUxiw7 PZDw== X-Gm-Message-State: ABuFfohXJQR9tQ4lrFwi0DQpdrfq+ICy6DcEs0xn9zhXk3MA1XC04XkY +QHPMQ7c8PY/jlRmv25S3VZWauV7JRBPtw== X-Google-Smtp-Source: ACcGV63OLTDX+N1EJdASAcWLavY3Y2BRXSpzKpkANdJBot6kjT/xyAuZ8fcSo2mKOqE/MXiLUaoFdA== X-Received: by 2002:a81:81c6:: with SMTP id r189-v6mr1057852ywf.494.1538577146950; Wed, 03 Oct 2018 07:32:26 -0700 (PDT) Received: from yyz.farcry.sitepalette.com (047-134-241-203.res.spectrum.com. [47.134.241.203]) by smtp.gmail.com with ESMTPSA id e82-v6sm689716ywa.60.2018.10.03.07.32.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Oct 2018 07:32:26 -0700 (PDT) From: Josh Gitlin Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Troubleshooting kernel panic with zfs Date: Wed, 3 Oct 2018 10:32:25 -0400 In-Reply-To: To: freebsd-fs@freebsd.org References: X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 14:32:29 -0000 Following up on this, a bug was just posted to the stable@freebsd.org = list where the stack trace exactly matches = what I was seeing. See: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231296 On our end we reduced ARC and other memory tunables and have not seen a = panic *yet* but they were unpredictable before so I am not 100% sure = that we've resolved the issue. CC'ing rainer@ultra-secure.de who posted the similar bug to stable@ -- =09 Josh Gitlin Senior Full Stack Developer (415) 690-1610 x155 Stay up to date and join the conversation in Relay = . > On Sep 20, 2018, at 8:27 PM, Josh Gitlin = wrote: >=20 > I am working to debug/troubleshoot a kernel panic with a FreeBSD ZFS = iSCSI server, specifically trying to determine if it's a bug or (more = likely) a misconfiguration in our settings. Server is running = 11.2-RELEASE-p2 with 15.6 GiB of RAM and has a single zpool with 4x2 = mirrored vdevs, 2x mirrored zil and 2x l2arc. Server runs pretty much = nothing other than SSH and iSCSI (via ctld) and serves VM virtual disks = to hypervisor servers over 10gbe LAN. >=20 > The server experienced a kernel panic and we unfortunately did not = have dumpdev set in /etc/rc.conf (we have since corrected this) so the = only info I have is what was on the screen before I rebooted it. = (Because it's a production system I couldn't mess around and had to = reboot ASAP) >=20 > trap number =3D 12 > panic: page fault > cpuid =3D 6 > KDB: stack backtrace: > #0 0xffffffff80b3d567 at kdb_backtrace+0x67 > #1 0xffffffff80af6b07 at vpanic+0x177 > #2 0xffffffff80af6983 at panic+0x43 > #3 0xffffffff80f77fcf at trap_fatal+0x35f > #4 0xffffffff80f78029 at trap_pfault+0x49 > #5 0xffffffff80f777f7 at trap+0x2c7 > #6 0xffffffff80f57dac at calltrap+0x8 > #7 0xffffffff80dee7e2 at kmem_back+0xf2 > #8 0xffffffff80dee6c0 at kmem_malloc+0x60 > #9 0xffffffff80de6172 at keg_alloc_slab+0xe2 > #10 0xffffffff80de8b7e at keg_fetch_slab+0x14e > #11 0xffffffff80de8364 at zone_fetch_slab+0x64 > #12 0xffffffff80de848f at zone_import+0x3f > #13 0xffffffff80de4b99 at uma_zalloc_arg+0x3d9 > #14 0xffffffff826e6ab2 at zio_write_compress+0x1e2 > #15 0xffffffff826e574c at zio_execute+0xac > #16 0xffffffff80bled74 at taskqueue_run_locked+0x154 > #17 0xffffffff80b4fed8 at taskqueue_thread_loop+0x98 > Uptime: 18d18h31m6s > mpr0: Sending StopUnit: path (xpt0:mpr0:0:10:ffffffff): handle 10=20 > mpr0: Incrementing SSU count > mpr0: Sending StopUnit: path (xpt0:mpr0:0:13:ffffffff): handle 13=20 > mpr0: Incrementing SSU count > mpr0: Sending StopUnit: path Ixpt0:mpr0:0:16:ffffffff): handle 16=20 > mpr0: Incrementing SSU count >=20 > My hunch is that, given this was inside kmem_malloc, we were unable to = allocate memory for a zio_write_compress call (the pool does have ZFS = compression on) and hence this is a tuning issue and not a bug... but I = am looking for confirmation and/or suggested changes/troubleshooting = steps. The ZFS tuning configuration has been stable for years, to it may = be a change in behavior or traffic... If this looks like it might be a = bug, I will be able to get more information from a minidump if it = reoccurs and can follow up on this thread. >=20 > Any advice or suggestions are welcome! >=20 > [jgitlin@zfs3 ~]$ zpool status > pool: srv > state: ONLINE > scan: scrub repaired 0 in 2h32m with 0 errors on Tue Sep 11 20:32:18 = 2018 > config: >=20 > NAME STATE READ WRITE CKSUM > srv ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/s5 ONLINE 0 0 0 > gpt/s9 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > gpt/s6 ONLINE 0 0 0 > gpt/s10 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > gpt/s7 ONLINE 0 0 0 > gpt/s11 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > gpt/s8 ONLINE 0 0 0 > gpt/s12 ONLINE 0 0 0 > logs > mirror-4 ONLINE 0 0 0 > gpt/s2-zil ONLINE 0 0 0 > gpt/s3-zil ONLINE 0 0 0 > cache > gpt/s2-cache ONLINE 0 0 0 > gpt/s3-cache ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > ZFS tuning: >=20 > vfs.zfs.delay_min_dirty_percent=3D90 > vfs.zfs.dirty_data_max=3D4294967296 > vfs.zfs.dirty_data_sync=3D3221225472 > vfs.zfs.prefetch_disable=3D1 > vfs.zfs.top_maxinflight=3D128 > vfs.zfs.trim.txg_delay=3D8 > vfs.zfs.txg.timeout=3D20 > vfs.zfs.vdev.aggregation_limit=3D524288 > vfs.zfs.vdev.scrub_max_active=3D3 > vfs.zfs.l2arc_write_boost=3D134217728 > vfs.zfs.l2arc_write_max=3D134217728 > vfs.zfs.l2arc_feed_min_ms=3D200 > vfs.zfs.min_auto_ashift=3D12 >=20 >=20 > -- > =09 > Josh Gitlin > Senior DevOps Engineer > (415) 690-1610 x155 >=20 > Stay up to date and join the conversation in Relay = . >=20 From owner-freebsd-fs@freebsd.org Wed Oct 3 19:54:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 891F310AB9DD for ; Wed, 3 Oct 2018 19:54:29 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from smtp.simplesystems.org (smtp.simplesystems.org [65.66.246.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 08F7F8CD2D for ; Wed, 3 Oct 2018 19:54:28 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from scrappy.simplesystems.org (scrappy.simplesystems.org [65.66.246.73]) by smtp.simplesystems.org (8.14.4+Sun/8.14.4) with ESMTP id w93JjHp0020406; Wed, 3 Oct 2018 14:45:17 -0500 (CDT) Date: Wed, 3 Oct 2018 14:45:17 -0500 (CDT) From: Bob Friesenhahn X-X-Sender: bfriesen@scrappy.simplesystems.org To: Jean-Marc LACROIX cc: freebsd-fs@freebsd.org Subject: Re: ZFS mix disks diffrent size and speed in a pool In-Reply-To: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> Message-ID: References: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> User-Agent: Alpine 2.20 (GSO 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (smtp.simplesystems.org [65.66.246.90]); Wed, 03 Oct 2018 14:45:18 -0500 (CDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 19:54:29 -0000 On Wed, 3 Oct 2018, Jean-Marc LACROIX wrote: > > Is it possible to replace a 1TB 7200rpm by a 1.2TB 10000rpm disk without any > consequences for the existing pool ? The disk native sector size (512 bytes or 4k bytes) should be the same as existing disks in the vdev. Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ From owner-freebsd-fs@freebsd.org Thu Oct 4 06:59:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6658E10C2115 for ; Thu, 4 Oct 2018 06:59:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F3414865E8 for ; Thu, 4 Oct 2018 06:59:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B4AAC10C2111; Thu, 4 Oct 2018 06:59:55 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A350C10C210F for ; Thu, 4 Oct 2018 06:59:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D37A865E6 for ; Thu, 4 Oct 2018 06:59:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 72BDA10824 for ; Thu, 4 Oct 2018 06:59:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w946xsCE059819 for ; Thu, 4 Oct 2018 06:59:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w946xska059816 for fs@FreeBSD.org; Thu, 4 Oct 2018 06:59:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231794] Panic in ZFS due to arc related memory exhaustion Date: Thu, 04 Oct 2018 06:59:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rainer@ultra-secure.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 06:59:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231794 rainer@ultra-secure.de changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rainer@ultra-secure.de --- Comment #1 from rainer@ultra-secure.de --- See also: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231296 There's something seriously wrong with the default settings. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 4 10:55:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E1E310C8394 for ; Thu, 4 Oct 2018 10:55:42 +0000 (UTC) (envelope-from felix@audiofair.de) Received: from mo6-p00-ob.smtp.rzone.de (mo6-p00-ob.smtp.rzone.de [IPv6:2a01:238:20a:202:5300::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.smtp.rzone.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE0528F655 for ; Thu, 4 Oct 2018 10:55:41 +0000 (UTC) (envelope-from felix@audiofair.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1538650539; s=strato-dkim-0002; d=audiofair.de; h=Date:Message-ID:Subject:From:To:X-RZG-CLASS-ID:X-RZG-AUTH:From: Subject:Sender; bh=+FVLkRSlIe9O76sINCbr4JBbNmNIZhPICywyY27qK4Q=; b=mtPcC8JyAJ4NRBbyaxwJBhdhujXXDq1nLAsQLvjInEmtEIZDG/vhX1oGRFbMoquyu7 grFB0SawbEaitBS6COR1BQ/WGZXF1diEJ1RnrSvJoLViYRcrApkh4TY7R+FlJebdminr k0bFGfG+6ZfsKREZtIm5UbV50kXAJcIdb3vxS6efxMFxdKEueqtC6+Tdprezs9EvfDpi q0/5wRBsvC2FQoz6b3ZNmZx272RoMST7tLzCnPYISJHaEpf13EXt41zGEdSQ9LuCaZxo +v3M43zd3vXuBfv5dCADAHAHJuqpVNcZAcRQL5Lei8XUggDRlEnV1tin5uZ+Ut9PQuU8 osaA== X-RZG-AUTH: ":KmkIfFiIeuobF0ryGDOmdGJTFqaolieRUBb8W7m+IlV8fTgVa1I1Nqqo7krB5f1Lh7qYsj9h" X-RZG-CLASS-ID: mo00 Received: from [131.188.166.118] by smtp.strato.de (RZmta 44.2 AUTH) with ESMTPSA id k02759u94Atd7ed (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (curve secp521r1 with 521 ECDH bits, eq. 15360 bits RSA)) (Client did not present a certificate) for ; Thu, 4 Oct 2018 12:55:39 +0200 (CEST) To: freebsd-fs@freebsd.org From: Felix Winterhalter Subject: NFSv4 Kerberos mount from Linux Message-ID: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> Date: Thu, 4 Oct 2018 12:55:39 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: de-DE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 10:55:42 -0000 Hello everyone, I've been trying to get a kerberized nfsv4 mount to work from a Debian Stretch client to a FreeBSD 11.2 server. My export file looks like: V4: / -sec=krb5p clients /testexport -maproot=root -sec=krb5p clients I am now trying to mount this directory as root first without having to deal with user keytabs or tickets. This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p. This does not however work with nfsv4 and krb5p or any other krb5 flavor. The only errors we have been able to get is an error by gssd: gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227 Searching for this error has lead us to an old entry in the mailing list: https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023132.html Which apparently has this problem unresolved with extremely similar symptoms. Mounting from the Linux client to a similar Linux server under the same KDC with nfsv4 krb5p works without any problem. Also access to the FreeBSD server with sshd and GSSAPI works fine. So the keytab for the FreeBSD host seems to work fine. This is extremely frustrating as I have been at this problem for days now without any real way to even debug the issue. Any help would be greatly appreciated. From owner-freebsd-fs@freebsd.org Thu Oct 4 12:48:03 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 022E110A8E2B for ; Thu, 4 Oct 2018 12:48:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7E2E094A56 for ; Thu, 4 Oct 2018 12:48:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4020510A8E29; Thu, 4 Oct 2018 12:48:02 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F03310A8E28 for ; Thu, 4 Oct 2018 12:48:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C65E094A54 for ; Thu, 4 Oct 2018 12:48:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 19F47138FE for ; Thu, 4 Oct 2018 12:48:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w94Cm0VL060163 for ; Thu, 4 Oct 2018 12:48:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w94Cm04G060159 for fs@FreeBSD.org; Thu, 4 Oct 2018 12:48:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231251] [ZFS assfail] 12-ALPHA5 r338529 panic Date: Thu, 04 Oct 2018 12:48:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: gldisater@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 12:48:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231251 --- Comment #10 from Jeremy Faulkner --- Core dump will be uploaded in about an hour. This panic happened over night while scrubbing and synchronizing siad, and of course daily scripts running. panic: solaris assert: HDR_EMPTY(hdr), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line: 6213 cpuid =3D 1 time =3D 1538641234 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0398ce0= 770 vpanic() at vpanic+0x1a3/frame 0xfffffe0398ce07d0 panic() at panic+0x43/frame 0xfffffe0398ce0830 assfail() at assfail+0x1a/frame 0xfffffe0398ce0840 arc_release() at arc_release+0x940/frame 0xfffffe0398ce08d0 dbuf_redirty() at dbuf_redirty+0x56/frame 0xfffffe0398ce08f0 dmu_buf_will_dirty() at dmu_buf_will_dirty+0x120/frame 0xfffffe0398ce0920 dmu_write_uio_dnode() at dmu_write_uio_dnode+0x129/frame 0xfffffe0398ce09a0 zvol_write() at zvol_write+0x151/frame 0xfffffe0398ce0a00 ctl_be_block_dispatch_zvol() at ctl_be_block_dispatch_zvol+0x228/frame 0xfffffe0398ce0a80 ctl_be_block_worker() at ctl_be_block_worker+0x6c/frame 0xfffffe0398ce0b20 taskqueue_run_locked() at taskqueue_run_locked+0x10c/frame 0xfffffe0398ce0b= 80 taskqueue_thread_loop() at taskqueue_thread_loop+0x88/frame 0xfffffe0398ce0= bb0 fork_exit() at fork_exit+0x84/frame 0xfffffe0398ce0bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0398ce0bf0 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- KDB: enter: panic __curthread () at ./machine/pcpu.h:230 230 __asm("movq %%gs:%1,%0" : "=3Dr" (td) (kgdb) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 4 13:37:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C23DB10AAD59 for ; Thu, 4 Oct 2018 13:37:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4ECE897053 for ; Thu, 4 Oct 2018 13:37:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1389210AAD56; Thu, 4 Oct 2018 13:37:28 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 025D110AAD55 for ; Thu, 4 Oct 2018 13:37:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 913659704F for ; Thu, 4 Oct 2018 13:37:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id CBB6A1407F for ; Thu, 4 Oct 2018 13:37:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w94DbQ3j035771 for ; Thu, 4 Oct 2018 13:37:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w94DbQVW035770 for fs@FreeBSD.org; Thu, 4 Oct 2018 13:37:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231251] [ZFS assfail] 12-ALPHA5 r338529 panic Date: Thu, 04 Oct 2018 13:37:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: gldisater@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 13:37:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231251 --- Comment #11 from Jeremy Faulkner --- tar zcvf HDR-core-dump.tar.gz /boot/kernel /boot/kernel.old /var/crash/*.2 https://drive.google.com/open?id=3D13JI8vxg161Iw_UD8b5Nl-PiZ970CknCT --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 4 14:39:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1964E10ACCB7 for ; Thu, 4 Oct 2018 14:39:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A7AAB717BD for ; Thu, 4 Oct 2018 14:39:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 674D910ACCB5; Thu, 4 Oct 2018 14:39:38 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5581510ACCB4 for ; Thu, 4 Oct 2018 14:39:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8C48717B7 for ; Thu, 4 Oct 2018 14:39:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4EAC4148D8 for ; Thu, 4 Oct 2018 14:39:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w94EdbFH081787 for ; Thu, 4 Oct 2018 14:39:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w94Edbhl081786 for fs@FreeBSD.org; Thu, 4 Oct 2018 14:39:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 231794] Panic in ZFS due to arc related memory exhaustion Date: Thu, 04 Oct 2018 14:39:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 14:39:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231794 --- Comment #2 from Andriy Gapon --- (In reply to Dave Robison from comment #0) I just want to note that the problem here is not with exhausting the physic= al memory, but rather with exhausting the kernel virtual address space (KVA). There could be many reasons for that such as incorrect tuning, bugs, KVA fragmentation, etc. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Thu Oct 4 15:21:41 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAC4B10AEB1E for ; Thu, 4 Oct 2018 15:21:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660055.outbound.protection.outlook.com [40.107.66.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 57B6D73F4A for ; Thu, 4 Oct 2018 15:21:41 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM (52.132.44.160) by YTOPR0101MB0906.CANPRD01.PROD.OUTLOOK.COM (52.132.43.153) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1207.18; Thu, 4 Oct 2018 15:21:39 +0000 Received: from YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM ([fe80::65af:417a:161f:f4eb]) by YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM ([fe80::65af:417a:161f:f4eb%3]) with mapi id 15.20.1207.021; Thu, 4 Oct 2018 15:21:39 +0000 From: Rick Macklem To: Felix Winterhalter , "freebsd-fs@freebsd.org" Subject: Re: NFSv4 Kerberos mount from Linux Thread-Topic: NFSv4 Kerberos mount from Linux Thread-Index: AQHUW9DS+OLl3kMYEUaTZ7pIOOpJR6UPKibn Date: Thu, 4 Oct 2018 15:21:39 +0000 Message-ID: References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> In-Reply-To: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB0906; 6:D7epXlhxvfCjTJs2Nx47NXeHjaiEcLvOa9IKLzEFfCJVxmWQniWLBkZ93kdJY6J47L6ZKTorJChMgUwsowTxITR+Z++ul2zDkYrlWwsnD5/2hNtqVxFgs5Iv5BRPy3rVwHJ+JsF+HP7pPVxt1HImPH62O+TcMiu7XQ8Booqx1StjwB2s2IjyYu0IxFMgoiDmaGxJQ3chEZ9eZxwR2+xT30DFPCOSgkYuAMl50aCZCfUY0kVibqUr1VMGvzv0n6sQ4r7PspsbCUWpc7zG1oYISoJEo/zZSSzM91/pqitGFX/qeCbgnhuW3SlFbw/KwjIDncRF/aVut3NZkJfHX/w+eSKBNBsa6O8Hccgcd3CQA9c/m30b/ympabN5CD6ISZamXMRnoC3/DVDzPHZgq9T2PFNDZQtIDtDfhQ8ama6hi4jCTHun7SRRTxZ4zZg0PXStwXmqYHxfbM1TE0LUDzg6vg==; 5:fVtZB5SuWOdppkZzkeeEplb5J/suFm2+8ekg1tI4xs9RgTJIUSlj9/ygV5xH93UWmA3yx7+m+Xp/gg5Ptl3BlxmGfX4oTqw10kx/AnzTUZa0UsMEU1UrJXjWjaBD6qXzdvrnMCCPtLR6SQQmedLCsbowT3fA5wSCXGL5BkneswY=; 7:zRAu54W26wlBCRxXqeYGaxnet9Rcd61wknSYYVUjjEPk6mgbEgqgs0HaFbXiKjbMAEY1BJAaVI5TL15wtJfDx6Qf7l7qua/bUxyWZKGKyRP4crB2taXjvlAWIcZR/iTGUnC2ZF1X8ISaTPAnYGDEw1D7TFicw1g378yy9s19fcZzWXBjFXlZDAd9AXdVNQ4SFvbwTZGmV4W2bSZAHfLiMrG31QV6i8jyaIZKyKY03u9Nc/aK3JQG9kv5q/MnkFmB x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 4502f850-e104-4d0a-cfd6-08d62a0d150a x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB0906; x-ms-traffictypediagnostic: YTOPR0101MB0906: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(192374486261705)(75325880899374); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231355)(944501410)(52105095)(149066)(150057)(6041310)(20161123560045)(20161123564045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123562045)(201708071742011)(7699051); SRVR:YTOPR0101MB0906; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB0906; x-forefront-prvs: 0815F8251E x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(376002)(346002)(39860400002)(396003)(136003)(53754006)(189003)(199004)(5660300001)(7696005)(46003)(86362001)(8936002)(76176011)(102836004)(2906002)(99286004)(81166006)(305945005)(8676002)(25786009)(786003)(316002)(551544002)(186003)(256004)(74482002)(81156014)(105586002)(2900100001)(110136005)(9686003)(446003)(106356001)(6246003)(71190400001)(6306002)(53936002)(74316002)(71200400001)(11346002)(14444005)(33656002)(476003)(68736007)(97736004)(229853002)(55016002)(6436002)(2501003)(478600001)(6506007)(14454004)(486006)(5250100002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB0906; H:YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: N9Sk7EsC6vhvd41xxbb4K+M6wfiuoZwuFvIw6bQfNDXm0vjWHOwBjgH353y3KPm4KbeFIe5528nav4X7RhC0G0mQ+m1unp58XUK1aUY6ocJDxRfWZJGbBo2hBcNM2XfntlD6Sfjpch/Cr0APS3xM6qxrQib2nWcOZ29MPqXPMKYW3UHOZrSK2ZykmQ3HBqFYDQTa2nJVEsaRwuGqCJLZZJwBvcsP5U3mtA488TgXJgjeNy5ZG/P6OEjf4xqzpxCvAeTtAY8f6c7DPLz/2eckQHeyxPmdMr6fT9soLyOeOk+43SQJHEdxzx6AyxaUdDyyMkX3pOCazi9o4rTTK7YVqGLSvxXwKaQHyRDQCfrakQA= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 4502f850-e104-4d0a-cfd6-08d62a0d150a X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Oct 2018 15:21:39.4875 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB0906 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 15:21:42 -0000 Felix Winterhalter wrote: >Hello everyone, > >I've been trying to get a kerberized nfsv4 mount to work from a Debian >Stretch client to a FreeBSD 11.2 server. > >My export file looks like: > >V4: / -sec=3Dkrb5p clients > >/testexport -maproot=3Droot -sec=3Dkrb5p clients > Btw, if you only mounting "/testexport", you can specify the "V4:" as V4: /testexport -sec=3Dkrb5p clients and then the mount on the client uses "/" as the server mountpoint, like # mount -t nfs -o nfsvers=3D4 :/ /mnt (This avoids the server having to search for "testexport" in the "/" direct= ory during mounting and might avoid some problems when "/" isn't an exported file system. There are "hooks" in the FreeBSD server to make the search wo= rk, but I've never been 100% certain they will work for Kerberos and/or ZFS.) Btw, in case the Linux client is falling back on using AUTH_SYS at some poi= nt during the mount, you could try allowing both krb5 and auth_sys by setting "-sec=3Dsys,krb5,krb5i,krb5p" for both of the above lines. (I'd also sugges= t you try krb5 or krb5i until you get it working, since any packet traces are easier to decode, although once one krb5 variant works, they all should.) >I am now trying to mount this directory as root first without having to >deal with user keytabs or tickets. > >This works fine with -sec=3Dsys and nfsv4.1 and nfsv3 and -sec=3Dkrb5p. > This does not however work with nfsv4 and krb5p or any other krb5 flavor. Sorry, I'm not sure what you are saying here. Is it 1 - no version of NFS works for krb5p or 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or 3 - only nfsv3 works for krb5p If it #3, that is what I would expect. For NFSv4 (and NFSv4.1, I believe) t= o work a host based initiator credential is needed for the client host. The only w= ays I know of that you can get this is by - creating such an entry in your KDC and then putting a keytab entry for it= in the client or - on the FreeBSD client, doing the mount as a user after that user has done= a kinit. (There are mount options for this on FreeBSD and it also requires setting= the sysctl vfs.usermount to 1 so non-root can do mounts.) Since you are using a Li= nux client I have no idea how this might be done on Linux. I have no idea how the Linux server might allow an NFSv4 mount to work with= out Kerberos credentials for the "state maintaining operations" done by root (o= r the user doing the mount for the FreeBSD client)? - Maybe they allow these operations to be done via AUTH_SYS. To me, this wo= uld sound like a security hole, but I'm not a security guy... If you have #3, I know the FreeBSD server won't allow what you are trying t= o do. If you want to find out what the Linux server does to make it work, you cou= ld capture packets via tcpdump or similar and look at them in wireshark. (I'd suggest krb5 or krb5i for this, so that the packet data isn't encrypte= d, since it makes the wireshark decoding a lot more useful.) - If you get such a packet trace, you could email it to me and I can take a= look. (I am curious how a Linux server might make this work.) Most of the above only applies if you are experiencing #3, where NFSv3 work= s for krb5, but NFSv4 (and 4.1) does not. >The only errors we have been able to get is an error by gssd: > >gssd_pname_to_uid: failed major=3D0xd0000 minor=3D-1765328227 I can't remember what this means, but I think it is saying that the princip= al name didn't exist in the password database. (Maybe Linux has some "special" reserved principal name it uses for "state maintaining operations"?) >Searching for this error has lead us to an old entry in the mailing list: > >https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023132.html > >Which apparently has this problem unresolved with extremely similar >symptoms. > >Mounting from the Linux client to a similar Linux server under the same >KDC with nfsv4 krb5p works without any problem. > >Also access to the FreeBSD server with sshd and GSSAPI works fine. So >the keytab for the FreeBSD host seems to work fine. > >This is extremely frustrating as I have been at this problem for days >now without any real way to even debug the issue. Here's what I would do: 1 - Try an NFSv3 mount with krb5. (It wasn't obvious if you already have th= at working or not.) If that works, then... 2 - Try a mount from a FreeBSD client as a user, by doing # sysctl vfs.usermount=3D1 - login as non-root user and kinit - as this user, try a mount like % mount -t nfs -o nfsv4,sec=3Dkrb5 :/ /mnt 3 - If this works, then you probably need a hostbased client credential in = the keytab on the client. rick From owner-freebsd-fs@freebsd.org Thu Oct 4 15:32:54 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAD3E10AF1C6 for ; Thu, 4 Oct 2018 15:32:54 +0000 (UTC) (envelope-from Jean-Marc.Lacroix@unice.fr) Received: from ip-nice01.unice.fr (ip-nice01.unice.fr [134.59.1.214]) by mx1.freebsd.org (Postfix) with ESMTP id 1730E745E9 for ; Thu, 4 Oct 2018 15:32:53 +0000 (UTC) (envelope-from Jean-Marc.Lacroix@unice.fr) X-IronPort-AV: E=Sophos;i="5.54,340,1534802400"; d="scan'208";a="235003317" Received: from naxos2.unice.fr ([134.59.1.112]) by ip-nice08.unice.fr with ESMTP; 04 Oct 2018 17:32:46 +0200 Received: from [134.59.10.29] (txmath29.unice.fr [134.59.10.29]) (authenticated bits=0) by naxos2.unice.fr (8.14.7/8.14.7) with ESMTP id w94FWknx404920 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 4 Oct 2018 17:32:46 +0200 To: freebsd-fs@freebsd.org From: Jean-Marc LACROIX Subject: Problem with ZFSD when replacing a failed disk Message-ID: <96665eb8-9ff6-9050-bd7d-7f9f1d3fe737@unice.fr> Date: Thu, 4 Oct 2018 17:32:46 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: fr X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 15:32:55 -0000 Hello,    we encounter a problem on our storage solution which is constituted of 1 R630 + 3 JOBS MD1420 sas connected.(DELL)    Our system is : 11.0-RELEASE-p9    We use ZFSD to manage Spare disks The problem appear when we have to change a failed disk as explained below. We don't understand why the second spare is activated when the replace command is done; Thanks in advance for your help. Regards JM root@math12:/ # zpool status   pool: zpool  state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for     the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'.    see: http://illumos.org/msg/ZFS-8000-2Q   scan: resilvered 238G in 11h3m with 0 errors on Sat Sep 29 19:47:28 2018 config:     NAME                       STATE     READ WRITE CKSUM     zpool                      DEGRADED     0     0     0       raidz2-0                 ONLINE       0     0     0         label/e0s0             ONLINE       0     0     0         label/e1s0             ONLINE       0     0     0         label/e2s0             ONLINE       0     0     0         label/e0s1             ONLINE       0     0     0         label/e1s1             ONLINE       0     0     0         label/e2s1             ONLINE       0     0     0       raidz2-1                 ONLINE       0     0     0         label/e0s2             ONLINE       0     0     0         label/e1s2             ONLINE       0     0     0         label/e2s2             ONLINE       0     0     0         label/e0s3             ONLINE       0     0     0         label/e1s3             ONLINE       0     0     0         label/e2s3             ONLINE       0     0     0       raidz2-2                 ONLINE       0     0     0         label/e0s4             ONLINE       0     0     0         label/e1s4             ONLINE       0     0     0         label/e2s4             ONLINE       0     0     0         label/e0s5             ONLINE       0     0     0         label/e1s5             ONLINE       0     0     0         label/e2s5             ONLINE       0     0     0       raidz2-3                 ONLINE       0     0     0         label/e0s6             ONLINE       0     0     0         label/e1s6             ONLINE       0     0     0         label/e2s6             ONLINE       0     0     0         label/e0s7             ONLINE       0     0     0         label/e1s7             ONLINE       0     0     0         label/e2s7             ONLINE       0     0     0       raidz2-4                 ONLINE       0     0     0         label/e0s8             ONLINE       0     0     0         label/e1s8             ONLINE       0     0     0         label/e2s8             ONLINE       0     0     0         label/e0s9             ONLINE       0     0     0         label/e1s9             ONLINE       0     0     0         label/e2s9             ONLINE       0     0     0       raidz2-5                 DEGRADED     0     0     0         label/e0s10            ONLINE       0     0     0         label/e1s10            ONLINE       0     0     0         label/e2s10            ONLINE       0     0     0         label/e0s11            ONLINE       0     0     0         spare-4                UNAVAIL      0     0     0           9796387366129075446  UNAVAIL      0     0 0  was /dev/label/e1s11           label/spare0         ONLINE       0     0     0         label/e2s11            ONLINE       0     0     0       raidz2-6                 ONLINE       0     0     0         label/e0s12            ONLINE       0     0     0         label/e1s12            ONLINE       0     0     0         label/e2s12            ONLINE       0     0     0         label/e0s13            ONLINE       0     0     0         label/e1s13            ONLINE       0     0     0         label/e2s13            ONLINE       0     0     0       raidz2-7                 ONLINE       0     0     0         label/e0s14            ONLINE       0     0     0         label/e1s14            ONLINE       0     0     0         label/e2s14            ONLINE       0     0     0         label/e0s15            ONLINE       0     0     0         label/e1s15            ONLINE       0     0     0         label/e2s15            ONLINE       0     0     0       raidz2-8                 ONLINE       0     0     0         label/e0s16            ONLINE       0     0     0         label/e1s16            ONLINE       0     0     0         label/e2s16            ONLINE       0     0     0         label/e0s17            ONLINE       0     0     0         label/e1s17            ONLINE       0     0     0         label/e2s17            ONLINE       0     0     0     logs       mirror-9                 ONLINE       0     0     0         da2                    ONLINE       0     0     0         da3                    ONLINE       0     0     0     spares       12553822586401141982     INUSE     was /dev/label/spare0       label/spare1             AVAIL errors: No known data errors =========================================================================== STEPS DONE: 1) unplug the blinking UNAVAIL disk 2) plug the new disk in the free slot    => the system console shows the new device name: ugen0.4: at usbus0 umass0: on usbus0 umass0:  SCSI over Bulk-Only; quirks = 0x4100 umass0:16:0: Attached to scbus16 da59 at umass-sim0 bus 0 scbus16 target 0 lun 0 da59: Removable Direct Access SCSI device da59: 40.000MB/s transfers da59: 308MB (630784 512 byte sectors) da59: quirks=0x2 ugen0.4: at usbus0 (disconnected) umass0: at uhub4, port 2, addr 4 (disconnected) da59 at umass-sim0 bus 0 scbus16 target 0 lun 0 da59: detached (da59:umass-sim0:0:0:0): Periph destroyed (da59:mrsas1:1:61:0): UNMAPPED da59 at mrsas1 bus 1 scbus3 target 61 lun 0 da59: Fixed Direct Access SPC-4 SCSI device da59: Serial Number 9XG9RH37 da59: 150.000MB/s transfers da59: 953869MB (1953525168 512 byte sectors) glabel label e1s11 /dev/da59 glabel status da59 => the disk is correctly labeled zpool replace zpool 9796387366129075446 label/e1s11 zpool status => PROBLEM DESCRIBED BELOW ========================================================================== AFTER the replace command, we can see that the second hotspare has been activated as follow: root@math12:/ # zpool status   pool: zpool  state: DEGRADED status: One or more devices is currently being resilvered.  The pool will     continue to function, possibly in a degraded state. action: Wait for the resilver to complete.   scan: resilver in progress since Thu Oct  4 15:39:49 2018         214G scanned out of 12.6T at 191M/s, 18h49m to go         7.89G resilvered, 1.66% done config:     NAME                           STATE     READ WRITE CKSUM     zpool                          DEGRADED     0 0     0       raidz2-0                     ONLINE       0 0     0         label/e0s0                 ONLINE       0 0     0         label/e1s0                 ONLINE       0 0     0         label/e2s0                 ONLINE       0 0     0         label/e0s1                 ONLINE       0 0     0         label/e1s1                 ONLINE       0 0     0         label/e2s1                 ONLINE       0 0     0       raidz2-1                     ONLINE       0 0     0         label/e0s2                 ONLINE       0 0     0         label/e1s2                 ONLINE       0 0     0         label/e2s2                 ONLINE       0 0     0         label/e0s3                 ONLINE       0 0     0         label/e1s3                 ONLINE       0 0     0         label/e2s3                 ONLINE       0 0     0       raidz2-2                     ONLINE       0 0     0         label/e0s4                 ONLINE       0 0     0         label/e1s4                 ONLINE       0 0     0         label/e2s4                 ONLINE       0 0     0         label/e0s5                 ONLINE       0 0     0         label/e1s5                 ONLINE       0 0     0         label/e2s5                 ONLINE       0 0     0       raidz2-3                     ONLINE       0 0     0         label/e0s6                 ONLINE       0 0     0         label/e1s6                 ONLINE       0 0     0         label/e2s6                 ONLINE       0 0     0         label/e0s7                 ONLINE       0 0     0         label/e1s7                 ONLINE       0 0     0         label/e2s7                 ONLINE       0 0     0       raidz2-4                     ONLINE       0 0     0         label/e0s8                 ONLINE       0 0     0         label/e1s8                 ONLINE       0 0     0         label/e2s8                 ONLINE       0 0     0         label/e0s9                 ONLINE       0 0     0         label/e1s9                 ONLINE       0 0     0         label/e2s9                 ONLINE       0 0     0       raidz2-5                     DEGRADED     0 0     0         label/e0s10                ONLINE       0 0     0         label/e1s10                ONLINE       0 0     0         label/e2s10                ONLINE       0 0     0         label/e0s11                ONLINE       0 0     0         spare-4                    UNAVAIL      0 0     0           replacing-0              UNAVAIL      0 0     0             spare-0                UNAVAIL      0 0     0               9796387366129075446  UNAVAIL      0 0     0  was /dev/label/e1s11/old               label/spare1         ONLINE       0 0     0  (resilvering)             label/e1s11            ONLINE       0 0     0  (resilvering)           label/spare0             ONLINE       0 0     0         label/e2s11                ONLINE       0 0     0       raidz2-6                     ONLINE       0 0     0         label/e0s12                ONLINE       0 0     0         label/e1s12                ONLINE       0 0     0         label/e2s12                ONLINE       0 0     0         label/e0s13                ONLINE       0 0     0         label/e1s13                ONLINE       0 0     0         label/e2s13                ONLINE       0 0     0       raidz2-7                     ONLINE       0 0     0         label/e0s14                ONLINE       0 0     0         label/e1s14                ONLINE       0 0     0         label/e2s14                ONLINE       0 0     0         label/e0s15                ONLINE       0 0     0         label/e1s15                ONLINE       0 0     0         label/e2s15                ONLINE       0 0     0       raidz2-8                     ONLINE       0 0     0         label/e0s16                ONLINE       0 0     0         label/e1s16                ONLINE       0 0     0         label/e2s16                ONLINE       0 0     0         label/e0s17                ONLINE       0 0     0         label/e1s17                ONLINE       0 0     0         label/e2s17                ONLINE       0 0     0     logs       mirror-9                     ONLINE       0 0     0         da2                        ONLINE       0 0     0         da3                        ONLINE       0 0     0     spares       12553822586401141982         INUSE     was /dev/label/spare0       15637882846021217179         INUSE     was /dev/label/spare1 errors: No known data errors From owner-freebsd-fs@freebsd.org Thu Oct 4 18:43:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBE3F10B7121 for ; Thu, 4 Oct 2018 18:43:26 +0000 (UTC) (envelope-from wollman@khavrinen.csail.mit.edu) Received: from khavrinen.csail.mit.edu (khavrinen.csail.mit.edu [IPv6:2603:400a:0:7ec::801e:1c14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "khavrinen.csail.mit.edu", Issuer "Client CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8084E7F6E8 for ; Thu, 4 Oct 2018 18:43:26 +0000 (UTC) (envelope-from wollman@khavrinen.csail.mit.edu) Received: from khavrinen.csail.mit.edu (localhost [127.0.0.1]) by khavrinen.csail.mit.edu (8.15.2/8.15.2) with ESMTPS id w94IhPS2080217 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL CN=khavrinen.csail.mit.edu issuer=Client+20CA) for ; Thu, 4 Oct 2018 14:43:25 -0400 (EDT) (envelope-from wollman@khavrinen.csail.mit.edu) Received: (from wollman@localhost) by khavrinen.csail.mit.edu (8.15.2/8.15.2/Submit) id w94IhPIb080216; Thu, 4 Oct 2018 14:43:25 -0400 (EDT) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <23478.24397.495369.226706@khavrinen.csail.mit.edu> Date: Thu, 4 Oct 2018 14:43:25 -0400 From: Garrett Wollman To: freebsd-fs@freebsd.org Subject: ZFS/NVMe layout puzzle X-Mailer: VM 8.2.0b under 26.1 (amd64-portbld-freebsd11.2) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (khavrinen.csail.mit.edu [127.0.0.1]); Thu, 04 Oct 2018 14:43:25 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2018 18:43:27 -0000 Say you're using an all-NVMe zpool with PCIe switches to multiplex drives (e.g., 12 4-lane NVMe drives on one side, 1 PCIe x8 slot on the other). Does it make more sense to spread each vdev across switches (and thus CPU sockets) or to have all of the drives in a vdev on the same switch? I have no intuition about this at all, and it may not even matter. (You can be sure I'll be doing some benchmarking.) I'm assuming the ZFS code doesn't have any sort of CPU affinity that would allow it to take account of the PCIe topology even if that information were made available to it. -GAWollman From owner-freebsd-fs@freebsd.org Fri Oct 5 02:22:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCCA710C03EB for ; Fri, 5 Oct 2018 02:22:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670044.outbound.protection.outlook.com [40.107.67.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EB538DB65 for ; Fri, 5 Oct 2018 02:22:18 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM (52.132.44.160) by YTOPR0101MB0921.CANPRD01.PROD.OUTLOOK.COM (52.132.43.157) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1207.18; Fri, 5 Oct 2018 02:22:17 +0000 Received: from YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM ([fe80::65af:417a:161f:f4eb]) by YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM ([fe80::65af:417a:161f:f4eb%3]) with mapi id 15.20.1207.021; Fri, 5 Oct 2018 02:22:17 +0000 From: Rick Macklem To: Felix Winterhalter , "freebsd-fs@freebsd.org" Subject: Re: NFSv4 Kerberos mount from Linux Thread-Topic: NFSv4 Kerberos mount from Linux Thread-Index: AQHUW9DS+OLl3kMYEUaTZ7pIOOpJR6UPKibngADATF4= Date: Fri, 5 Oct 2018 02:22:17 +0000 Message-ID: References: <30f6446c-6fed-4b1e-9cae-9c417974ec46@audiofair.de>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YTOPR0101MB0921; 6:3NZLLIu11twd/tKR4ORUCZvkS1JTEhbC4tB8uU+ePkVSrFIUSrmtjU3ziEYMtP2kXpG+qrIFlPlqX10CTYXG9twZ1fJ/LMDmCgaBTo/FJPBAjXX050R8Xb/zMI7cI5dXay0u2kHymy7mAf0mUED0R2HwX3St0WzwzIQZMX+eZpoJ1BoNZN6OjUi4zCv5QbDbJiDa9Lp7HptgI38flbRAPvsfrnr2lPscXbIfej9MKm769diyTYTqwSlVSzb9+1SK8zz53Aa0CceLFg+vnlAKsWCP+Ss1CCZiD3Xi004IEnA87R2zhh6FKaZ8IBIGruH7L5Rx/WuQxxtWTH/aC41S6B5B5ulyQD+eYB0F1UTeuyHT8m0Xl2KfsFreoXy+wDo5j8xWmP2qa0BlMj/syzSfHoJdmdqWq5R9PMGbr+14/ta+Jlzln0XK3Ein1I10cwJfrujUD2MDXce/HYYgjBH6ew==; 5:7ATJ5Z0gkX46HFlnmVTuZhgZMB1MxtFLjRQ5r7ovqNzf5/2NDszwMqYB5IF88iTka0U0+eFW8w2MuvJoeaKIpf7XUW9Hip/9ELo3F0s/P7G3r5f+nT9VbFn+nU7Gu8QwjHwvxFn97XxqPMz6Bc4OGj76bfjtsZt1QjRkc8BTCa0=; 7:kmaO0JjcxBQLvNeCxXgJuVEwkKFIwY0NyXfUbBcUR4NMn5gdDeCn2PuTcXEOJV9w6ygSaQuGrgH9xoT8PzX9QhpkSIRnJ9MdWYEiNMSVI9CAJJxPyArGVRAAtHDyrMZVPrDGPmBm8KmOSeM9l+znEBbNRDOy07EIyq1L/wQ6AIw65MUG9Plf41KkLsGXKJDk6OqZN3tn3T59ZBsXXQ7DUW87SSCV8xrrC715uCrYX3xwI0DRdI8TYELk8GaSgQgQ x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 37216fc2-646f-48a1-b41b-08d62a695efc x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YTOPR0101MB0921; x-ms-traffictypediagnostic: YTOPR0101MB0921: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3002001)(3231355)(944501410)(52105095)(10201501046)(149066)(150057)(6041310)(20161123560045)(20161123564045)(201703131423095)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123558120)(201708071742011)(7699051)(76991048); SRVR:YTOPR0101MB0921; BCL:0; PCL:0; RULEID:; SRVR:YTOPR0101MB0921; x-forefront-prvs: 0816F1D86E x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(376002)(346002)(136003)(396003)(39860400002)(189003)(199004)(46003)(446003)(71200400001)(6246003)(9686003)(55016002)(74482002)(71190400001)(8676002)(8936002)(81166006)(6436002)(186003)(81156014)(2900100001)(11346002)(53936002)(106356001)(105586002)(110136005)(102836004)(229853002)(97736004)(5250100002)(5660300001)(6506007)(256004)(25786009)(99286004)(74316002)(86362001)(786003)(76176011)(316002)(2906002)(7696005)(33656002)(2501003)(305945005)(14454004)(68736007)(478600001)(476003)(486006); DIR:OUT; SFP:1101; SCL:1; SRVR:YTOPR0101MB0921; H:YTOPR0101MB1820.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: 8/90bx24ihlbkrjjQJ5bmzDfn219Z0wpUh5DetLleyFbDgRDv3XJSNQ1TdsgmI+b4gG8U2AVpsjKXt+r6Tfb+lDQd63v1woD12knZaYopMCUdJlgZhQTw0O5qxXoyjQYI4Im8rQD5kgipqdmsUK3hS6YNgOOIfjZg6ATEPn9K+9oeveDi8LWQ/EE7AJg0EOyWYNT2G+scBlShLNPAduM1ew53ID9N8nlI4Be6K5W05+jDkRqKxfl9CFnh1Ye6FzK3h1s4aCJt77ydQ2EXWLp+ZorrQJdl0XK/rm9pmlc69RqcK2Cq0JOSVnO543Js8wUDffRuPFH7Itv7wiz5nspV+qhNb7Isa0ZED0nvLmAmqk= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 37216fc2-646f-48a1-b41b-08d62a695efc X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Oct 2018 02:22:17.2545 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTOPR0101MB0921 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Oct 2018 02:22:19 -0000 I wrote: [stuff snipped] Btw, if you only mounting "/testexport", you can specify the "V4:" as "if you are only.." typo [more stuff snipped] >Btw, in case the Linux client is falling back on using AUTH_SYS at some po= int >during the mount, you could try allowing both krb5 and auth_sys by setting >"-sec=3Dsys,krb5,krb5i,krb5p" for both of the above lines. (I'd also sugge= st you Oops, the syntax is "-sec=3Dsys:krb5:krb5i:krb5p" (':'s and not ','s) And if you want to capture packets during a Linux mount attempt, you can run this on the FreeBSD server: # tcpdump -s 0 -w out.pcap host However you will want to look at out.pcap in wireshark, since it can decode= NFS. Good luck with it and please let us know if you learn more, rick From owner-freebsd-fs@freebsd.org Sat Oct 6 15:42:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B47810BD3BA for ; Sat, 6 Oct 2018 15:42:22 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from echo.brtsvcs.net (echo.brtsvcs.net [IPv6:2607:f740:c::4ae]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1073078547 for ; Sat, 6 Oct 2018 15:42:21 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from chombo.houseloki.net (unknown [IPv6:2601:1c2:1402:1771:6a05:caff:fe2d:8b38]) by echo.brtsvcs.net (Postfix) with ESMTPS id 8022A38D09; Sat, 6 Oct 2018 08:42:13 -0700 (PDT) Received: from [IPv6:fe80::dd2a:ba9e:2d4a:7c5f] (unknown [IPv6:fe80::dd2a:ba9e:2d4a:7c5f]) by chombo.houseloki.net (Postfix) with ESMTPSA id 5C1D3B3CF; Sat, 6 Oct 2018 08:42:12 -0700 (PDT) Subject: Re: ZFS/NVMe layout puzzle To: Garrett Wollman , freebsd-fs@freebsd.org References: <23478.24397.495369.226706@khavrinen.csail.mit.edu> From: Mel Pilgrim Message-ID: <775752d2-1e66-7db9-5a4f-7cd775e366a6@bluerosetech.com> Date: Sat, 6 Oct 2018 08:42:10 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <23478.24397.495369.226706@khavrinen.csail.mit.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 15:42:22 -0000 On 2018-10-04 11:43, Garrett Wollman wrote: > Say you're using an all-NVMe zpool with PCIe switches to multiplex > drives (e.g., 12 4-lane NVMe drives on one side, 1 PCIe x8 slot on the > other). Does it make more sense to spread each vdev across switches > (and thus CPU sockets) or to have all of the drives in a vdev on the > same switch? I have no intuition about this at all, and it may not > even matter. (You can be sure I'll be doing some benchmarking.) > > I'm assuming the ZFS code doesn't have any sort of CPU affinity that > would allow it to take account of the PCIe topology even if that > information were made available to it. In this scenario, the PCIe switch takes the role of an HBA in terms of fault vulnerability. From owner-freebsd-fs@freebsd.org Sat Oct 6 21:00:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3EFC310C64A1 for ; Sat, 6 Oct 2018 21:00:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CFC9383651 for ; Sat, 6 Oct 2018 21:00:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 93ED610C64A0; Sat, 6 Oct 2018 21:00:38 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8274F10C649F for ; Sat, 6 Oct 2018 21:00:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 235A18364E for ; Sat, 6 Oct 2018 21:00:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 70E981164F for ; Sat, 6 Oct 2018 21:00:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96L0bxv061825 for ; Sat, 6 Oct 2018 21:00:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96L0bY2061824 for fs@FreeBSD.org; Sat, 6 Oct 2018 21:00:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 208130] smbfs is slow because it (apparently) doesn't do any caching/buffering Date: Sat, 06 Oct 2018 21:00:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: noah.bergbauer@tum.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 21:00:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208130 noah.bergbauer@tum.de changed: What |Removed |Added ---------------------------------------------------------------------------- Version|10.2-RELEASE |11.1-RELEASE --- Comment #1 from noah.bergbauer@tum.de --- Revisiting this 2.5 years later, no improvement. But one possible workaround is to mount the smbfs, then create an md(4) disk from a file on the mount and finally mount UFS on top of that. The filesystem-level buffer management ensures that inefficiently small IO will never hit the network (increase the md disk sector size as needed, 4K alrea= dy gives me well over 40 MiB/s). This obviously breaks file sharing but at least it allows using samba shares for simple remote storage (poor man's iSCSI) with decent performance. It's a shame there is no filesystem-level equivalent of gcache(8) (which by= the way is the solution if you just need a block device instead of a filesystem= ). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Oct 6 21:08:38 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C92BA10C6A26 for ; Sat, 6 Oct 2018 21:08:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5B9B083D1C for ; Sat, 6 Oct 2018 21:08:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1FEBA10C6A25; Sat, 6 Oct 2018 21:08:38 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0EA5D10C6A24 for ; Sat, 6 Oct 2018 21:08:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4E7783D1A for ; Sat, 6 Oct 2018 21:08:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id EB88B116BA for ; Sat, 6 Oct 2018 21:08:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96L8a19007533 for ; Sat, 6 Oct 2018 21:08:36 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96L8aoT007525 for fs@FreeBSD.org; Sat, 6 Oct 2018 21:08:36 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 208130] smbfs is slow because it (apparently) doesn't do any caching/buffering Date: Sat, 06 Oct 2018 21:08:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc version Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 21:08:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208130 Conrad Meyer changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |cem@freebsd.org Version|11.1-RELEASE |CURRENT --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Oct 6 21:10:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A44910C6ACC for ; Sat, 6 Oct 2018 21:10:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id EC4CB83DDC for ; Sat, 6 Oct 2018 21:10:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B0C4C10C6ACB; Sat, 6 Oct 2018 21:10:06 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F83110C6ACA for ; Sat, 6 Oct 2018 21:10:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4138883DDA for ; Sat, 6 Oct 2018 21:10:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 75A1C116C3 for ; Sat, 6 Oct 2018 21:10:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96LA5dS048285 for ; Sat, 6 Oct 2018 21:10:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96LA5YO048280 for fs@FreeBSD.org; Sat, 6 Oct 2018 21:10:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 208130] smbfs is slow because it (apparently) doesn't do any caching/buffering Date: Sat, 06 Oct 2018 21:10:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 21:10:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208130 --- Comment #2 from Conrad Meyer --- As I understand it, our smbfs also does not support newer SMB protocol vers= ions (2 and 3) =E2=80=94 and SMB1 has known security problems, so newer versions= of Windows (and Samba, by default) do not speak the v1 protocol at all. It might be b= est to use something other than smbfs. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Oct 6 21:42:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FAA810C7B59 for ; Sat, 6 Oct 2018 21:42:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id ED72F85BC2 for ; Sat, 6 Oct 2018 21:42:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id AF6CD10C7B58; Sat, 6 Oct 2018 21:42:19 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9D72710C7B56 for ; Sat, 6 Oct 2018 21:42:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BE1C85BC0 for ; Sat, 6 Oct 2018 21:42:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 8E48711B60 for ; Sat, 6 Oct 2018 21:42:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96LgI0f064644 for ; Sat, 6 Oct 2018 21:42:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96LgIQN064643 for fs@FreeBSD.org; Sat, 6 Oct 2018 21:42:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 208130] smbfs is slow because it (apparently) doesn't do any caching/buffering Date: Sat, 06 Oct 2018 21:42:18 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: noah.bergbauer@tum.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 21:42:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208130 --- Comment #3 from noah.bergbauer@tum.de --- (In reply to Conrad Meyer from comment #2) >SMB1 has known security problems, [...] It might be best to use something = other than smbfs. True that. But what else is there? At least in this particular case I'm for= ced to choose between Samba, FTP, SFTP and WebDAV. This is all I'm given. The kernel has no SSHFS (judging by how the protocol works, performance probably wouldn't be too great either) and last I checked davfs was awfully slow. Le= t's not even talk about FTP. This leaves me with smbfs as my only choice. Fun fact: The cloud storage provider in question was (as far as I remember) actually running FreeBSD on their storage servers as of ~2 years ago and probably still is today. The handful of times I tried to play around with NFS (a few years ago) I got disappointing performance even on loopback/tap links (bhyve VM), especially considering how complicated it is to work with. Right now I'm just not confident that I could properly secure an NFS server. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Oct 6 21:59:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A916B10C7F56 for ; Sat, 6 Oct 2018 21:59:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 45AA6860DC for ; Sat, 6 Oct 2018 21:59:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0A4CF10C7F55; Sat, 6 Oct 2018 21:59:46 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED14310C7F54 for ; Sat, 6 Oct 2018 21:59:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8BFB4860D7 for ; Sat, 6 Oct 2018 21:59:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D242911DE6 for ; Sat, 6 Oct 2018 21:59:44 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96Lxi94092367 for ; Sat, 6 Oct 2018 21:59:44 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96Lxi2j092366 for fs@FreeBSD.org; Sat, 6 Oct 2018 21:59:44 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 208130] smbfs is slow because it (apparently) doesn't do any caching/buffering Date: Sat, 06 Oct 2018 21:59:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cem@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 21:59:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208130 --- Comment #4 from Conrad Meyer --- (In reply to noah.bergbauer from comment #3) Yes, all of these seem like bad options unfortunately. There is a FUSE SMB which may support newer versions of SMB and have block caching. Even with additional round-trips between kernel and user, it may be faster than sendi= ng every single 1kB request out to the network. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Sat Oct 6 23:37:44 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E956710C9AEC for ; Sat, 6 Oct 2018 23:37:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 71BBC8932D for ; Sat, 6 Oct 2018 23:37:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 356DE10C9AEB; Sat, 6 Oct 2018 23:37:43 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE7ED10C9AEA for ; Sat, 6 Oct 2018 23:37:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8ED9489329 for ; Sat, 6 Oct 2018 23:37:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id DCC1612BBE for ; Sat, 6 Oct 2018 23:37:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w96Nbfts037245 for ; Sat, 6 Oct 2018 23:37:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w96NbfSo037244 for fs@FreeBSD.org; Sat, 6 Oct 2018 23:37:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Sat, 06 Oct 2018 23:37:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: allanjude@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Oct 2018 23:37:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 --- Comment #20 from Allan Jude --- in my recreation: 55649 100482 python3.6 - mi_switch+0xe6 sleepq_wait+0x2c _sx_xlock_hard+0x306 zil_commit_impl+0x11d zfs_freebsd_putpages+0x635 VOP_PUTPAGES_APV+0x82 vnode_pager_putpages+0x8e vm_pageout_flush+0xea vm_object_page_collect_flush+0x213 vm_object_page_clean+0x146 vm_object_terminate+0x93 zfs_freebsd_reclaim+0x1e VOP_RECLAIM_APV+0x82 vgonel+0x208 vrecycle+0x4a zfs_freebsd_inactive+0xd VOP_INACTIVE_APV+0x82 vinactive+0xfc the sync process does seem to be making doing work, but not making progress: sampled 10000 times: NUM PID TID COMM TDNAME KSTACK=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20 1 56618 100457 sync - mi_switch critical_exit intr_event_handle intr_execute_handlers lapic_handle_intr Xapic_isr1 dbuf_read dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 1 56618 100457 sync - mi_switch critical_exit ipi_bitmap_handler Xipi_intr_bitmap_handler VOP_ISLOCKED_APV zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 1 56618 100457 sync - mi_switch critical_exit ipi_bitmap_handler Xipi_intr_bitmap_handler zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 23 56618 100457 sync - VOP_ISLOCKED_APV zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 112 56618 100457 sync - arc_access arc_buf_access dbuf_hold_impl dbuf_hold dnode_hold_impl dmu_bonus_hold zfs_= zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 118 56618 100457 sync - dnode_rele dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 148 56618 100457 sync - vref zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 150 56618 100457 sync - dbuf_find dbuf_hold_impl dbuf_hold dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_da= ta zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 248 56618 100457 sync - dbuf_hold dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sy= nc sys_sync amd64_syscall fast_syscall_common 263 56618 100457 sync - vputx zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 273 56618 100457 sync -=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20 dmu_object_info_from_dnode dmu_object_info_from_db zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 292 56618 100457 sync - zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 428 56618 100457 sync - =20=20= =20=20=20=20=20=20=20=20=20=20=20 458 56618 100457 sync - arc_buf_access dbuf_hold_impl dbuf_hold dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_da= ta zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 572 56618 100457 sync -=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20 dmu_object_info_from_db zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_= sync amd64_syscall fast_syscall_common 671 56618 100457 sync - dbuf_read dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 690 56618 100457 sync - dbuf_read dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sy= nc sys_sync amd64_syscall fast_syscall_common 937 56618 100457 sync - dbuf_hold_impl dbuf_hold dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_data zil_commit_i= mpl zfs_sync sys_sync amd64_syscall fast_syscall_common 1272 56618 100457 sync - dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 1433 56618 100457 sync - dnode_hold_impl dmu_bonus_hold zfs_zget zfs_get_data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common 1909 56618 100457 sync - zfs_zget zfs_get_= data zil_commit_impl zfs_sync sys_sync amd64_syscall fast_syscall_common --=20 You are receiving this mail because: You are the assignee for the bug.=