From owner-freebsd-fs@freebsd.org Sun Dec 16 07:28:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E02133FDFA for ; Sun, 16 Dec 2018 07:28:02 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BE967449C for ; Sun, 16 Dec 2018 07:28:01 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm1-x342.google.com with SMTP id g67so9537338wmd.2 for ; Sat, 15 Dec 2018 23:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=zqwdm2drma+jOzB0s5zTDzO8vvW1lXG7LwplmwKZ+iQ=; b=Xyg879AmvIrQrImLpc/2qE4a5KSJtcwPR6/ug94XbUCcNgThMc4ISPDr1Vo0POfTzh 2P1Xo0ODUJiG44WHEDb46/9mabbI8LpYJ2qoKzzVSx7jsC4Tpvyn7FOELIQzel7oufXD KY9zE0iuimoiO8yvVCmLROBpBt2RI3L0vYeIfOhwCWKKoB5cAmPvA/43jIGsUhlccQBP qmP3im3oLa8nBxwZyRG9D4jPaIvSv64RGh8wtEnyVJG+jdNDQXKRPQQ0sUXP8BjnsDuf FufV8vgfmlu84k9yX4ax2iRhfmI9MI+F3N/AmYFb2Ax+rDZ+oTlmy1I7NOFtYVUBNHTA iF2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=zqwdm2drma+jOzB0s5zTDzO8vvW1lXG7LwplmwKZ+iQ=; b=pnrON7W3sjP+OFWeDU4PQ74IHUSnjaBEY87rzUZveULujGOVByNx6QlZO29SAA7teZ OrryfBQJFcjfJZCyqQLMi2cjXT22BjZyV3bzwP9HAutGNSiIdp6D3cqNPprkdrBfchFi x5sk8VZnJVnVQ4MnHAkG++H+EP1N2okmZTtckMzwPB2bBrk8QYOiv9u7A+fhWmkubBEy kkBZMtrYSHDAYlmvi16yGbffNg2OTQgKKaYN9iTFDQFqFoFXlTd91Hfy2XN1qf5VSdoq FkPNIg+d+IV/22K7S7qMG3t+hhM57o3tfD446XSQf/yxKaszDjr0E4u54oArwdeRD49J qFyA== X-Gm-Message-State: AA+aEWbzHBEtaD0WG5yojfQ1m2m4vzfskCPafquzipWHabWQ+skIfNk0 3o//GwlryFp9p1VWh9VO7AK3fbNm X-Google-Smtp-Source: AFSGD/XUt4yabjVZH7ee0r97dyVY+dtYYNPZ5XZg4irMTdDVvfADf+9hPeJUD3x5jwhksb2nAQ0fiA== X-Received: by 2002:a1c:2e0c:: with SMTP id u12mr7980336wmu.81.1544945279585; Sat, 15 Dec 2018 23:27:59 -0800 (PST) Received: from Sting-Ray.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id o82sm9540966wmo.29.2018.12.15.23.27.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 15 Dec 2018 23:27:58 -0800 (PST) To: freebsd-fs@freebsd.org From: Kaya Saman Subject: Upgrade to 12-Release system not booting kernel panic Message-ID: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> Date: Sun, 16 Dec 2018 07:27:58 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US-large X-Rspamd-Queue-Id: 5BE967449C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Xyg879Am; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kayasaman@gmail.com designates 2a00:1450:4864:20::342 as permitted sender) smtp.mailfrom=kayasaman@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.76)[-0.762,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.983,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.24)[ip: (1.73), ipnet: 2a00:1450::/32(-1.52), asn: 15169(-1.32), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Dec 2018 07:28:02 -0000 Hi, I've been digging around a little on this one and it seems that there is some ZFS wierdness going on. My original install was an 11.2-RELEASE fresh install and upon an upgrade to 12-RELEASE the system is having issues booting. I have a ZFS root pool called zroot; by default this mounts at zroot/ROOT/default and a few other non-root / bootable ZFS pools. The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. I have the boot disks ada0 and ada1 plugged into the rear of the chassis and directly into the systemboard which is also a SuperMicro. These drives are both Samsung SSD's. The 22 drive slots at the front of the chassis are occupied by the other various data pools. So here is some strangeness... if I remove all 22 drives from the front, the system boots fine but straight after boot goes into kernel panic mode and reboots before I can even look at the error or get to the login prompt. With the non-root pools installed at the BTX loader after scanning through all the bios drives I get a bunch of: read 264 from ... to 0x...., error 0x10 errors then: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool after this the system simply hangs? I have tried looking around but everything mentioning the MOS error is talking about the root pool, a particularly good reference is here: http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html In fact I did try to boot with a USB stick and go into Live mode then import all the pools on the system. This works without any issue! The pools are fine the data is there everything looks normal. - I also rebuilt the zpool.cache according to the link just incase there was some kind of corruption there, however upon reboot I still get the same issue?? Looking at a bug report with a kernel panic: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 I have attempted to add: kern.cam.scsi_delay="50000" kern.cam.boot_delay="50000" into the /boot/loader.conf file but unfortunately the issue still continues :-( I wonder if there is a way to tell to tell the system to only look at certain drives for booting?? There is this line in my loader.conf: vfs.root.mountfrom="zfs:zroot" It maybe the wrong hunch I have but it seems like the system is looking for "zroot" on all pools instead of the actual root pool hence the above errors?? Would anyone be able to suggest anything or have any ideas about how to get the system back online and booting?? Thanks. Kaya From owner-freebsd-fs@freebsd.org Sun Dec 16 20:41:42 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54087133ABB7 for ; Sun, 16 Dec 2018 20:41:42 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (static-50-125-237-106.rdmd.wa.frontiernet.net [50.125.237.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DD18A9625A for ; Sun, 16 Dec 2018 20:41:29 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from localhost (localhost [127.0.0.1]) by btw.pki2.com (8.15.2/8.15.2) with ESMTP id wBGKEQqd090383; Sun, 16 Dec 2018 12:14:26 -0800 (PST) (envelope-from freebsd@pki2.com) Message-ID: <16e381cea5ea7294dbdef9e771af8b5c20174f6d.camel@pki2.com> Subject: Re: Upgrade to 12-Release system not booting kernel panic From: Dennis Glatting To: Kaya Saman , freebsd-fs@freebsd.org Date: Sun, 16 Dec 2018 12:14:26 -0800 In-Reply-To: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> References: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Please contact the ISP for more information X-yoursite-MailScanner-ID: wBGKEQqd090383 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: freebsd@pki2.com X-Spam-Status: No X-Rspamd-Queue-Id: DD18A9625A X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=pki2.com (policy=none) X-Spamd-Result: default: False [2.42 / 15.00]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[pki2.com : No valid SPF, No valid DKIM,none]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_SPAM_SHORT(0.31)[0.309,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.77)[0.770,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[btw.pki2.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.87)[0.868,0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5650, ipnet:50.120.0.0/13, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.02)[asn: 5650(-0.03), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Dec 2018 20:41:42 -0000 I had the exact problem with RC2. I reinstalled 11.2 and called it a day. On Sun, 2018-12-16 at 07:27 +0000, Kaya Saman wrote: > Hi, > > > I've been digging around a little on this one and it seems that there > is > some ZFS wierdness going on. > > > My original install was an 11.2-RELEASE fresh install and upon an > upgrade to 12-RELEASE the system is having issues booting. > > > I have a ZFS root pool called zroot; by default this mounts at > zroot/ROOT/default and a few other non-root / bootable ZFS pools. > > > The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. > > > I have the boot disks ada0 and ada1 plugged into the rear of the > chassis > and directly into the systemboard which is also a SuperMicro. These > drives are both Samsung SSD's. > > The 22 drive slots at the front of the chassis are occupied by the > other > various data pools. > > > So here is some strangeness... if I remove all 22 drives from the > front, > the system boots fine but straight after boot goes into kernel panic > mode and reboots before I can even look at the error or get to the > login > prompt. > > > With the non-root pools installed at the BTX loader after scanning > through all the bios drives I get a bunch of: > > read 264 from ... to 0x...., error 0x10 errors > > then: > > ZFS: i/o error - all block copies unavailable > > ZFS: can't read MOS of pool > > > after this the system simply hangs? > > > I have tried looking around but everything mentioning the MOS error > is > talking about the root pool, a particularly good reference is here: > http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html > > > In fact I did try to boot with a USB stick and go into Live mode then > import all the pools on the system. This works without any issue! The > pools are fine the data is there everything looks normal. > > - I also rebuilt the zpool.cache according to the link just incase > there > was some kind of corruption there, however upon reboot I still get > the > same issue?? > > > Looking at a bug report with a kernel panic: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 > > > I have attempted to add: > > > kern.cam.scsi_delay="50000" > > kern.cam.boot_delay="50000" > > > into the /boot/loader.conf file but unfortunately the issue still > continues :-( > > > I wonder if there is a way to tell to tell the system to only look at > certain drives for booting?? > > > There is this line in my loader.conf: > > vfs.root.mountfrom="zfs:zroot" > > > It maybe the wrong hunch I have but it seems like the system is > looking > for "zroot" on all pools instead of the actual root pool hence the > above > errors?? > > > Would anyone be able to suggest anything or have any ideas about how > to > get the system back online and booting?? > > > Thanks. > > > Kaya > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Dec 17 00:01:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 11CF9134283C for ; Mon, 17 Dec 2018 00:01:48 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 952426F570 for ; Mon, 17 Dec 2018 00:01:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x832.google.com with SMTP id l11so12340215qtp.0 for ; Sun, 16 Dec 2018 16:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=EOZ3BgywR2+9GYZEdne81YJwuTZkq7qZjGtEdNd1gcQ=; b=ApTELMXcutBuSNpRKJaSHVDopcHQ9lYoybsIsUdw0RB/r9OVQaEdqNz2Mr2Bc767Z7 vq784b177CbIIXfH14NRfmp/9LfLwDqvaz7r2dif8anQ26idv2Ldc0VCqpvcs2Z+CI8l JaGEE3TJzUdKNrCYr1SLYbFBdsfolCH44KfKMrzJM6astDAF8NEnRM6wdWroThIjtxgX w4S6E7z9XLQ4N1Hw0lW9Sk0e6MzMrZ5laVuZc7eWk43S8YUbAU0TTMyF76d/gX9W3SPD iBBxF4cy/Rp0qCHJdkgDpYgvMLBzkl0TzZO6meBcik4cRwYr/zWIAoWjASYhw3voGuWf 4neg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=EOZ3BgywR2+9GYZEdne81YJwuTZkq7qZjGtEdNd1gcQ=; b=jrfuJsC6M2mq5VqJhlUwGXesZ+8Kxa3QyBX8Ya12bWqPMqfkfWtml+kEK8lThFL3DG sQdwUiCsafCsRBDDd2ArNuLYjy/WmYyFL8M6PZS3JvMWOG5RdNZu2XIb06i8svjTtPwx IuILFeMCNs51+klEadLrC/NUFk2Vxq+nOZ+IkvnImqgcFSK03t2XMEMt+mCs9BxjSGBX cWRIzQF01Gwv/CUXaiLsamOM6Xpb3QzukwJXwBVoYiKb77HfsWr8qWBpYqjCzUo2TU2g 5KkEBRhuA/mzrl/F3g8GfTJx8ulNbq6CiVvzXdLiRibwY9OBpUfOigTgX/d5z8NY90Od 7eSQ== X-Gm-Message-State: AA+aEWa7H8l9ecWgYhMWbm5EHesSVqcN2WNoCmbz5tvbPlRQ8uMJa151 R3VUE0rYMTdTnjmats7Nj4RLDhFBEAOWi3e2IoeYFldT X-Google-Smtp-Source: AFSGD/WMzgiW2QUD0bLtes5qNxoLtDtCjKV+G5U/7srAx4PCdrklDcJ7DTMEwBKqEZvHz7MN2/TJwEPBXhYzvXKnJ9o= X-Received: by 2002:a0c:f143:: with SMTP id y3mr11366432qvl.21.1545004906006; Sun, 16 Dec 2018 16:01:46 -0800 (PST) MIME-Version: 1.0 References: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> In-Reply-To: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> From: Warner Losh Date: Sun, 16 Dec 2018 17:01:34 -0700 Message-ID: Subject: Re: Upgrade to 12-Release system not booting kernel panic To: Kaya Saman Cc: FreeBSD FS X-Rspamd-Queue-Id: 952426F570 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=ApTELMXc X-Spamd-Result: default: False [-4.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.84)[-0.838,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[ALT1.aspmx.l.google.com,aspmx.l.google.com,ALT2.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-1.99)[ip: (-6.92), ipnet: 2607:f8b0::/32(-1.63), asn: 15169(-1.33), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 00:01:48 -0000 Did you get a FreeBSD copyright notice before the crash? Warner On Sun, Dec 16, 2018, 8:45 AM Kaya Saman Hi, > > > I've been digging around a little on this one and it seems that there is > some ZFS wierdness going on. > > > My original install was an 11.2-RELEASE fresh install and upon an > upgrade to 12-RELEASE the system is having issues booting. > > > I have a ZFS root pool called zroot; by default this mounts at > zroot/ROOT/default and a few other non-root / bootable ZFS pools. > > > The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. > > > I have the boot disks ada0 and ada1 plugged into the rear of the chassis > and directly into the systemboard which is also a SuperMicro. These > drives are both Samsung SSD's. > > The 22 drive slots at the front of the chassis are occupied by the other > various data pools. > > > So here is some strangeness... if I remove all 22 drives from the front, > the system boots fine but straight after boot goes into kernel panic > mode and reboots before I can even look at the error or get to the login > prompt. > > > With the non-root pools installed at the BTX loader after scanning > through all the bios drives I get a bunch of: > > read 264 from ... to 0x...., error 0x10 errors > > then: > > ZFS: i/o error - all block copies unavailable > > ZFS: can't read MOS of pool > > > after this the system simply hangs? > > > I have tried looking around but everything mentioning the MOS error is > talking about the root pool, a particularly good reference is here: > > http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html > > > In fact I did try to boot with a USB stick and go into Live mode then > import all the pools on the system. This works without any issue! The > pools are fine the data is there everything looks normal. > > - I also rebuilt the zpool.cache according to the link just incase there > was some kind of corruption there, however upon reboot I still get the > same issue?? > > > Looking at a bug report with a kernel panic: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 > > > I have attempted to add: > > > kern.cam.scsi_delay="50000" > > kern.cam.boot_delay="50000" > > > into the /boot/loader.conf file but unfortunately the issue still > continues :-( > > > I wonder if there is a way to tell to tell the system to only look at > certain drives for booting?? > > > There is this line in my loader.conf: > > vfs.root.mountfrom="zfs:zroot" > > > It maybe the wrong hunch I have but it seems like the system is looking > for "zroot" on all pools instead of the actual root pool hence the above > errors?? > > > Would anyone be able to suggest anything or have any ideas about how to > get the system back online and booting?? > > > Thanks. > > > Kaya > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sun Dec 16 21:00:15 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0B6EE133B5BC for ; Sun, 16 Dec 2018 21:00:15 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9A7D196C8C for ; Sun, 16 Dec 2018 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5F1B9133B5BA; Sun, 16 Dec 2018 21:00:14 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39D94133B5B8 for ; Sun, 16 Dec 2018 21:00:14 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AEDA896C89 for ; Sun, 16 Dec 2018 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 12BD71EB24 for ; Sun, 16 Dec 2018 21:00:13 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBGL0CSU015650 for ; Sun, 16 Dec 2018 21:00:12 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBGL0C0h015644 for fs@FreeBSD.org; Sun, 16 Dec 2018 21:00:12 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201812162100.wBGL0C0h015644@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 16 Dec 2018 21:00:12 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Dec 2018 21:00:15 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Dec 17 00:32:16 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE0471343B74 for ; Mon, 17 Dec 2018 00:32:15 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (static-50-125-237-106.rdmd.wa.frontiernet.net [50.125.237.106]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0422706E4 for ; Mon, 17 Dec 2018 00:32:12 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from localhost (localhost [127.0.0.1]) by btw.pki2.com (8.15.2/8.15.2) with ESMTP id wBH0VkTM090943; Sun, 16 Dec 2018 16:31:46 -0800 (PST) (envelope-from freebsd@pki2.com) Message-ID: <2cb7b5531a41b676dae2c5730e2dc5641d57e320.camel@pki2.com> Subject: Re: Upgrade to 12-Release system not booting kernel panic From: Dennis Glatting To: Warner Losh , Kaya Saman Cc: FreeBSD FS Date: Sun, 16 Dec 2018 16:31:46 -0800 In-Reply-To: References: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Please contact the ISP for more information X-yoursite-MailScanner-ID: wBH0VkTM090943 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: freebsd@pki2.com X-Spam-Status: No X-Rspamd-Queue-Id: C0422706E4 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dmarc=fail reason="" header.from=pki2.com (policy=none) X-Spamd-Result: default: False [2.46 / 15.00]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[pki2.com : No valid SPF, No valid DKIM,none]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; NEURAL_SPAM_SHORT(0.42)[0.419,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.75)[0.748,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: btw.pki2.com]; NEURAL_SPAM_LONG(0.82)[0.823,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5650, ipnet:50.120.0.0/13, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.02)[asn: 5650(-0.02), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 00:32:16 -0000 On Sun, 2018-12-16 at 17:01 -0700, Warner Losh wrote: > Did you get a FreeBSD copyright notice before the crash? > Sorry. I deleted the pictures. I think it was stage2. If I pulled the ZFS disks, it would boot. When I first installed 12.0 (BETA?) there wasn't a problem but I updated and poof. Sorry but I deleted the information. > Warner > > On Sun, Dec 16, 2018, 8:45 AM Kaya Saman > > Hi, > > > > > > I've been digging around a little on this one and it seems that > > there is > > some ZFS wierdness going on. > > > > > > My original install was an 11.2-RELEASE fresh install and upon an > > upgrade to 12-RELEASE the system is having issues booting. > > > > > > I have a ZFS root pool called zroot; by default this mounts at > > zroot/ROOT/default and a few other non-root / bootable ZFS pools. > > > > > > The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. > > > > > > I have the boot disks ada0 and ada1 plugged into the rear of the > > chassis > > and directly into the systemboard which is also a SuperMicro. These > > drives are both Samsung SSD's. > > > > The 22 drive slots at the front of the chassis are occupied by the > > other > > various data pools. > > > > > > So here is some strangeness... if I remove all 22 drives from the > > front, > > the system boots fine but straight after boot goes into kernel panic > > mode and reboots before I can even look at the error or get to the > > login > > prompt. > > > > > > With the non-root pools installed at the BTX loader after scanning > > through all the bios drives I get a bunch of: > > > > read 264 from ... to 0x...., error 0x10 errors > > > > then: > > > > ZFS: i/o error - all block copies unavailable > > > > ZFS: can't read MOS of pool > > > > > > after this the system simply hangs? > > > > > > I have tried looking around but everything mentioning the MOS error > > is > > talking about the root pool, a particularly good reference is here: > > > > http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html > > > > > > In fact I did try to boot with a USB stick and go into Live mode > > then > > import all the pools on the system. This works without any issue! > > The > > pools are fine the data is there everything looks normal. > > > > - I also rebuilt the zpool.cache according to the link just incase > > there > > was some kind of corruption there, however upon reboot I still get > > the > > same issue?? > > > > > > Looking at a bug report with a kernel panic: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 > > > > > > I have attempted to add: > > > > > > kern.cam.scsi_delay="50000" > > > > kern.cam.boot_delay="50000" > > > > > > into the /boot/loader.conf file but unfortunately the issue still > > continues :-( > > > > > > I wonder if there is a way to tell to tell the system to only look > > at > > certain drives for booting?? > > > > > > There is this line in my loader.conf: > > > > vfs.root.mountfrom="zfs:zroot" > > > > > > It maybe the wrong hunch I have but it seems like the system is > > looking > > for "zroot" on all pools instead of the actual root pool hence the > > above > > errors?? > > > > > > Would anyone be able to suggest anything or have any ideas about how > > to > > get the system back online and booting?? > > > > > > Thanks. > > > > > > Kaya > > > > _______________________________________________ > > freebsd-fs@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org > > " > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Dec 17 00:36:57 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 621EA1343EC5 for ; Mon, 17 Dec 2018 00:36:57 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5CC5C70B06 for ; Mon, 17 Dec 2018 00:36:56 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wr1-x435.google.com with SMTP id c14so10582492wrr.0 for ; Sun, 16 Dec 2018 16:36:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=IcHuru1KHM2W1bAxb0PnGSZxnffnVswf30cQrvvtrys=; b=Aa6gQqzbSLQNobdpolJqfVTqMY7HzH5xHYRR0NK3X720tUUHYOyp5023rzFqEKag1a ztAvM88gcJsDinz25Fy2Sl3ZHpw+NG4+N1hhMvd80h/LZitQ32Fk865ed5KR5hQL+Wu7 uN8PPuojRS2OEnK+4rHMNzGJQRq+dbVpI2tx4JE9PCV4jfBdkMPaRJCDSWlRWoFx8BWM KEmvuDvDpYnHVG5O7EbS/2CXMiPvy/w65L1mP0KdkDUTgeOPurREY7trC8Bc38/srjBj r7r18EyNAPjBu5c3GDuDJ9xqMi26cV/a8YRDIudYCoRLtrqtIgtQBGJOoPDnanmpX2kz CKJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=IcHuru1KHM2W1bAxb0PnGSZxnffnVswf30cQrvvtrys=; b=TU4mwCqod2LODMuGy5HCjT8YiM+5ygvvg7YW1WO9/V8IgEbZQNzLzNgFDTAiZvYCQb vugXTzY9ctxCPJXIy6sXcTcwul7QCclZ0nTS3tRxf26PaGsbShzbkU5OI+9stgRRfm+S iUROa70ugZa+gIhPkvdnyvzcyxd6/HjQQiDgh5zFLKVuYts7N/yJkJfLEpeogrzg9hZI NHmkN7t1cuCFNvkEGUPOxrbeMH9H+zgib8eWb2cNfcu2AciVMuOxW6qj+NocGhjo3ClP u8FtAP6esrG9SBboBcVREt7BEMk+E2m2zyu33B3Sos55RBQcvmhGGpeU4CIwL9wfLL/e 6TNQ== X-Gm-Message-State: AA+aEWYnhAed8awwvnN5lQKfvL9Me95G2EpBUsjex8xy7826nhl4YjMT JmLsYccEreCG7SMLK4j8X3DD47LH X-Google-Smtp-Source: AFSGD/XdOj5hkXFcCLQ0T1zSIKLvwdOA8kAhv0NHM5pe+xB+xHsds3vilD3JNngFIohCx98ou9G2zg== X-Received: by 2002:adf:de91:: with SMTP id w17mr9520854wrl.320.1545007014751; Sun, 16 Dec 2018 16:36:54 -0800 (PST) Received: from Sting-Ray.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id q9sm11585555wrp.0.2018.12.16.16.36.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 16 Dec 2018 16:36:54 -0800 (PST) Subject: Re: Upgrade to 12-Release system not booting kernel panic To: "freebsd-fs@freebsd.org >> freebsd-fs" References: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> From: Kaya Saman Message-ID: Date: Mon, 17 Dec 2018 00:36:53 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US-large X-Rspamd-Queue-Id: 5CC5C70B06 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Aa6gQqzb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kayasaman@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) smtp.mailfrom=kayasaman@gmail.com X-Spamd-Result: default: False [-4.31 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.71)[-0.708,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.59)[ipnet: 2a00:1450::/32(-1.54), asn: 15169(-1.33), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[5.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 00:36:57 -0000 On 12/17/18 12:01 AM, Warner Losh wrote: > Did you get a FreeBSD copyright notice before the crash? I don't recall as the screen went so fast. One thing I can say is that I managed to get the system up! :-) <- though I don't think it is stable :-( Basically it seems the kernel panic was from a setting(s) in /boot/loader.conf Using the installer img usb image in Live mode I commented out most of the modifications in there. I then proceeded to remove all my mechanical drives and left only the ssd's, then import the zroot and create a new cache file. After that the system managed to boot and I was able to continue with the upgrade. To get the pools back I installed the drives while operational (hot swap) and then did a zpool import on them. ....So far the system is up and functioning though after rebuilding @ports I need to reboot again to finish up the upgrade. This is something I am not too confident about doing as I do not wish to remove all my drives again then re-import the pools. Regards, Kaya > > Warner > > On Sun, Dec 16, 2018, 8:45 AM Kaya Saman wrote: > > Hi, > > > I've been digging around a little on this one and it seems that > there is > some ZFS wierdness going on. > > > My original install was an 11.2-RELEASE fresh install and upon an > upgrade to 12-RELEASE the system is having issues booting. > > > I have a ZFS root pool called zroot; by default this mounts at > zroot/ROOT/default and a few other non-root / bootable ZFS pools. > > > The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. > > > I have the boot disks ada0 and ada1 plugged into the rear of the > chassis > and directly into the systemboard which is also a SuperMicro. These > drives are both Samsung SSD's. > > The 22 drive slots at the front of the chassis are occupied by the > other > various data pools. > > > So here is some strangeness... if I remove all 22 drives from the > front, > the system boots fine but straight after boot goes into kernel panic > mode and reboots before I can even look at the error or get to the > login > prompt. > > > With the non-root pools installed at the BTX loader after scanning > through all the bios drives I get a bunch of: > > read 264 from ... to 0x...., error 0x10 errors > > then: > > ZFS: i/o error - all block copies unavailable > > ZFS: can't read MOS of pool > > > after this the system simply hangs? > > > I have tried looking around but everything mentioning the MOS > error is > talking about the root pool, a particularly good reference is here: > http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html > > > In fact I did try to boot with a USB stick and go into Live mode then > import all the pools on the system. This works without any issue! The > pools are fine the data is there everything looks normal. > > - I also rebuilt the zpool.cache according to the link just incase > there > was some kind of corruption there, however upon reboot I still get > the > same issue?? > > > Looking at a bug report with a kernel panic: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 > > > I have attempted to add: > > > kern.cam.scsi_delay="50000" > > kern.cam.boot_delay="50000" > > > into the /boot/loader.conf file but unfortunately the issue still > continues :-( > > > I wonder if there is a way to tell to tell the system to only look at > certain drives for booting?? > > > There is this line in my loader.conf: > > vfs.root.mountfrom="zfs:zroot" > > > It maybe the wrong hunch I have but it seems like the system is > looking > for "zroot" on all pools instead of the actual root pool hence the > above > errors?? > > > Would anyone be able to suggest anything or have any ideas about > how to > get the system back online and booting?? > > > Thanks. > > > Kaya > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to > "freebsd-fs-unsubscribe@freebsd.org > " > From owner-freebsd-fs@freebsd.org Mon Dec 17 08:44:44 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAF701335A1B for ; Mon, 17 Dec 2018 08:44:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A1EE88A256; Mon, 17 Dec 2018 08:44:43 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.5] (unknown [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPA id 005709DC8C7; Mon, 17 Dec 2018 09:44:32 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Dell R710 with PERC H310. GPT issues and hardware raid or ZFS? From: Borja Marcos In-Reply-To: <1372904516.1427.4.camel@localhost> Date: Mon, 17 Dec 2018 09:44:32 +0100 Cc: Steven Hartland , freebsd-fs@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <51C5B4EE.3010601@growveg.net> <81EDE2A7D67241DAB985C38662DA57DD@multiplay.co.uk> <51C5EAB0.7040009@growveg.net> <20130622191619.GA73246@icarus.home.lan> <12FB82C142E74924B27F4DBB642D0D1F@multiplay.co.uk> <1372904516.1427.4.camel@localhost> To: sbruno@freebsd.org X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: A1EE88A256 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of borjam@sarenet.es designates 195.16.151.151 as permitted sender) smtp.mailfrom=borjam@sarenet.es X-Spamd-Result: default: False [1.63 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:195.16.150.0/23]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sarenet.es]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_MEDIUM(0.31)[0.310,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[smtp.sarenet.es,smtp.sarenet.es,smtp.sarenet.es]; NEURAL_SPAM_LONG(0.49)[0.490,0]; RCVD_IN_DNSWL_NONE(0.00)[151.151.16.195.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.38)[-0.376,0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:3262, ipnet:195.16.128.0/19, country:ES]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.01)[country: ES(0.06)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 08:44:44 -0000 > On 4 Jul 2013, at 04:21, Sean Bruno wrote: >=20 >> Some mfi controllers actually have a JBOD option, if it does you'll >> be able to this controller if not watch out as using the RAID 0 = option >> will almost certainly cause you pain. >>=20 >> mps as Jeremy has said is better option. >>=20 >> Regards >> Steve >=20 > I note that the H310 in a test R420 that I have does indeed have a = JBOD > option. This presents a disk as /dev/mfisyspdX which is totally > befuddling. In my opinion the best option is not to create any kind of volumes (a = JBOD =E2=80=9Cmfisyspd=E2=80=9D is still equivalent to creating a =E2=80=9CRAID 0=E2=80=9D volume per = disk) and rely on the hw.mfi.allow_cam_disk_passthrough tunable instead. The "mfisyspd=E2=80=9D pseudo devices are not CAM devices.=20 At least in my case it has worked perfectly for many years. Before that = tunable was available I made a change suggested by Scott Long.=20 Borja. From owner-freebsd-fs@freebsd.org Wed Dec 19 01:56:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1277C1346FA4 for ; Wed, 19 Dec 2018 01:56:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BFAD792761 for ; Wed, 19 Dec 2018 01:56:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 7BA3A1346F9F; Wed, 19 Dec 2018 01:56:21 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68ED11346F9D for ; Wed, 19 Dec 2018 01:56:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE0DF92754 for ; Wed, 19 Dec 2018 01:56:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 12E6E140DC for ; Wed, 19 Dec 2018 01:56:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBJ1uGrW038148 for ; Wed, 19 Dec 2018 01:56:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBJ1uGv6038147 for fs@FreeBSD.org; Wed, 19 Dec 2018 01:56:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 195485] [ufs] mksnap_ffs(8) cannot create snapshot with journaled soft updates enabled Date: Wed, 19 Dec 2018 01:56:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: t_uemura@macome.co.jp X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 01:56:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195485 --- Comment #3 from t_uemura@macome.co.jp --- Sorry for my delay, and thank you so much for detailed explanation of the issue. As you mentioned, the reason I want to use the snapshot feature on UFS is to get live dumps. So the snapshots are usually removed once they are rsync'ed successfully, and personally, they can be removed and/or truncated at occasional filesystem recovery. Anyway, I'll use UFS without snapshot as I can't offer $25k. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 19 06:50:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F09A9134D7E7; Wed, 19 Dec 2018 06:50:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26BD36C4C1; Wed, 19 Dec 2018 06:50:01 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it1-f175.google.com (mail-it1-f175.google.com [209.85.166.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id DC0F41F864; Wed, 19 Dec 2018 06:50:00 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it1-f175.google.com with SMTP id h65so7847523ith.3; Tue, 18 Dec 2018 22:50:00 -0800 (PST) X-Gm-Message-State: AA+aEWb9d9IJj5VPhyCpu4gJv/qMURmy9sNQJmadVV4IFjPqhGBJ/SB2 WfqLPus3J55rEC2CStDMkJG3iOEB1qdH6cGmYY4= X-Google-Smtp-Source: AFSGD/VOhsA4adaigEJ11jv/JUmSBn/tNwxMk2hQNju4DIc4op2NBvwTnRjfPmXSfHlM+ag0Zks4PD1VAxdcP+PLWIA= X-Received: by 2002:a02:94eb:: with SMTP id x98mr18169446jah.88.1545202200039; Tue, 18 Dec 2018 22:50:00 -0800 (PST) MIME-Version: 1.0 From: Matthew Macy Date: Tue, 18 Dec 2018 22:49:48 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: The future of ZFS in FreeBSD To: freebsd-fs , freebsd-current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 26BD36C4C1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; ASN(0.00)[asn:11403, ipnet:96.47.64.0/20, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 06:50:02 -0000 The sources for FreeBSD's ZFS support are currently taken directly from Illumos with local ifdefs to support the peculiarities of FreeBSD where the Solaris Portability Layer (SPL) shims fall short. FreeBSD has regularly pulled changes from Illumos and tried to push back any bug fixes and new features done in the context of FreeBSD. In the past few years the vast majority of new development in ZFS has taken place in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced that they will be moving to ZoL https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means that there will be little to no net new development of Illumos. While working through the git history of ZoL I have also discovered that many races and locking bugs have been fixed in ZoL and never made it back to Illumos and thus FreeBSD. This state of affairs has led to a general agreement among the stakeholders that I have spoken to that it makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf has graciously encouraged me to add FreeBSD support directly to ZoL https://github.com/zfsonfreebsd/ZoF so that we might all have a single shared code base. A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port Before it can be committed some additional functionality needs to be added to the FreeBSD opencrypto framework. These can be found at https://reviews.freebsd.org/D18520 This port will provide FreeBSD users with multi modifier protection, project quotas, encrypted datasets, allocation classes, vectorized raidz, vectorized checksums, and various command line improvements. Before ZoF can be merged back in to ZoL several steps need to be taken: - Integrate FreeBSD support into ZoL CI - Have most of the ZFS test suite passing - Complete additional QA testing at iX We at iX Systems need to port ZoL's EC2 CI scripts to work with FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. Being integrated in to their CI will mean that, among other things, most integration issues will be caught before a PR is merged upstream rather than many months later when it is MFVed into FreeBSD. I=E2=80=99m hoping to submit the PR to ZoL some time in January. This port will make it easy for end users on a range of releases to run the latest version of ZFS. Nonetheless, transitioning away from an Illumos based ZFS is not likely to be entirely seamless. The stakeholders I=E2=80=99ve spoken to all agree that this is the best path forward but some degree of effort needs to be made to accommodate downstream consumers. The current plan is to import ZoF and unhook the older Illumos based sources from the build on April 15th or two months after iX systems QA deems ZoF stable - which ever comes later. The Illumos based sources will be removed some time later - but well before 13. This will give users a 3 month period during which both the port and legacy Illumos based ZFS will be available to users. Pools should interoperate between ZoF and legacy provided the user does not enable any features available only in ZoF. We will try to accommodate any downstream consumers in the event that they need that date pushed back. We ask that any downstream consumers who are particularly sensitive to changes start testing the port when it is formally announced and report back any issues they have. I will do my best to ensure that this message is communicated to all those who it may concern. However, I can=E2=80=99t ensure that everyone reads these lists. T= hat is the responsibility of -CURRENT users. -M From owner-freebsd-fs@freebsd.org Wed Dec 19 07:49:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CD10134EFB9; Wed, 19 Dec 2018 07:49:18 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CFB56E3F5; Wed, 19 Dec 2018 07:49:17 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pf1-x430.google.com with SMTP id c123so9468163pfb.0; Tue, 18 Dec 2018 23:49:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=YxZATs9eih0qlUdDiJN17xxyi6ockrN7+QperSHg7Es=; b=A77ywLzRjUFednjbLZ5Tx90LahEcwkADQtZh2JF6jEffSC3ChK+c1RcsF+hqiDHX7A anvtW00OPKmcz3gVAlVDiZtnfCVlvcmKdP0l7bhi53c5h9k2fvyxOJJfUaUNC3Nzhati kv8jwJWHmNbTOgGsZWuSMk7gBzG4LlL4oKtGrX8rM/UmbtmzYlMfKCvTZcNJSjXAbE/F IPWMuzyUHrKh5E0kolwbmkWkFy5JiWZ9WeN3GLwg5ypD9GTyFL+4K0mN7DcL/AeOocsc saAkiW5HrPaQMTXWeDdTEbwYHKl4kJWjFAzWx8MOakpok0qwuGW64JjfLnEYI0QWSbaG a0uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=YxZATs9eih0qlUdDiJN17xxyi6ockrN7+QperSHg7Es=; b=oUXcnnVMDjPAP8inkQcROPGTkBYT5QO37TwBh6UFdBnh0umr+N2GzQiUGdx+287eRW C6mT6L8tp6K3CAmwwwvdmbeeIDRh1XW+g3toRE+qGXh+5/QcH6iHO120blDMC6bQ7U7H KEkAvrtc9V4kHrWuq0Dk9sppMCN+R6Duc8hIb8FFNmpwinezhFACRfJWIrGVHwAqLBSO 0sqlgyG+AxAJfEJToep8LKbOsBGhu7EIGKn7Ge8YSs9j+kUW1Zlj1/M9CuaGrTZsF89b PuE9uN1w6kCaQAMm10MrzCNxzWt15lI0w7A6wYAlau5dGmT5g5JY2ed9gZC3pky9AUx+ y4YQ== X-Gm-Message-State: AA+aEWYLsUp81ovGis+F8JcE5sf598ckjVL1/UuTBqUN3OOh60Z0a+DS H78Js2ELuM4Ttu/oEKT4+Et9ATKw X-Google-Smtp-Source: AFSGD/UNeqrTFV84LIZ7kkpM5G4kt3vl+imi5naYsM94ELRl5MMmZHw5oYF1QNpFu15hqf5ECIKQSQ== X-Received: by 2002:a63:5402:: with SMTP id i2mr17787893pgb.79.1545205755990; Tue, 18 Dec 2018 23:49:15 -0800 (PST) Received: from [192.168.20.7] (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id m20sm18816651pgv.93.2018.12.18.23.49.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Dec 2018 23:49:15 -0800 (PST) From: Enji Cooper Message-Id: <9C23F0C0-DEF7-45A4-ADEF-58A00F9714D8@gmail.com> Content-Type: multipart/signed; boundary="Apple-Mail=_5B9C960B-5159-4A51-ACDD-81A425A795AA"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: The future of ZFS in FreeBSD Date: Tue, 18 Dec 2018 23:49:13 -0800 In-Reply-To: Cc: freebsd-fs , freebsd-current To: Matthew Macy References: X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 4CFB56E3F5 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=A77ywLzR; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of yaneurabeya@gmail.com designates 2607:f8b0:4864:20::430 as permitted sender) smtp.mailfrom=yaneurabeya@gmail.com X-Spamd-Result: default: False [-6.11 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; HAS_ATTACHMENT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_SHORT(-0.86)[-0.861,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; IP_SCORE(-0.64)[ipnet: 2607:f8b0::/32(-1.74), asn: 15169(-1.38), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[0.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 07:49:18 -0000 --Apple-Mail=_5B9C960B-5159-4A51-ACDD-81A425A795AA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Matthew, I appreciate the long write up, as someone who still uses FreeBSD ZFS on = my NAS box and knowing some of the history with ZFS on *Solaris, etc. = Something like this was bound to happen with post the Oracle buyout. > On Dec 18, 2018, at 10:49 PM, Matthew Macy wrote: >=20 > The sources for FreeBSD's ZFS support are currently taken directly > from Illumos with local ifdefs to support the peculiarities of FreeBSD > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD > has regularly pulled changes from Illumos and tried to push back any > bug fixes and new features done in the context of FreeBSD. In the past > few years the vast majority of new development in ZFS has taken place > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced > that they will be moving to ZoL > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means > that there will be little to no net new development of Illumos. While > working through the git history of ZoL I have also discovered that > many races and locking bugs have been fixed in ZoL and never made it > back to Illumos and thus FreeBSD. This state of affairs has led to a > general agreement among the stakeholders that I have spoken to that it > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf > has graciously encouraged me to add FreeBSD support directly to ZoL > https://github.com/zfsonfreebsd/ZoF so that we might all have a single > shared code base. >=20 > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port > Before it can be committed some additional functionality needs to be > added to the FreeBSD opencrypto framework. These can be found at > https://reviews.freebsd.org/D18520 >=20 > This port will provide FreeBSD users with multi modifier protection, > project quotas, encrypted datasets, allocation classes, vectorized > raidz, vectorized checksums, and various command line improvements. >=20 > Before ZoF can be merged back in to ZoL several steps need to be = taken: > - Integrate FreeBSD support into ZoL CI > - Have most of the ZFS test suite passing > - Complete additional QA testing at iX Can you please describe the testing process that will be employed to = verify the sanity of the ZoL on FreeBSD port? Should other large = companies who use ZFS on FreeBSD (Panzura?) chime in and the ZFS on = FreeBSD community (as a whole) collaborate to better suss out issues = with the ZoL port? > We at iX Systems need to port ZoL's EC2 CI scripts to work with > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. > Being integrated in to their CI will mean that, among other things, > most integration issues will be caught before a PR is merged upstream > rather than many months later when it is MFVed into FreeBSD. I=E2=80=99m= > hoping to submit the PR to ZoL some time in January. Could you please better describe this, in particular provide pointers to = any scripts that might be executed with the ZoL port? > This port will make it easy for end users on a range of releases to > run the latest version of ZFS. Nonetheless, transitioning away from an > Illumos based ZFS is not likely to be entirely seamless. The > stakeholders I=E2=80=99ve spoken to all agree that this is the best = path > forward but some degree of effort needs to be made to accommodate > downstream consumers. The current plan is to import ZoF and unhook the > older Illumos based sources from the build on April 15th or two months > after iX systems QA deems ZoF stable - which ever comes later. The > Illumos based sources will be removed some time later - but well > before 13. This will give users a 3 month period during which both the > port and legacy Illumos based ZFS will be available to users. Pools > should interoperate between ZoF and legacy provided the user does not > enable any features available only in ZoF. We will try to accommodate > any downstream consumers in the event that they need that date pushed > back. We ask that any downstream consumers who are particularly > sensitive to changes start testing the port when it is formally > announced and report back any issues they have. I will do my best to > ensure that this message is communicated to all those who it may > concern. However, I can=E2=80=99t ensure that everyone reads these = lists. That > is the responsibility of -CURRENT users. As a thought: is there a way for the ZoL port to run in parallel with = the non-ZoL port, kind of like in an audit/read-only/verify-only mode? Thanks so much, -Enji --Apple-Mail=_5B9C960B-5159-4A51-ACDD-81A425A795AA Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE5bk3FaGcY5rvqmb79YOpJmkwhhUFAlwZ9/kACgkQ9YOpJmkw hhXeBA//aK5RivUGqkKcORtDQ+P+l/0jXGs0lE42C7Yo89QO3QXnfM2po7PL/ZFV GaZGc8J+LIsiM2T48lx1u9S8qVbFRhdSbOnQKRFgm4ouQVJJUKiSCFLdugP2ADG4 q/EmRaO6tPo5Dc1XeyONahLymH/izPv0E97ua+HHTG2LSpe3uG0Wq4HMVGvqFklS b5Zy1HfNmBD8MxklHe6WDaCAtyns5l/zrNrIH8J3p9votYJ8zjJy3FMcXpu0BqSy 6eohgGykDtG/rL3BjgKaIi1JtSjZ/LBOJtkwM+owagBpnFvBoUHiem7dWxYuzosc 9NoFppsPflnAS3OC+2xwLg9ZtPcsOlDg88y9iChRBZYG/Lit6lBYxbjYC7E8GgFO ceqstQTgjkRTAe3+C4tGD0thI9DcjtUX5a3zoeu0glH8KcNDLAPy+ovqY8UaY4CK k18rVpk/j2pI7lmfPBZssuJop0idcJ6dUW+bicN1XsJZgdrzAIQQpqdrwjON35Ho bmseHWTQbhlit0+I+/HYchM1j1oAwdizuNF2MmMJGWrb/fiDqWp6n55kA1zmnTHV F7/v0qS4zqeyxq0zJrk177wb/ZhA/GcFpi+wAJ8+a4vEKFzkXgeYnmik9BucOplB ecjdqN1CsFR5E+PiVTUSk9dtoP7pUBVqvJjLC+EuYn212AO9syQ= =ypW9 -----END PGP SIGNATURE----- --Apple-Mail=_5B9C960B-5159-4A51-ACDD-81A425A795AA-- From owner-freebsd-fs@freebsd.org Wed Dec 19 08:35:28 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 952CD1350202; Wed, 19 Dec 2018 08:35:28 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3D0176FB69; Wed, 19 Dec 2018 08:35:28 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it1-f174.google.com (mail-it1-f174.google.com [209.85.166.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 140E369B; Wed, 19 Dec 2018 08:35:28 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it1-f174.google.com with SMTP id g85so8459395ita.3; Wed, 19 Dec 2018 00:35:28 -0800 (PST) X-Gm-Message-State: AA+aEWaWw6echKikPcvFd6zEKLVqI+4J/Obb6EGc+Oe6kWcH1tKPEDHL N/jDA68UtfKyZkeq2he6FwsiZmiYK6Z+IIZLTxU= X-Google-Smtp-Source: AFSGD/Xx9pI73/QS6uetP/Df1kR6ujk+9O9i6ahs72BMZ0oHhSvZJIcM2EGmvJ4/VbNDWMWkl/VDiYUXNoFHupWLbfY= X-Received: by 2002:a02:8904:: with SMTP id o4mr18360276jaj.35.1545208527463; Wed, 19 Dec 2018 00:35:27 -0800 (PST) MIME-Version: 1.0 References: <9C23F0C0-DEF7-45A4-ADEF-58A00F9714D8@gmail.com> In-Reply-To: <9C23F0C0-DEF7-45A4-ADEF-58A00F9714D8@gmail.com> From: Matthew Macy Date: Wed, 19 Dec 2018 00:35:15 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Enji Cooper Cc: freebsd-fs , freebsd-current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 3D0176FB69 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.99 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.99)[-0.992,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 08:35:28 -0000 On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper wrote: > > Hello Matthew, > > I appreciate the long write up, as someone who still uses FreeBSD ZFS on = my NAS box and knowing some of the history with ZFS on *Solaris, etc. Somet= hing like this was bound to happen with post the Oracle buyout. > > > On Dec 18, 2018, at 10:49 PM, Matthew Macy wrote: > > > > The sources for FreeBSD's ZFS support are currently taken directly > > from Illumos with local ifdefs to support the peculiarities of FreeBSD > > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD > > has regularly pulled changes from Illumos and tried to push back any > > bug fixes and new features done in the context of FreeBSD. In the past > > few years the vast majority of new development in ZFS has taken place > > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced > > that they will be moving to ZoL > > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means > > that there will be little to no net new development of Illumos. While > > working through the git history of ZoL I have also discovered that > > many races and locking bugs have been fixed in ZoL and never made it > > back to Illumos and thus FreeBSD. This state of affairs has led to a > > general agreement among the stakeholders that I have spoken to that it > > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf > > has graciously encouraged me to add FreeBSD support directly to ZoL > > https://github.com/zfsonfreebsd/ZoF so that we might all have a single > > shared code base. > > > > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port > > Before it can be committed some additional functionality needs to be > > added to the FreeBSD opencrypto framework. These can be found at > > https://reviews.freebsd.org/D18520 > > > > This port will provide FreeBSD users with multi modifier protection, > > project quotas, encrypted datasets, allocation classes, vectorized > > raidz, vectorized checksums, and various command line improvements. > > > > Before ZoF can be merged back in to ZoL several steps need to be taken: > > - Integrate FreeBSD support into ZoL CI > > - Have most of the ZFS test suite passing > > - Complete additional QA testing at iX > > Can you please describe the testing process that will be employed to veri= fy the sanity of the ZoL on FreeBSD port? Should other large companies who = use ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as= a whole) collaborate to better suss out issues with the ZoL port? The ZFS test suite itself provides ~80% coverage https://codecov.io/gh/zfsonlinux/zfs/branch/master - FreeBSD currently lacks equivalent gcov support, but presumably it would provide comparable coverage here. Andrew Turner has some form of kernel gcov support that he uses with syzkaller. However, I believe that it isn't sufficient for this purpose. ZoL requires that coverage only increase over time. If a change would decrease total coverage new tests have to be included with the change. There is a command called ztest which runs in user space that quickly covers a large percentage of the code base where it doesn't integrate directly with geom or VFS (disk access is emulated with files, kernel threads are emulated with pthreads, etc). ztest has been broken in HEAD for the past 2 years (it triggers an assertion failure), whereas it is not broken in ZoF. Joe Maloney is the head of QA at iX and can likely give you more detail on their approach to testing. This e-mail was an implicit request to Panzura to do whatever testing that they can. > > > We at iX Systems need to port ZoL's EC2 CI scripts to work with > > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. > > Being integrated in to their CI will mean that, among other things, > > most integration issues will be caught before a PR is merged upstream > > rather than many months later when it is MFVed into FreeBSD. I=E2=80=99= m > > hoping to submit the PR to ZoL some time in January. > > Could you please better describe this, in particular provide pointers to = any scripts that might be executed with the ZoL port? The ZFS test suite is in ZoL under tests, it makes some Linux specific assumptions about paths to binaries like ksh etc and it appears to require some additional configuration options. I started fixing things before I decided to punt and ask that a porter at iX fix it so that I could focus on areas that make more sense for me to spend my time on. If it doesn't get handled I will of course go back to it. The CI scripts are not something I've looked at. My intention is again that the people who would be responsible for keeping them up at iX would close the loop. It's close to Christmas time right now and folks are inevitably preoccupied with family matters. I sent this mail in advance of resolving these details to give any outside stakeholders a heads up as far out as possible. "Shadow" stakeholders coming out of the wood work to cry foul only after disruptive changes have been made - in spite of advance notice having been given on the mailing list(s) - has been an issue in the past. In this instance I intend to keep posting updates to the list so that no one can claim that sufficient effort was not made. > > This port will make it easy for end users on a range of releases to > > run the latest version of ZFS. Nonetheless, transitioning away from an > > Illumos based ZFS is not likely to be entirely seamless. The > > stakeholders I=E2=80=99ve spoken to all agree that this is the best pat= h > > forward but some degree of effort needs to be made to accommodate > > downstream consumers. The current plan is to import ZoF and unhook the > > older Illumos based sources from the build on April 15th or two months > > after iX systems QA deems ZoF stable - which ever comes later. The > > Illumos based sources will be removed some time later - but well > > before 13. This will give users a 3 month period during which both the > > port and legacy Illumos based ZFS will be available to users. Pools > > should interoperate between ZoF and legacy provided the user does not > > enable any features available only in ZoF. We will try to accommodate > > any downstream consumers in the event that they need that date pushed > > back. We ask that any downstream consumers who are particularly > > sensitive to changes start testing the port when it is formally > > announced and report back any issues they have. I will do my best to > > ensure that this message is communicated to all those who it may > > concern. However, I can=E2=80=99t ensure that everyone reads these list= s. That > > is the responsibility of -CURRENT users. > > > As a thought: is there a way for the ZoL port to run in parallel with the= non-ZoL port, kind of like in an audit/read-only/verify-only mode? Only one zfs kernel module can be loaded at a time. The command line utilities can't coexist due to libzfs differences. The closest one could come to this would be to switch between boot environments with the different kmods and utilities and run zdb from one on the pools that had been managed by the other. -M From owner-freebsd-fs@freebsd.org Wed Dec 19 15:07:54 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1AEB21338320 for ; Wed, 19 Dec 2018 15:07:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A96DC84797 for ; Wed, 19 Dec 2018 15:07:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 69FEA133831D; Wed, 19 Dec 2018 15:07:53 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5841E133831C for ; Wed, 19 Dec 2018 15:07:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E95EA84795 for ; Wed, 19 Dec 2018 15:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 41BFD1ABE1 for ; Wed, 19 Dec 2018 15:07:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBJF7qgK037771 for ; Wed, 19 Dec 2018 15:07:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBJF7qIG037764 for fs@FreeBSD.org; Wed, 19 Dec 2018 15:07:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 220604] Kernel panic when deleting ZFS snapshot Date: Wed, 19 Dec 2018 15:07:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: heppner.mark@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 15:07:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220604 --- Comment #2 from heppner.mark@gmail.com --- (In reply to Andriy Gapon from comment #1) The issue was consistent, I would always get a panic when trying to delete those particular snapshots. The box needed some hardware upgrades, so I ended up building a whole new machine anyways, since there was nothing I could do with the faulty ZFS. The box is still around if I can help to pull any useful info from it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 19 15:12:52 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EFCD133866E for ; Wed, 19 Dec 2018 15:12:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BD15284B9E for ; Wed, 19 Dec 2018 15:12:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 802EE133866C; Wed, 19 Dec 2018 15:12:51 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E7F6133866A for ; Wed, 19 Dec 2018 15:12:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C3BB84B9D for ; Wed, 19 Dec 2018 15:12:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5F8D91AD4D for ; Wed, 19 Dec 2018 15:12:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBJFCove023102 for ; Wed, 19 Dec 2018 15:12:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBJFCoIN023093 for fs@FreeBSD.org; Wed, 19 Dec 2018 15:12:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 220604] Kernel panic when deleting ZFS snapshot Date: Wed, 19 Dec 2018 15:12:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 15:12:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D220604 --- Comment #3 from Andriy Gapon --- (In reply to heppner.mark from comment #2) I suspect that what you have is an on-disk corruption that affects the spec= ific snapshot. I guess that it originated in memory, so its checksum is correct= , so it is invisible to the scrub. If you are interested, we can try to look at the details using kgdb and zdb. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 19 16:31:39 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4717F133BBD1 for ; Wed, 19 Dec 2018 16:31:39 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C0B888808 for ; Wed, 19 Dec 2018 16:31:38 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-ot1-x336.google.com with SMTP id a11so19598796otr.10 for ; Wed, 19 Dec 2018 08:31:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=cPuAQaogYXEWkURGYic2nUqfsUFib1b9wz0LZwZEeVI=; b=kcFWAXxZ7wl7ViYFZK1s30H/x9w0xJTWqfJANuPO1S8xmV0O8c9EpU9bhxS9pKxUiA h2Wb91QicEz/QropBzNuuF3OZV5Q6s1p+A+iiISIWcxks4a+ouBx8MYT0ZmJzZje4gOT biKsZMwJsJMTCXJdJzPL/HxAdvEa1IRGkP8FLdNBNT1uTm15CEeuaMX1I/+fXZ/RkgTw psXe/1Apj87XqE4IujYGOZI7L1rAiEcdZ2dqikp8DdnIWY7bLmbF7hUs+iawJi8wtsIV DhEgoVKL1DVdusgcUB+GWX6Tpjrrck98N25Xw2HCuJR6qV2ckJho4ngNWCVJY/2FDRuZ ylCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=cPuAQaogYXEWkURGYic2nUqfsUFib1b9wz0LZwZEeVI=; b=LkXOVMIENgBvEf7aXuLvkVYKX/X9iCfiwb4CXT4OanejxuXVoC4XDOSXp9y/y16isC 6HwMFLTkQ8meIwOANOdq/rN6j1WurUw7fG43LjrZW9FKYtPuJlpWD0BL4FZWhX2OunBM peF1wfvvWkcDHoImah1+qk3D4mmlhhXzgbg2+8Vz4Hss7VjKScd5Tew08djmCq+N5FTB nG78KfdyKS1NlobiMDgYSA4FyoXl2noX3Fgv0glrWL4YpNUV1Ip9uT/MJOBTc2cxm64s Ni7iN3weQkVFIjr/BOJeTUdmB6F1OuptVlRLlI+xpD9PnKFtvwujLjZqBRyLvsggdizB xYYA== X-Gm-Message-State: AA+aEWapjeHZ6cayQEQ9R+2GztlyavpgGphd3UUuEgqzlxUlAWb6S6KJ lZNVBkZB3ImNvAdEhIYS9UcVBQ== X-Google-Smtp-Source: AFSGD/VsWFsJZrpM4UL1eBiWLxBEyZeAV76xSq+Dwa1Sv4Q7C7wQI0qq0T0iuLQmHid2v1B1w9+Ing== X-Received: by 2002:a9d:3b77:: with SMTP id z110mr5049525otb.352.1545237096850; Wed, 19 Dec 2018 08:31:36 -0800 (PST) Received: from mutt-hbsd ([185.100.87.129]) by smtp.gmail.com with ESMTPSA id k13sm16950451otj.19.2018.12.19.08.31.33 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Dec 2018 08:31:36 -0800 (PST) Date: Wed, 19 Dec 2018 11:30:33 -0500 From: Shawn Webb To: Matthew Macy Cc: freebsd-fs , freebsd-current Subject: Re: The future of ZFS in FreeBSD Message-ID: <20181219163033.jwm7opiwmdhbk6p3@mutt-hbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rlmvscpkcfr6pslq" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 13.0-CURRENT FreeBSD 13.0-CURRENT HARDENEDBSD-13-CURRENT amd64 X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180716 X-Rspamd-Queue-Id: 0C0B888808 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hardenedbsd.org header.s=google header.b=kcFWAXxZ; spf=pass (mx1.freebsd.org: domain of shawn.webb@hardenedbsd.org designates 2607:f8b0:4864:20::336 as permitted sender) smtp.mailfrom=shawn.webb@hardenedbsd.org X-Spamd-Result: default: False [-2.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[hardenedbsd.org:+]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; NEURAL_HAM_SHORT(-0.91)[-0.915,0]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; IP_SCORE(-0.64)[ipnet: 2607:f8b0::/32(-1.75), asn: 15169(-1.39), country: US(-0.08)]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(3.00)[129.87.100.185.zen.spamhaus.org : 127.0.0.4]; R_DKIM_ALLOW(0.00)[hardenedbsd.org:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[hardenedbsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[6.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MID_RHS_NOT_FQDN(0.50)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 16:31:39 -0000 --rlmvscpkcfr6pslq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote: > The sources for FreeBSD's ZFS support are currently taken directly > from Illumos with local ifdefs to support the peculiarities of FreeBSD > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD > has regularly pulled changes from Illumos and tried to push back any > bug fixes and new features done in the context of FreeBSD. In the past > few years the vast majority of new development in ZFS has taken place > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced > that they will be moving to ZoL > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means > that there will be little to no net new development of Illumos. While > working through the git history of ZoL I have also discovered that > many races and locking bugs have been fixed in ZoL and never made it > back to Illumos and thus FreeBSD. This state of affairs has led to a > general agreement among the stakeholders that I have spoken to that it > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf > has graciously encouraged me to add FreeBSD support directly to ZoL > https://github.com/zfsonfreebsd/ZoF so that we might all have a single > shared code base. >=20 > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port > Before it can be committed some additional functionality needs to be > added to the FreeBSD opencrypto framework. These can be found at > https://reviews.freebsd.org/D18520 >=20 > This port will provide FreeBSD users with multi modifier protection, > project quotas, encrypted datasets, allocation classes, vectorized > raidz, vectorized checksums, and various command line improvements. >=20 > Before ZoF can be merged back in to ZoL several steps need to be taken: > - Integrate FreeBSD support into ZoL CI > - Have most of the ZFS test suite passing > - Complete additional QA testing at iX >=20 > We at iX Systems need to port ZoL's EC2 CI scripts to work with > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. > Being integrated in to their CI will mean that, among other things, > most integration issues will be caught before a PR is merged upstream > rather than many months later when it is MFVed into FreeBSD. I???m > hoping to submit the PR to ZoL some time in January. >=20 > This port will make it easy for end users on a range of releases to > run the latest version of ZFS. Nonetheless, transitioning away from an > Illumos based ZFS is not likely to be entirely seamless. The > stakeholders I???ve spoken to all agree that this is the best path > forward but some degree of effort needs to be made to accommodate > downstream consumers. The current plan is to import ZoF and unhook the > older Illumos based sources from the build on April 15th or two months > after iX systems QA deems ZoF stable - which ever comes later. The > Illumos based sources will be removed some time later - but well > before 13. This will give users a 3 month period during which both the > port and legacy Illumos based ZFS will be available to users. Pools > should interoperate between ZoF and legacy provided the user does not > enable any features available only in ZoF. We will try to accommodate > any downstream consumers in the event that they need that date pushed > back. We ask that any downstream consumers who are particularly > sensitive to changes start testing the port when it is formally > announced and report back any issues they have. I will do my best to > ensure that this message is communicated to all those who it may > concern. However, I can???t ensure that everyone reads these lists. That > is the responsibility of -CURRENT users. Hey Matt, Thank you for your detailed and informative post. It really helps downstream consumers of FreeBSD. I'm curious what this means for OpenZFS. I was under the impression that OpenZFS was the upstream for all the ZFS implementations (sans Oracle). Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --rlmvscpkcfr6pslq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAlwaciUACgkQaoRlj1JF bu7GnxAAkKRQBJ1ypOD0J+FxS/K5fqXH+mBD2DW66/iU0EPcAsTZFPj5nhsCuhLL znBjvCH4nnMu1vYoTwn0Yeism1ioYz6EGxfdxVfyz/X+cOv1Bf9tGiKwvm5U8TvM AK2YIaRHL8lWLeQzMapJx5xm1PVPIo6FPyVOVZSCP0JetfJ96ahTPOOWcOozk8h6 vYurzIXnsA7IBxEy22/h3h1ZyJ4IxRFW81XAGsdQPdhWA2V2eBGka9UGisk5cgmZ vxRnZH/8RBjDaGNnzrpZBlND7Ukyu1PZeJmdF01wInB7ruTfDmdTYeQK2VMfmHOP feU70wyFI7AhVPQ2SbM1fUDVvwkG0kS1Q+qIX6X5EHqbOER9g0cUbphEYIvFpMQD ICr2m2LpeJGem1HMCtZb/j+rChRJtvKytIk60QC+lXDzhRhiR9irM7pfCNiJ1RW2 xCLhjEkCNtMYVw/5KpdqmN91GNBnp7BdhzKSKSW6xiB5CECWzEv9qexQlmxT7OzS 0bsg2EdYLc4C4YkL2fw2UF2y5oYUMI9niS7n0rRZpMbD8eTK6yhAlhHEBWDvhDUT c/ITYnduUf0yW39n9uQNQgYdauxnj8yJwK0eIQNFAIMgBxDZAgm+s0KmxAyCNg/P c8uSzZBPFdpHCBbeSybGCBEpABn9EULervJJ8KEFBeoaoNSqMvg= =IVGV -----END PGP SIGNATURE----- --rlmvscpkcfr6pslq-- From owner-freebsd-fs@freebsd.org Wed Dec 19 17:53:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5BB7133E3EB; Wed, 19 Dec 2018 17:53:10 +0000 (UTC) (envelope-from andrew@fubar.geek.nz) Received: from fry.fubar.geek.nz (fry.fubar.geek.nz [139.59.165.16]) by mx1.freebsd.org (Postfix) with ESMTP id 2A1CF8BF81; Wed, 19 Dec 2018 17:53:07 +0000 (UTC) (envelope-from andrew@fubar.geek.nz) Received: from [192.168.100.30] (unknown [185.201.60.232]) by fry.fubar.geek.nz (Postfix) with ESMTPSA id AF27A4E693; Wed, 19 Dec 2018 17:52:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fubar.geek.nz; s=mail; t=1545241947; bh=2jTW0oXQtVXSFTNoTNFg6fuGj3HRIfreps/0dhdiIJ8=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=pJutvXEZMl+P/UQlP83/gQq2JpAfrNBX/Nc8WCDliG0oGxh/Us68tMiK8in54H8NP eeizQf/NPaOXjckY2qyRhaW8o+4AsZjiU0dH5jcNdlOKkOKfPjd5+u+99nZJQjGz9b zqffgmGKO3p49JA8n8ZYey+DHAnjMrEfY5hz/LAPZeBqfQV5UmFQUb3UWU5zo2MKCT b0ga5VDyeQaXuLHcMC76WtiLaDWwYw0AFwTAkawISG4mDL8+QEq6VyUmp70onF9iiI kwcmHv6gRI2SdyelPAQLx4BtgMa4imTEKSlLb7WlWOcccRXN/kgoPrFfpYJPXRnPKE u61L/MMso0Y6PN75ImSfody/Z+fNY8PIn0sYOJ3nG9y9RCVdfYw72Yv8+Y/TH/g2/L 6JdSSEcErAVNZ8k0X8yw54S3QtKlPVrbgmube/YoUXW9YCR4dIVR3NSNJMP2jToi68 Y+3iVxrrlgQGUO1tsCoxRMsQCMhqO4WMKmOqUxFatS9p7sCnY89qwuedaIdsQTKc/D d+4po97KCPaMKzjI1F08jVlKe1JY2cN/O7fQdPAZniGqDYC8VKN0xw6rGwVhkBcxum NZ0XZ8QBskqE5cHZ9Asb0gR6fO3sHQZ2CZ+8apVlmliELiHcxGX6jxQJlzjulguGh9 RYTTdDi+2jwmLBV6GQ5JD554= From: Andrew Turner Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: The future of ZFS in FreeBSD Date: Wed, 19 Dec 2018 17:52:24 +0000 In-Reply-To: Cc: Enji Cooper , freebsd-fs , freebsd-current To: Matthew Macy References: <9C23F0C0-DEF7-45A4-ADEF-58A00F9714D8@gmail.com> X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 2A1CF8BF81 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=fubar.geek.nz header.s=mail header.b=pJutvXEZ; dmarc=pass (policy=none) header.from=fubar.geek.nz; spf=pass (mx1.freebsd.org: domain of andrew@fubar.geek.nz designates 139.59.165.16 as permitted sender) smtp.mailfrom=andrew@fubar.geek.nz X-Spamd-Result: default: False [3.25 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[fubar.geek.nz:s=mail]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; NEURAL_SPAM_SHORT(0.75)[0.752,0]; NEURAL_SPAM_MEDIUM(0.60)[0.604,0]; URI_COUNT_ODD(1.00)[11]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[fubar.geek.nz:+]; MX_GOOD(-0.01)[fry.fubar.geek.nz]; DMARC_POLICY_ALLOW(-0.50)[fubar.geek.nz,none]; NEURAL_SPAM_LONG(0.68)[0.679,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; IP_SCORE(0.63)[asn: 14061(3.22), country: US(-0.08)]; ASN(0.00)[asn:14061, ipnet:139.59.160.0/20, country:US]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 17:53:11 -0000 > On 19 Dec 2018, at 08:35, Matthew Macy wrote: >=20 > On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper > wrote: >>=20 >> Hello Matthew, >>=20 >> I appreciate the long write up, as someone who still uses FreeBSD ZFS = on my NAS box and knowing some of the history with ZFS on *Solaris, etc. = Something like this was bound to happen with post the Oracle buyout. >>=20 >>> On Dec 18, 2018, at 10:49 PM, Matthew Macy = wrote: >>>=20 >>> The sources for FreeBSD's ZFS support are currently taken directly >>> from Illumos with local ifdefs to support the peculiarities of = FreeBSD >>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD >>> has regularly pulled changes from Illumos and tried to push back any >>> bug fixes and new features done in the context of FreeBSD. In the = past >>> few years the vast majority of new development in ZFS has taken = place >>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix = announced >>> that they will be moving to ZoL >>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift = means >>> that there will be little to no net new development of Illumos. = While >>> working through the git history of ZoL I have also discovered that >>> many races and locking bugs have been fixed in ZoL and never made it >>> back to Illumos and thus FreeBSD. This state of affairs has led to a >>> general agreement among the stakeholders that I have spoken to that = it >>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf >>> has graciously encouraged me to add FreeBSD support directly to ZoL >>> https://github.com/zfsonfreebsd/ZoF so that we might all have a = single >>> shared code base. >>>=20 >>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port >>> Before it can be committed some additional functionality needs to be >>> added to the FreeBSD opencrypto framework. These can be found at >>> https://reviews.freebsd.org/D18520 >>>=20 >>> This port will provide FreeBSD users with multi modifier protection, >>> project quotas, encrypted datasets, allocation classes, vectorized >>> raidz, vectorized checksums, and various command line improvements. >>>=20 >>> Before ZoF can be merged back in to ZoL several steps need to be = taken: >>> - Integrate FreeBSD support into ZoL CI >>> - Have most of the ZFS test suite passing >>> - Complete additional QA testing at iX >>=20 >> Can you please describe the testing process that will be employed to = verify the sanity of the ZoL on FreeBSD port? Should other large = companies who use ZFS on FreeBSD (Panzura?) chime in and the ZFS on = FreeBSD community (as a whole) collaborate to better suss out issues = with the ZoL port? >=20 > The ZFS test suite itself provides ~80% coverage > https://codecov.io/gh/zfsonlinux/zfs/branch/master = - FreeBSD currently > lacks equivalent gcov support, but presumably it would provide > comparable coverage here. Andrew Turner has some form of kernel gcov > support that he uses with syzkaller. However, I believe that it isn't > sufficient for this purpose. The code I have is to trace the kernel part of a single thread, e.g. = what happens in the kernel when you make a system call. It can trace = function either function entry or places in the code with a conditional = statement, e.g. an if, while, or switch statement. If this is enough for = your case a tool could be written to track coverage from the tests. I expect to update the review soon. I=E2=80=99m working on a man page, = but have been too busy with work and other projects recently to finish = writing it. Andrew From owner-freebsd-fs@freebsd.org Wed Dec 19 18:06:19 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 473FA133EB54 for ; Wed, 19 Dec 2018 18:06:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D28848CA11 for ; Wed, 19 Dec 2018 18:06:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 962CB133EB53; Wed, 19 Dec 2018 18:06:18 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84AA2133EB52 for ; Wed, 19 Dec 2018 18:06:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 241BD8CA0D for ; Wed, 19 Dec 2018 18:06:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 676B71C64A for ; Wed, 19 Dec 2018 18:06:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBJI6HlE026591 for ; Wed, 19 Dec 2018 18:06:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBJI6HbB026587 for fs@FreeBSD.org; Wed, 19 Dec 2018 18:06:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 229614] ZFS lockup in zil_commit_impl Date: Wed, 19 Dec 2018 18:06:13 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: allanjude@FreeBSD.org X-Bugzilla-Flags: mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 18:06:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D229614 --- Comment #45 from commit-hook@freebsd.org --- A commit references this bug: Author: emaste Date: Wed Dec 19 18:05:50 UTC 2018 New revision: 342226 URL: https://svnweb.freebsd.org/changeset/base/342226 Log: MFS11 r341828: Resolve a hang in ZFS during vnode reclaimation This is caused by a deadlock between zil_commit() and zfs_zget() Add a way for zfs_zget() to break out of the retry loop in the common c= ase PR: 229614, 231117 Submitted by: allanjude Approved by: so Security: FreeBSD-EN-18:18.zfs Sponsored by: Klara Systems, The FreeBSD Foundation Changes: _U releng/11.2/ releng/11.2/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-fs@freebsd.org Wed Dec 19 18:11:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1DD9133F1A4 for ; Wed, 19 Dec 2018 18:11:47 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: from mail-ua1-x92a.google.com (mail-ua1-x92a.google.com [IPv6:2607:f8b0:4864:20::92a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 613578CECF for ; Wed, 19 Dec 2018 18:11:46 +0000 (UTC) (envelope-from matthew.ahrens@delphix.com) Received: by mail-ua1-x92a.google.com with SMTP id p9so7298527uaa.5 for ; Wed, 19 Dec 2018 10:11:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=delphix.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hOzIIq2vNzJWteRgLorlAboeBmU/3kELJ8R/3Rt+0nE=; b=FqSMie6OnvCIQZmLoa2UC2ejz1MKrWEyoq7TQGr/MjnLwRX6N0kCH0DsW/i37hE4zd FVPkCEywLyiiNxbcsCakPehMu6vGdeX5yiHyWC/GRgMNPo3pvk0BP+L6SFWx9nMpF/a/ J39Aw6Am+7ynYKbDpL39Ik+79HrxpyH6zXgvs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hOzIIq2vNzJWteRgLorlAboeBmU/3kELJ8R/3Rt+0nE=; b=HS0ihl1llm3Szop9Jo1Dd0S/MTH7qL34u3Zu5R/RBc0zseV1sY7rAlXPc6WOBmJanp 2j9VTye3MJysY1RFvF0d2dTgo+NSagY23htE1Wk+gkKoLxtUdaLrdYbzT0+ULaPu5VLy HXaPiwLgsAl7IdV+o4o7oO9B3M34Lpb1T/QwtizHdpnXP4C2PVzu0L7p474pTVasyYrM W8pmkZZMAN1tVY+dItBmUnxQdpuQrMpMUs0qyxDx7V5LJE+eQP9yDLK15xIVLNMkgHSf +Nu4vwMBV4FStVyjTLPhp5IlEGKJYLiljNOJq5gQAGVi+ZvhEJKMYgAiEhA3eRGqPo72 Jc9g== X-Gm-Message-State: AA+aEWa+aypgmwWNg+XC7NR1Tokcr0lS0q2eME4IfYei/bwuLu4qNxDa b+86i2o+xdZXybLmDERUkja1V3vFblz9LFDQSHQbVg== X-Google-Smtp-Source: AFSGD/UdGkh2X9fm0/b9FmZz0MVGS6rHngE9l9h4oLagdoqh8SN93eQ/YwxuC837w/49ok6d08m3pECV7zCpw2w5pqE= X-Received: by 2002:ab0:73c4:: with SMTP id m4mr10762504uaq.101.1545243105000; Wed, 19 Dec 2018 10:11:45 -0800 (PST) MIME-Version: 1.0 References: <20181219163033.jwm7opiwmdhbk6p3@mutt-hbsd> In-Reply-To: <20181219163033.jwm7opiwmdhbk6p3@mutt-hbsd> From: Matthew Ahrens Date: Wed, 19 Dec 2018 10:11:32 -0800 Message-ID: Subject: Re: The future of ZFS in FreeBSD To: shawn.webb@hardenedbsd.org Cc: mmacy@freebsd.org, freebsd-fs , freebsd-current@freebsd.org X-Rspamd-Queue-Id: 613578CECF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=delphix.com header.s=google header.b=FqSMie6O; dmarc=pass (policy=none) header.from=delphix.com; spf=pass (mx1.freebsd.org: domain of matthew.ahrens@delphix.com designates 2607:f8b0:4864:20::92a as permitted sender) smtp.mailfrom=matthew.ahrens@delphix.com X-Spamd-Result: default: False [-3.82 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[delphix.com:s=google]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[delphix.com:+]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx3.googlemail.com]; RCVD_IN_DNSWL_NONE(0.00)[a.2.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.47)[-0.468,0]; DMARC_POLICY_ALLOW(-0.50)[delphix.com,none]; IP_SCORE(-0.64)[ipnet: 2607:f8b0::/32(-1.75), asn: 15169(-1.39), country: US(-0.08)]; FORGED_SENDER(0.30)[mahrens@delphix.com,matthew.ahrens@delphix.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[mahrens@delphix.com,matthew.ahrens@delphix.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 18:11:48 -0000 On Wed, Dec 19, 2018 at 8:35 AM Shawn Webb wrote: > I'm curious what this means for OpenZFS. > OpenZFS will continue to be an umbrella project for coordinating all work on open-source ZFS. The primary activities of OpenZFS are the annual OpenZFS Developer Summit and the monthly OpenZFS Leadership Meetings . There is also an OpenZFS GitHub repo, which has identical code to illumos. This is used to streamline the process of getting changes into illumos (removing some of the burden of testing and the illumos RTI process). This repo will continue to exist as long as there's need for it and folks able to keep the automation running. The naming of this GitHub project was perhaps unfortunate, since it is specific to illumos and not platform-agnostic. I was under the impression that OpenZFS was the upstream for all the ZFS > implementations (sans > Oracle). Being "upstream" is more a matter of degree than an absolute ordering. In the past, most ZFS features have originated in illumos, and those changes have been ported to the other platforms (notably FreeBSD and ZoL). Looking forward, we see more features originating in ZoL. The OpenZFS community (including FreeBSD) is working to put in place technical processes to ensure that new ZFS features are available on all platforms. The original post in this thread was a proposal for how to do that. --matt From owner-freebsd-fs@freebsd.org Wed Dec 19 22:46:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93277134B9EA for ; Wed, 19 Dec 2018 22:46:59 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AB9D1742DA for ; Wed, 19 Dec 2018 22:46:57 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-lf1-x12a.google.com with SMTP id v5so16293148lfe.7 for ; Wed, 19 Dec 2018 14:46:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=j4JVyF3Ru+/bxAAy2sknhbV8VkaTEa8gvgJAlGIhRL8=; b=q7/EAzxt6gSGQzz0WsX3SXkmyEoRUFGQdq0d2NJvlwA3ZSFUYKDQC2eB4OGdonIuEW 1YJs7SKbQOiSkMRrIHyS+oAazfqM7kmiMoD64vutd0o3M9Ou9+b6SXxlc5Ssd7HZBgmr sXUiReTlaT2qtZblSelPRq0Ec01mw0XukI0Ho7/FjYmr2im+ZGePkFzDUbJHjH7xzgv/ KSQ9PNkJYU0eJObhNQaFDAn/EdIhE6JsSdF6JQa3zzk8xquqyaVF6do2Jv10RJv5EhLp NWIggirYie97EEFTjtrSphUcolnSazw0Dj9qV6P8xDiw9NZXWfYzFOaIj68MEAzmUF86 Czig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=j4JVyF3Ru+/bxAAy2sknhbV8VkaTEa8gvgJAlGIhRL8=; b=DmY02AJ7ixumSHCwpfPvTfNO3kzmlM8CkiS2n2+uAn6pV/9Y5ej4jmXP3xi8xBQPUA pizNs/RJxyfQm130bzRrzvOl1nTH9+OLqCRih+a55O45PxUR5f/Uf7LHMiVtNym54PmF tlFtkN0B40UrJVN/+ww2LRQrVwPBO8/NaXhPPP3abDfq1KABlMUyqJth9/4i/LM/B/WF e8sBqcS10BWR345PkXStehc5sstIlH1P3HhhYdso2GDgKaJavbhIeyUAPRqXAuXh5XzD kApjEwxU5BT3aIfUFMzv4JPx1yN8rhXwWuIpkaewuHc+Y9fB5qs39SGvMwmEXsyVPUDd DV3A== X-Gm-Message-State: AA+aEWaDXIFngJ3K4aBHb/tJpNfGiK2IBDr08OrkmGwT8B+0IkerZPap TACcaHwUQXUy2NBJSpsp1aJ9CAUsiRI8MLx2Sh5uvg== X-Google-Smtp-Source: AFSGD/WQD/RE+5tEzWyD025ciYbkObwvlK2dicWiBxl6Xcxv07Pb0jIn/BVxO9WqgLe2tXKCJLdb5c2CAiCdMjKnitI= X-Received: by 2002:a19:2755:: with SMTP id n82mr12762571lfn.94.1545259615325; Wed, 19 Dec 2018 14:46:55 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Steven Hartland Date: Wed, 19 Dec 2018 22:46:44 +0000 Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Matthew Macy Cc: freebsd-current , freebsd-fs X-Rspamd-Queue-Id: AB9D1742DA X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=q7/EAzxt; spf=pass (mx1.freebsd.org: domain of steven@multiplay.co.uk designates 2a00:1450:4864:20::12a as permitted sender) smtp.mailfrom=steven@multiplay.co.uk X-Spamd-Result: default: False [-5.65 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ASPMX.L.GOOGLE.COM,ALT2.ASPMX.L.GOOGLE.COM,ALT1.ASPMX.L.GOOGLE.COM,ASPMX2.GOOGLEMAIL.COM,ASPMX3.GOOGLEMAIL.COM]; RCVD_IN_DNSWL_NONE(0.00)[a.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; IP_SCORE(-2.46)[ip: (-9.25), ipnet: 2a00:1450::/32(-1.60), asn: 15169(-1.40), country: US(-0.08)]; FORGED_SENDER(0.30)[killing@multiplay.co.uk,steven@multiplay.co.uk]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[killing@multiplay.co.uk,steven@multiplay.co.uk]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 22:46:59 -0000 Thanks for the write up most appreciated. One of the more meaty differences is that FreeBSD ZFS still has the only merged and production ready TRIM support so my question would be are their any plans to address this before creating the new port as going back to a world without TRIM support wouldn=E2=80=99t be something I=E2=80=99d look forward to. On Wed, 19 Dec 2018 at 06:51, Matthew Macy wrote: > The sources for FreeBSD's ZFS support are currently taken directly > from Illumos with local ifdefs to support the peculiarities of FreeBSD > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD > has regularly pulled changes from Illumos and tried to push back any > bug fixes and new features done in the context of FreeBSD. In the past > few years the vast majority of new development in ZFS has taken place > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced > that they will be moving to ZoL > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means > that there will be little to no net new development of Illumos. While > working through the git history of ZoL I have also discovered that > many races and locking bugs have been fixed in ZoL and never made it > back to Illumos and thus FreeBSD. This state of affairs has led to a > general agreement among the stakeholders that I have spoken to that it > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf > has graciously encouraged me to add FreeBSD support directly to ZoL > https://github.com/zfsonfreebsd/ZoF so that we might all have a single > shared code base. > > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port > Before it can be committed some additional functionality needs to be > added to the FreeBSD opencrypto framework. These can be found at > https://reviews.freebsd.org/D18520 > > This port will provide FreeBSD users with multi modifier protection, > project quotas, encrypted datasets, allocation classes, vectorized > raidz, vectorized checksums, and various command line improvements. > > Before ZoF can be merged back in to ZoL several steps need to be taken: > - Integrate FreeBSD support into ZoL CI > - Have most of the ZFS test suite passing > - Complete additional QA testing at iX > > We at iX Systems need to port ZoL's EC2 CI scripts to work with > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. > Being integrated in to their CI will mean that, among other things, > most integration issues will be caught before a PR is merged upstream > rather than many months later when it is MFVed into FreeBSD. I=E2=80=99m > hoping to submit the PR to ZoL some time in January. > > This port will make it easy for end users on a range of releases to > run the latest version of ZFS. Nonetheless, transitioning away from an > Illumos based ZFS is not likely to be entirely seamless. The > stakeholders I=E2=80=99ve spoken to all agree that this is the best path > forward but some degree of effort needs to be made to accommodate > downstream consumers. The current plan is to import ZoF and unhook the > older Illumos based sources from the build on April 15th or two months > after iX systems QA deems ZoF stable - which ever comes later. The > Illumos based sources will be removed some time later - but well > before 13. This will give users a 3 month period during which both the > port and legacy Illumos based ZFS will be available to users. Pools > should interoperate between ZoF and legacy provided the user does not > enable any features available only in ZoF. We will try to accommodate > any downstream consumers in the event that they need that date pushed > back. We ask that any downstream consumers who are particularly > sensitive to changes start testing the port when it is formally > announced and report back any issues they have. I will do my best to > ensure that this message is communicated to all those who it may > concern. However, I can=E2=80=99t ensure that everyone reads these lists.= That > is the responsibility of -CURRENT users. > > -M > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Wed Dec 19 22:52:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49E50134C10C; Wed, 19 Dec 2018 22:52:30 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E199A74C53; Wed, 19 Dec 2018 22:52:29 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id AB4795E3C; Wed, 19 Dec 2018 22:52:29 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io1-f43.google.com with SMTP id l14so16928294ioj.5; Wed, 19 Dec 2018 14:52:29 -0800 (PST) X-Gm-Message-State: AA+aEWYcclioekmN1H96Whvq+gldXirvD0Sa+KZic86Te1jc4/tSbJwF f38HRUv2hUzT9G1fYOGysXoeeynHNyDm2acWnCc= X-Google-Smtp-Source: AFSGD/VstMykI/2JblKobGbwAbThi/E5wXJtdfND8/+aJ+XCT4RFjTq3poqgS9+ZTP6DefoC+JlTovszjnU/ji8jXcw= X-Received: by 2002:a6b:8f8d:: with SMTP id r135mr19621676iod.5.1545259949011; Wed, 19 Dec 2018 14:52:29 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Wed, 19 Dec 2018 14:52:17 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Steven Hartland Cc: freebsd-current , freebsd-fs X-Rspamd-Queue-Id: E199A74C53 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.985,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 22:52:30 -0000 On Wed, Dec 19, 2018 at 14:47 Steven Hartland wrote: > Thanks for the write up most appreciated. One of the more meaty > differences is that FreeBSD ZFS still has the only merged and production > ready TRIM support so my question would be are their any plans to address > this before creating the new port as going back to a world without TRIM > support wouldn=E2=80=99t be something I=E2=80=99d look forward to. > Well, then please follow up on the request I CC'd you on a week ago asking that you engage on the deadlist based TRIM PR. That's a better forum for discussing these details than lamenting in public lists. Thanks. -M > On Wed, 19 Dec 2018 at 06:51, Matthew Macy wrote: > >> The sources for FreeBSD's ZFS support are currently taken directly >> from Illumos with local ifdefs to support the peculiarities of FreeBSD >> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD >> has regularly pulled changes from Illumos and tried to push back any >> bug fixes and new features done in the context of FreeBSD. In the past >> few years the vast majority of new development in ZFS has taken place >> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced >> that they will be moving to ZoL >> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means >> that there will be little to no net new development of Illumos. While >> working through the git history of ZoL I have also discovered that >> many races and locking bugs have been fixed in ZoL and never made it >> back to Illumos and thus FreeBSD. This state of affairs has led to a >> general agreement among the stakeholders that I have spoken to that it >> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf >> has graciously encouraged me to add FreeBSD support directly to ZoL >> https://github.com/zfsonfreebsd/ZoF so that we might all have a single >> shared code base. >> >> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port >> Before it can be committed some additional functionality needs to be >> added to the FreeBSD opencrypto framework. These can be found at >> https://reviews.freebsd.org/D18520 >> >> This port will provide FreeBSD users with multi modifier protection, >> project quotas, encrypted datasets, allocation classes, vectorized >> raidz, vectorized checksums, and various command line improvements. >> >> Before ZoF can be merged back in to ZoL several steps need to be taken: >> - Integrate FreeBSD support into ZoL CI >> - Have most of the ZFS test suite passing >> - Complete additional QA testing at iX >> >> We at iX Systems need to port ZoL's EC2 CI scripts to work with >> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. >> Being integrated in to their CI will mean that, among other things, >> most integration issues will be caught before a PR is merged upstream >> rather than many months later when it is MFVed into FreeBSD. I=E2=80=99m >> hoping to submit the PR to ZoL some time in January. >> >> This port will make it easy for end users on a range of releases to >> run the latest version of ZFS. Nonetheless, transitioning away from an >> Illumos based ZFS is not likely to be entirely seamless. The >> stakeholders I=E2=80=99ve spoken to all agree that this is the best path >> forward but some degree of effort needs to be made to accommodate >> downstream consumers. The current plan is to import ZoF and unhook the >> older Illumos based sources from the build on April 15th or two months >> after iX systems QA deems ZoF stable - which ever comes later. The >> Illumos based sources will be removed some time later - but well >> before 13. This will give users a 3 month period during which both the >> port and legacy Illumos based ZFS will be available to users. Pools >> should interoperate between ZoF and legacy provided the user does not >> enable any features available only in ZoF. We will try to accommodate >> any downstream consumers in the event that they need that date pushed >> back. We ask that any downstream consumers who are particularly >> sensitive to changes start testing the port when it is formally >> announced and report back any issues they have. I will do my best to >> ensure that this message is communicated to all those who it may >> concern. However, I can=E2=80=99t ensure that everyone reads these lists= . That >> is the responsibility of -CURRENT users. >> >> -M >> > _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> > From owner-freebsd-fs@freebsd.org Wed Dec 19 23:11:13 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7CAA134CC13 for ; Wed, 19 Dec 2018 23:11:12 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B505B76094 for ; Wed, 19 Dec 2018 23:11:11 +0000 (UTC) (envelope-from steven@multiplay.co.uk) Received: by mail-lf1-x12c.google.com with SMTP id a16so16321972lfg.3 for ; Wed, 19 Dec 2018 15:11:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=c7Dea6s43TH+P3wVxfgVcN5wbIs+u84XhTeMRL6khbk=; b=e/9uaC1pKHXURuDsd1YoFWsJFUey3ACuiUKWQTjm7MTLY7Kgz/fWv9R4w1dTZJbX+R yE1yypRgT7LZJRMjI727z7BP6Z/o6tgcTD9iW+chMro/RdbyHnKvAHAdCuGEJzdx+RrX zoBWRU+q2KnRrqSe/BK7u8+Wk4C+QGrxN9Hdfzy9GfbjGYxFSQPd3Uz/Ng8P4MIbE4ye qOClWbvD4ATzalCQcBJMVkr5i20e7kQ5xC04KBGRyF5RblaClEGkN4VQxYZ/YTNpQ9/6 zgWgOEIyKsa2YffQz6vWPq8p3AD/uBl07vak7MdR3lWtFymfsahv4n5iZdEhpilU5PZf EQ3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=c7Dea6s43TH+P3wVxfgVcN5wbIs+u84XhTeMRL6khbk=; b=BhI4xkjqKSQF3Fm1Nsw9oWyAqeZpVjp3d8X3rYcA0/AY+yAew/2Wwkaa92XbUeevHe 30swyv6MnTu4rBEbmT1v9AnO0/oJbifmEW2hdkjWrhpsI3uR70BuBQTSk2Ir53VniNnF y0RfjGGmEc8qp7XISJxhi4XylMFW+STHKNNf1Gr/OtHHkYwQkJKgneTGN3DyozcqAK/m dl8zALu+YN+UEsyD9O8Qg8Lrn4zWmmjrJ/XZkkVyErllI67TY0YvhbaTuroSdh5vny4m 3z593pXtJaMXSkgRPtvzlf75ZUqfkWnWoUznMOo6roBfrQ0B5PC8bEjfq/jgiFOxfWrl 9X2g== X-Gm-Message-State: AA+aEWY8ZRX8QpvTbqzXn7S2FQjV7aUuemePFI4y1IxpO3KsGZGUvvzB 3GIT1Ke31Qn5dacmmcTmf3Ky/coM+4/RbhPI5JdWFsYE X-Google-Smtp-Source: AFSGD/WkshJRYak8k9UVXV6nUDeBUZ3JSbUNY1YYEeLf17YgUK1L0LvsyKxuOo59khaRovJCBD3z71RJB1Adc2syZ/w= X-Received: by 2002:a19:5154:: with SMTP id f81mr14370165lfb.96.1545261070008; Wed, 19 Dec 2018 15:11:10 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Steven Hartland Date: Wed, 19 Dec 2018 23:10:58 +0000 Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Matthew Macy Cc: freebsd-current , freebsd-fs X-Rspamd-Queue-Id: B505B76094 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=e/9uaC1p; spf=pass (mx1.freebsd.org: domain of steven@multiplay.co.uk designates 2a00:1450:4864:20::12c as permitted sender) smtp.mailfrom=steven@multiplay.co.uk X-Spamd-Result: default: False [-3.33 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-0.99)[-0.992,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; RCVD_IN_DNSWL_NONE(0.00)[c.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.52)[-0.516,0]; IP_SCORE(-0.62)[ipnet: 2a00:1450::/32(-1.60), asn: 15169(-1.40), country: US(-0.08)]; FORGED_SENDER(0.30)[killing@multiplay.co.uk,steven@multiplay.co.uk]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[killing@multiplay.co.uk,steven@multiplay.co.uk]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 23:11:13 -0000 Sorry been off for a few weeks so must have missed that, please do prod me on again if you don=E2=80=99t see any response to anything not just this. L= ike many others I get so may emails across so many lists it=E2=80=99s more than like= ly I just missed it. That said would you say that with the right support we can make progress on the this prior to the port? I have to ask as the alternative version has been on the cusp for many years now so it=E2=80=99s feels more like a dista= nt memory than something that may happen, no disrespect to anyone involved, as I know all too well how hard it can be to get something like this over the line, especially when people have competing priorities. On Wed, 19 Dec 2018 at 22:52, Matthew Macy wrote: > > > On Wed, Dec 19, 2018 at 14:47 Steven Hartland > wrote: > >> Thanks for the write up most appreciated. One of the more meaty >> differences is that FreeBSD ZFS still has the only merged and production >> ready TRIM support so my question would be are their any plans to addres= s >> this before creating the new port as going back to a world without TRIM >> support wouldn=E2=80=99t be something I=E2=80=99d look forward to. >> > > Well, then please follow up on the request I CC'd you on a week ago askin= g > that you engage on the deadlist based TRIM PR. That's a better forum for > discussing these details than lamenting in public lists. > > Thanks. > > -M > > > > >> On Wed, 19 Dec 2018 at 06:51, Matthew Macy wrote: >> >>> The sources for FreeBSD's ZFS support are currently taken directly >>> from Illumos with local ifdefs to support the peculiarities of FreeBSD >>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD >>> has regularly pulled changes from Illumos and tried to push back any >>> bug fixes and new features done in the context of FreeBSD. In the past >>> few years the vast majority of new development in ZFS has taken place >>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced >>> that they will be moving to ZoL >>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means >>> that there will be little to no net new development of Illumos. While >>> working through the git history of ZoL I have also discovered that >>> many races and locking bugs have been fixed in ZoL and never made it >>> back to Illumos and thus FreeBSD. This state of affairs has led to a >>> general agreement among the stakeholders that I have spoken to that it >>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf >>> has graciously encouraged me to add FreeBSD support directly to ZoL >>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single >>> shared code base. >>> >>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port >>> Before it can be committed some additional functionality needs to be >>> added to the FreeBSD opencrypto framework. These can be found at >>> https://reviews.freebsd.org/D18520 >>> >>> This port will provide FreeBSD users with multi modifier protection, >>> project quotas, encrypted datasets, allocation classes, vectorized >>> raidz, vectorized checksums, and various command line improvements. >>> >>> Before ZoF can be merged back in to ZoL several steps need to be taken: >>> - Integrate FreeBSD support into ZoL CI >>> - Have most of the ZFS test suite passing >>> - Complete additional QA testing at iX >>> >>> We at iX Systems need to port ZoL's EC2 CI scripts to work with >>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. >>> Being integrated in to their CI will mean that, among other things, >>> most integration issues will be caught before a PR is merged upstream >>> rather than many months later when it is MFVed into FreeBSD. I=E2=80=99= m >>> hoping to submit the PR to ZoL some time in January. >>> >>> This port will make it easy for end users on a range of releases to >>> run the latest version of ZFS. Nonetheless, transitioning away from an >>> Illumos based ZFS is not likely to be entirely seamless. The >>> stakeholders I=E2=80=99ve spoken to all agree that this is the best pat= h >>> forward but some degree of effort needs to be made to accommodate >>> downstream consumers. The current plan is to import ZoF and unhook the >>> older Illumos based sources from the build on April 15th or two months >>> after iX systems QA deems ZoF stable - which ever comes later. The >>> Illumos based sources will be removed some time later - but well >>> before 13. This will give users a 3 month period during which both the >>> port and legacy Illumos based ZFS will be available to users. Pools >>> should interoperate between ZoF and legacy provided the user does not >>> enable any features available only in ZoF. We will try to accommodate >>> any downstream consumers in the event that they need that date pushed >>> back. We ask that any downstream consumers who are particularly >>> sensitive to changes start testing the port when it is formally >>> announced and report back any issues they have. I will do my best to >>> ensure that this message is communicated to all those who it may >>> concern. However, I can=E2=80=99t ensure that everyone reads these list= s. That >>> is the responsibility of -CURRENT users. >>> >>> -M >>> >> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> From owner-freebsd-fs@freebsd.org Wed Dec 19 23:17:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92A2D134D050; Wed, 19 Dec 2018 23:17:05 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 098AE76645; Wed, 19 Dec 2018 23:17:05 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id CA0DD6073; Wed, 19 Dec 2018 23:17:04 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-io1-f43.google.com with SMTP id p7so9335175iog.12; Wed, 19 Dec 2018 15:17:04 -0800 (PST) X-Gm-Message-State: AJcUukdtrOziwIgjDi250OnYQgqszR9CrLwDlSV4yc1vnyrGibLU4vMQ HFUYsrr3TjfX8MyF2p6dzK3mtK+eu3omhB7rvls= X-Google-Smtp-Source: ALg8bN46ZS+RqLXvrnex9vobP5Mrft7kW8Uzh/eHSZZcBAeWcp1BipVqM8aBGLPVOUqk9mUnTbHfPQQD5+7JIp635dY= X-Received: by 2002:a6b:e014:: with SMTP id z20mr4910iog.237.1545261424604; Wed, 19 Dec 2018 15:17:04 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Wed, 19 Dec 2018 15:16:53 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Steven Hartland Cc: freebsd-current , freebsd-fs X-Rspamd-Queue-Id: 098AE76645 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.964,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Dec 2018 23:17:05 -0000 On Wed, Dec 19, 2018 at 15:11 Steven Hartland wrote: > Sorry been off for a few weeks so must have missed that, please do prod m= e > on again if you don=E2=80=99t see any response to anything not just this.= Like many > others I get so may emails across so many lists it=E2=80=99s more than li= kely I > just missed it. > > That said would you say that with the right support we can make progress > on the this prior to the port? I have to ask as the alternative version h= as > been on the cusp for many years now so it=E2=80=99s feels more like a dis= tant > memory than something that may happen, no disrespect to anyone involved, = as > I know all too well how hard it can be to get something like this over th= e > line, especially when people have competing priorities. > I am hoping that it's sufficiently important to FreeBSD ZFS developers that they'll give the PR the attention it needs so that it can be merged before summer. My understanding is that it's mostly suffered from neglect. TRIM is most important to FreeBSD and it already had its own implementation. https://github.com/zfsonlinux/zfs/pull/5925 I forwarded you the private communication again as well. -M > > On Wed, 19 Dec 2018 at 22:52, Matthew Macy wrote: > >> >> >> On Wed, Dec 19, 2018 at 14:47 Steven Hartland >> wrote: >> >>> Thanks for the write up most appreciated. One of the more meaty >>> differences is that FreeBSD ZFS still has the only merged and productio= n >>> ready TRIM support so my question would be are their any plans to addre= ss >>> this before creating the new port as going back to a world without TRIM >>> support wouldn=E2=80=99t be something I=E2=80=99d look forward to. >>> >> >> Well, then please follow up on the request I CC'd you on a week ago >> asking that you engage on the deadlist based TRIM PR. That's a better >> forum for discussing these details than lamenting in public lists. >> >> Thanks. >> >> -M >> >> >> >> >>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy wrote: >>> >>>> The sources for FreeBSD's ZFS support are currently taken directly >>>> from Illumos with local ifdefs to support the peculiarities of FreeBSD >>>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD >>>> has regularly pulled changes from Illumos and tried to push back any >>>> bug fixes and new features done in the context of FreeBSD. In the past >>>> few years the vast majority of new development in ZFS has taken place >>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced >>>> that they will be moving to ZoL >>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means >>>> that there will be little to no net new development of Illumos. While >>>> working through the git history of ZoL I have also discovered that >>>> many races and locking bugs have been fixed in ZoL and never made it >>>> back to Illumos and thus FreeBSD. This state of affairs has led to a >>>> general agreement among the stakeholders that I have spoken to that it >>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf >>>> has graciously encouraged me to add FreeBSD support directly to ZoL >>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single >>>> shared code base. >>>> >>>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port >>>> Before it can be committed some additional functionality needs to be >>>> added to the FreeBSD opencrypto framework. These can be found at >>>> https://reviews.freebsd.org/D18520 >>>> >>>> This port will provide FreeBSD users with multi modifier protection, >>>> project quotas, encrypted datasets, allocation classes, vectorized >>>> raidz, vectorized checksums, and various command line improvements. >>>> >>>> Before ZoF can be merged back in to ZoL several steps need to be taken= : >>>> - Integrate FreeBSD support into ZoL CI >>>> - Have most of the ZFS test suite passing >>>> - Complete additional QA testing at iX >>>> >>>> We at iX Systems need to port ZoL's EC2 CI scripts to work with >>>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes. >>>> Being integrated in to their CI will mean that, among other things, >>>> most integration issues will be caught before a PR is merged upstream >>>> rather than many months later when it is MFVed into FreeBSD. I=E2=80= =99m >>>> hoping to submit the PR to ZoL some time in January. >>>> >>>> This port will make it easy for end users on a range of releases to >>>> run the latest version of ZFS. Nonetheless, transitioning away from an >>>> Illumos based ZFS is not likely to be entirely seamless. The >>>> stakeholders I=E2=80=99ve spoken to all agree that this is the best pa= th >>>> forward but some degree of effort needs to be made to accommodate >>>> downstream consumers. The current plan is to import ZoF and unhook the >>>> older Illumos based sources from the build on April 15th or two months >>>> after iX systems QA deems ZoF stable - which ever comes later. The >>>> Illumos based sources will be removed some time later - but well >>>> before 13. This will give users a 3 month period during which both the >>>> port and legacy Illumos based ZFS will be available to users. Pools >>>> should interoperate between ZoF and legacy provided the user does not >>>> enable any features available only in ZoF. We will try to accommodate >>>> any downstream consumers in the event that they need that date pushed >>>> back. We ask that any downstream consumers who are particularly >>>> sensitive to changes start testing the port when it is formally >>>> announced and report back any issues they have. I will do my best to >>>> ensure that this message is communicated to all those who it may >>>> concern. However, I can=E2=80=99t ensure that everyone reads these lis= ts. That >>>> is the responsibility of -CURRENT users. >>>> >>>> -M >>>> >>> _______________________________________________ >>>> freebsd-fs@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>>> >>> From owner-freebsd-fs@freebsd.org Thu Dec 20 11:04:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63E3B1340F3C; Thu, 20 Dec 2018 11:04:20 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9BD3172E3E; Thu, 20 Dec 2018 11:04:09 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from [194.32.164.27] ([194.32.164.27]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id wBKB40so076885; Thu, 20 Dec 2018 11:04:00 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: The future of ZFS in FreeBSD From: Bob Bishop In-Reply-To: Date: Thu, 20 Dec 2018 11:03:59 +0000 Cc: Steven Hartland , freebsd-current , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Matthew Macy X-Mailer: Apple Mail (2.3273) X-Rspamd-Queue-Id: 9BD3172E3E X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rb@gid.co.uk designates 194.32.164.250 as permitted sender) smtp.mailfrom=rb@gid.co.uk X-Spamd-Result: default: False [0.17 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.35)[-0.354,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[gid.co.uk]; NEURAL_SPAM_SHORT(0.28)[0.282,0]; NEURAL_HAM_LONG(-0.12)[-0.121,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[250.164.32.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.34)[asn: 42831(-1.58), country: GB(-0.10)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42831, ipnet:194.32.164.0/24, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 11:04:20 -0000 Hi, > On 19 Dec 2018, at 23:16, Matthew Macy wrote: >=20 > On Wed, Dec 19, 2018 at 15:11 Steven Hartland = > wrote: >=20 >> Sorry been off for a few weeks so must have missed that, please do = prod me >> on again if you don=E2=80=99t see any response to anything not just = this. Like many >> others I get so may emails across so many lists it=E2=80=99s more = than likely I >> just missed it. >>=20 >> That said would you say that with the right support we can make = progress >> on the this prior to the port? I have to ask as the alternative = version has >> been on the cusp for many years now so it=E2=80=99s feels more like a = distant >> memory than something that may happen, no disrespect to anyone = involved, as >> I know all too well how hard it can be to get something like this = over the >> line, especially when people have competing priorities. >>=20 >=20 > I am hoping that it's sufficiently important to FreeBSD ZFS developers = that > they'll give the PR the attention it needs so that it can be merged = before > summer. My understanding is that it's mostly suffered from neglect. = TRIM is > most important to FreeBSD and it already had its own implementation. >=20 > https://github.com/zfsonlinux/zfs/pull/5925 Please correct me if I=E2=80=99m wrong but this looks a lot less mature = than FreeBSD=E2=80=99s existing TRIM support for ZFS which we=E2=80=99ve = had in production for six years. What is the rationale here? I=E2=80=99m concerned that it looks like an = opportunity for mighty regressions. > I forwarded you the private communication again as well. >=20 > -M >=20 >=20 >>=20 >> On Wed, 19 Dec 2018 at 22:52, Matthew Macy wrote: >>=20 >>>=20 >>>=20 >>> On Wed, Dec 19, 2018 at 14:47 Steven Hartland = >>> wrote: >>>=20 >>>> Thanks for the write up most appreciated. One of the more meaty >>>> differences is that FreeBSD ZFS still has the only merged and = production >>>> ready TRIM support so my question would be are their any plans to = address >>>> this before creating the new port as going back to a world without = TRIM >>>> support wouldn=E2=80=99t be something I=E2=80=99d look forward to. >>>>=20 >>>=20 >>> Well, then please follow up on the request I CC'd you on a week ago >>> asking that you engage on the deadlist based TRIM PR. That's a = better >>> forum for discussing these details than lamenting in public lists. >>>=20 >>> Thanks. >>>=20 >>> -M >>>=20 >>>=20 >>>=20 >>>=20 >>>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy = wrote: >>>>=20 >>>>> The sources for FreeBSD's ZFS support are currently taken directly >>>>> from Illumos with local ifdefs to support the peculiarities of = FreeBSD >>>>> where the Solaris Portability Layer (SPL) shims fall short. = FreeBSD >>>>> has regularly pulled changes from Illumos and tried to push back = any >>>>> bug fixes and new features done in the context of FreeBSD. In the = past >>>>> few years the vast majority of new development in ZFS has taken = place >>>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix = announced >>>>> that they will be moving to ZoL >>>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift = means >>>>> that there will be little to no net new development of Illumos. = While >>>>> working through the git history of ZoL I have also discovered that >>>>> many races and locking bugs have been fixed in ZoL and never made = it >>>>> back to Illumos and thus FreeBSD. This state of affairs has led to = a >>>>> general agreement among the stakeholders that I have spoken to = that it >>>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf >>>>> has graciously encouraged me to add FreeBSD support directly to = ZoL >>>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a = single >>>>> shared code base. >>>>>=20 >>>>> A port for ZoF can be found at = https://github.com/miwi-fbsd/zof-port >>>>> Before it can be committed some additional functionality needs to = be >>>>> added to the FreeBSD opencrypto framework. These can be found at >>>>> https://reviews.freebsd.org/D18520 >>>>>=20 >>>>> This port will provide FreeBSD users with multi modifier = protection, >>>>> project quotas, encrypted datasets, allocation classes, vectorized >>>>> raidz, vectorized checksums, and various command line = improvements. >>>>>=20 >>>>> Before ZoF can be merged back in to ZoL several steps need to be = taken: >>>>> - Integrate FreeBSD support into ZoL CI >>>>> - Have most of the ZFS test suite passing >>>>> - Complete additional QA testing at iX >>>>>=20 >>>>> We at iX Systems need to port ZoL's EC2 CI scripts to work with >>>>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) = passes. >>>>> Being integrated in to their CI will mean that, among other = things, >>>>> most integration issues will be caught before a PR is merged = upstream >>>>> rather than many months later when it is MFVed into FreeBSD. I=E2=80= =99m >>>>> hoping to submit the PR to ZoL some time in January. >>>>>=20 >>>>> This port will make it easy for end users on a range of releases = to >>>>> run the latest version of ZFS. Nonetheless, transitioning away = from an >>>>> Illumos based ZFS is not likely to be entirely seamless. The >>>>> stakeholders I=E2=80=99ve spoken to all agree that this is the = best path >>>>> forward but some degree of effort needs to be made to accommodate >>>>> downstream consumers. The current plan is to import ZoF and unhook = the >>>>> older Illumos based sources from the build on April 15th or two = months >>>>> after iX systems QA deems ZoF stable - which ever comes later. The >>>>> Illumos based sources will be removed some time later - but well >>>>> before 13. This will give users a 3 month period during which both = the >>>>> port and legacy Illumos based ZFS will be available to users. = Pools >>>>> should interoperate between ZoF and legacy provided the user does = not >>>>> enable any features available only in ZoF. We will try to = accommodate >>>>> any downstream consumers in the event that they need that date = pushed >>>>> back. We ask that any downstream consumers who are particularly >>>>> sensitive to changes start testing the port when it is formally >>>>> announced and report back any issues they have. I will do my best = to >>>>> ensure that this message is communicated to all those who it may >>>>> concern. However, I can=E2=80=99t ensure that everyone reads these = lists. That >>>>> is the responsibility of -CURRENT users. >>>>>=20 >>>>> -M >>>>>=20 >>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to = "freebsd-fs-unsubscribe@freebsd.org" >>>>>=20 >>>>=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to = "freebsd-current-unsubscribe@freebsd.org" -- Bob Bishop rb@gid.co.uk From owner-freebsd-fs@freebsd.org Thu Dec 20 11:57:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABF3313431C8 for ; Thu, 20 Dec 2018 11:57:56 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD768753A6 for ; Thu, 20 Dec 2018 11:57:55 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x542.google.com with SMTP id g22so1488653edr.7 for ; Thu, 20 Dec 2018 03:57:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=q/IVkfXVjGSGbH41787peBsjZ0SwQ15XqaAT+t9Yxyk=; b=CDKdmYIleCmOl0Bw7gqat9RtOTmJN1p34fnU86dIpC4cd1c/QMzNj+faDWKoms98AM EwUif9SWBooEyuzH5ZAoQhyWnLYnXWDpD+lEnkAnD9/UyuKp9ngRKrO7XPVhILG3ZOXM FOpEgo+iQlKQvOeZjqAf87vIacEQoIbJOMVOAzeDq62PDAv3HYD0/IFY4kCOnzZQU7PS ssjnm//Pi2/xr9i/pp8r+ktrA0qP3RFLQ0uiK+EBxBHYCQqHhJOSBR5DsO5AgEYk/C8F 3Yhwt+BfsOSsHnCChCKxTsM1FojjhPNpiCasRI7c6MzG+0WYULaD6BSMJEcS6DnsxiX2 Lasg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=q/IVkfXVjGSGbH41787peBsjZ0SwQ15XqaAT+t9Yxyk=; b=CPsgU4LfIm+IULDCraAvaCyNc9m+yOLcu9UxDTW1zSReH4JubMteB7N6QtXKI7n21x /rOqTn6DQhogb04JAuoK7xZTY4VE45D6pDgtETQxTsfMxGkd6jTmWM+Wuje2p/2wpteL f4mjjr340Ege5uDaqZwV5h09xR1YeWSCCdeICFIWPDHWslAmAOGFybuQRDgstHjnAjuK euV1W67oKT6nm8sgnona2LO1iRECfRsHcIR5TlHgjJXeN1xYaGF9f8TZ6CWNofjkmQlW mExE2e2+acR0eqahW+SQQKDDsFR94afeLO878hhbdzEOM1bO529Q8uCjBW3am9wl18wu MHzA== X-Gm-Message-State: AA+aEWahkCDS4/qsBzNILDeVG6yy2TDrtNf/gFbjv2a/nUGSnQWowlrb 2P67Vf2eCCPcDEeE5K9o9Q5KIN9f5/g= X-Google-Smtp-Source: AFSGD/XJEKnHDAwbrVuDFaUHxuDtptT/+HjSRsGJIgzXvcBPLoL8SsMEIWsZ1ktbqy6QcfrMDDSWcQ== X-Received: by 2002:a17:906:55a:: with SMTP id k26-v6mr19490110eja.218.1545307074354; Thu, 20 Dec 2018 03:57:54 -0800 (PST) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id e35sm6191762eda.13.2018.12.20.03.57.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 20 Dec 2018 03:57:53 -0800 (PST) Subject: Re: The future of ZFS in FreeBSD To: Bob Bishop , Matthew Macy Cc: freebsd-current , freebsd-fs References: From: Steven Hartland Message-ID: Date: Thu, 20 Dec 2018 11:58:03 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: CD768753A6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=CDKdmYIl; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::542 as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-4.04 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; RCVD_IN_DNSWL_NONE(0.00)[2.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.97)[-0.974,0]; IP_SCORE(-0.55)[ip: (0.34), ipnet: 2a00:1450::/32(-1.61), asn: 15169(-1.41), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 11:57:56 -0000 On 20/12/2018 11:03, Bob Bishop wrote: > Hi, > >> On 19 Dec 2018, at 23:16, Matthew Macy wrote: >> >> On Wed, Dec 19, 2018 at 15:11 Steven Hartland >> wrote: >> >>> Sorry been off for a few weeks so must have missed that, please do prod me >>> on again if you don’t see any response to anything not just this. Like many >>> others I get so may emails across so many lists it’s more than likely I >>> just missed it. >>> >>> That said would you say that with the right support we can make progress >>> on the this prior to the port? I have to ask as the alternative version has >>> been on the cusp for many years now so it’s feels more like a distant >>> memory than something that may happen, no disrespect to anyone involved, as >>> I know all too well how hard it can be to get something like this over the >>> line, especially when people have competing priorities. >>> >> I am hoping that it's sufficiently important to FreeBSD ZFS developers that >> they'll give the PR the attention it needs so that it can be merged before >> summer. My understanding is that it's mostly suffered from neglect. TRIM is >> most important to FreeBSD and it already had its own implementation. >> >> https://github.com/zfsonlinux/zfs/pull/5925 > Please correct me if I’m wrong but this looks a lot less mature than FreeBSD’s existing TRIM support for ZFS which we’ve had in production for six years. > > What is the rationale here? I’m concerned that it looks like an opportunity for mighty regressions. > This is the case, but overall this solution is thought to be a better approach. With anything like this there is always a risk, so we all need a concerted effort to get to one solution.     Regards     Steve From owner-freebsd-fs@freebsd.org Thu Dec 20 13:04:58 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4B52513466D4; Thu, 20 Dec 2018 13:04:58 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0DC577E65; Thu, 20 Dec 2018 13:04:47 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from [194.32.164.27] ([194.32.164.27]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id wBKD4jLj093718; Thu, 20 Dec 2018 13:04:45 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: The future of ZFS in FreeBSD From: rb@gid.co.uk In-Reply-To: Date: Thu, 20 Dec 2018 13:04:45 +0000 Cc: Matthew Macy , freebsd-current , freebsd-fs Content-Transfer-Encoding: quoted-printable Message-Id: <2A394CBF-7739-4F64-B559-BBF513EC141B@gid.co.uk> References: To: Steven Hartland X-Mailer: Apple Mail (2.3273) X-Rspamd-Queue-Id: A0DC577E65 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rb@gid.co.uk designates 194.32.164.250 as permitted sender) smtp.mailfrom=rb@gid.co.uk X-Spamd-Result: default: False [0.56 / 15.00]; ARC_NA(0.00)[]; MX_INVALID(0.50)[greylisted]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+mx]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[gid.co.uk]; NEURAL_SPAM_MEDIUM(0.07)[0.073,0]; NEURAL_SPAM_SHORT(0.17)[0.174,0]; NEURAL_HAM_LONG(-0.07)[-0.072,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NO_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[250.164.32.194.list.dnswl.org : 127.0.10.0]; IP_SCORE(-0.31)[asn: 42831(-1.48), country: GB(-0.10)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42831, ipnet:194.32.164.0/24, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 13:04:58 -0000 > On 20 Dec 2018, at 11:58, Steven Hartland = wrote: >=20 >=20 >=20 > On 20/12/2018 11:03, Bob Bishop wrote: >> Hi, >>=20 >>> On 19 Dec 2018, at 23:16, Matthew Macy wrote: >>>=20 >>> On Wed, Dec 19, 2018 at 15:11 Steven Hartland = >>> wrote: >>>=20 >>>> Sorry been off for a few weeks so must have missed that, please do = prod me >>>> on again if you don=E2=80=99t see any response to anything not just = this. Like many >>>> others I get so may emails across so many lists it=E2=80=99s more = than likely I >>>> just missed it. >>>>=20 >>>> That said would you say that with the right support we can make = progress >>>> on the this prior to the port? I have to ask as the alternative = version has >>>> been on the cusp for many years now so it=E2=80=99s feels more like = a distant >>>> memory than something that may happen, no disrespect to anyone = involved, as >>>> I know all too well how hard it can be to get something like this = over the >>>> line, especially when people have competing priorities. >>>>=20 >>> I am hoping that it's sufficiently important to FreeBSD ZFS = developers that >>> they'll give the PR the attention it needs so that it can be merged = before >>> summer. My understanding is that it's mostly suffered from neglect. = TRIM is >>> most important to FreeBSD and it already had its own implementation. >>>=20 >>> https://github.com/zfsonlinux/zfs/pull/5925 >> Please correct me if I=E2=80=99m wrong but this looks a lot less = mature than FreeBSD=E2=80=99s existing TRIM support for ZFS which = we=E2=80=99ve had in production for six years. >>=20 >> What is the rationale here? I=E2=80=99m concerned that it looks like = an opportunity for mighty regressions. >>=20 > This is the case, but overall this solution is thought to be a better = approach. >=20 > With anything like this there is always a risk, so we all need a = concerted effort to get to one solution. Not sure what I can contribute, but I can certainly put a box up for = testing when there=E2=80=99s something to test. > Regards > Steve >=20 -- Bob Bishop rb@gid.co.uk From owner-freebsd-fs@freebsd.org Thu Dec 20 15:32:50 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E50E0134AC09 for ; Thu, 20 Dec 2018 15:32:49 +0000 (UTC) (envelope-from freebsd-fs=freebsd.org@mail224.pae1.iasrv.net) Received: from mail224.pae1.iasrv.net (mail224.pae1.iasrv.net [177.85.221.224]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7814A857A5 for ; Thu, 20 Dec 2018 15:32:46 +0000 (UTC) (envelope-from freebsd-fs=freebsd.org@mail224.pae1.iasrv.net) Received: from mail224.pae1.iasrv.net (www.iagentemail.com.br [168.196.188.33]) by mail224.pae1.iasrv.net (Postfix) with ESMTP id C4DF323FB4F1A for ; Thu, 20 Dec 2018 12:25:56 -0200 (BRST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=mail224.pae1.iasrv.net; s=k1; t=1545315958; i=contato=3Dtrindadeimoveis.com.br@mail224.pae1.iasrv.net; bh=rN/latj4B+4DVaFE1zpfuAMMLpQ=; h=Date:To:From:Reply-to:Subject:Message-ID:List-Unsubscribe; b=YlcqzoVwUOVeZ53ag3IpgdvjYn/WWmSxG5oJxSPW+Z3DxnMvQprZwbVGhbgFlBOCb yp9tAIlzdT5CVwEpDbELz2J/0Awfp2NWCJvc8coiICc6rTdFQRHIxCkiS5nigVcAjH SAnFppc3YEJNSkoapOh301X+TgvMhyagCem4oVTU= Date: Thu, 20 Dec 2018 13:28:15 -0200 To: Freebsd-fs From: =?iso-8859-1?Q?Trindade_Im=F3veis?= Reply-to: =?iso-8859-1?Q?Trindade_Im=F3veis?= Subject: =?iso-8859-1?Q?Boas_Festas!_(C=F3pia)?= Message-ID: <1fad8015cf66a26c370650d9e8339547@mail224.pae1.iasrv.net> X-Priority: 3 X-ClienteID: 3513 X-AgendamentoID: 408638 X-CadastroID: 202922759 X-Destinatario: freebsd-fs@freebsd.org X-Mailer: IAMAIL-3513 X-P: MTc3Ljg1LjIyMS4yMjQ= Precedence: bulk MIME-Version: 1.0 X-Rspamd-Queue-Id: 7814A857A5 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail224.pae1.iasrv.net header.s=k1 header.b=YlcqzoVw; spf=pass (mx1.freebsd.org: domain of freebsd-fs=freebsd.org@mail224.pae1.iasrv.net designates 177.85.221.224 as permitted sender) smtp.mailfrom=freebsd-fs=freebsd.org@mail224.pae1.iasrv.net X-Spamd-Result: default: False [3.95 / 15.00]; HAS_REPLYTO(0.00)[contato@trindadeimoveis.com.br]; R_SPF_ALLOW(-0.20)[+ip4:177.85.221.224]; TO_DN_ALL(0.00)[]; SUBJECT_HAS_EXCLAIM(0.00)[]; DKIM_TRACE(0.00)[mail224.pae1.iasrv.net:+]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[mx.iasrv.net]; FORGED_SENDER(0.00)[contato@trindadeimoveis.com.br,freebsd-fs=freebsd.org@mail224.pae1.iasrv.net]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:52848, ipnet:177.85.208.0/20, country:BR]; FROM_NEQ_ENVFROM(0.00)[contato@trindadeimoveis.com.br,freebsd-fs=freebsd.org@mail224.pae1.iasrv.net]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[mail224.pae1.iasrv.net:s=k1]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PRECEDENCE_BULK(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; HAS_PHPMAILER_SIG(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; HAS_LIST_UNSUB(-0.01)[]; RCPT_COUNT_ONE(0.00)[1]; HTML_SHORT_LINK_IMG_1(2.00)[]; NEURAL_SPAM_MEDIUM(0.83)[0.828,0]; DMARC_NA(0.00)[trindadeimoveis.com.br]; FORGED_SENDER_VERP_SRS(0.00)[]; NEURAL_SPAM_LONG(0.81)[0.809,0]; ENVFROM_VERP(0.00)[]; IP_SCORE(0.01)[country: BR(0.06)]; NEURAL_SPAM_SHORT(0.82)[0.817,0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset = "iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 15:32:50 -0000 Esta mensagem foi enviada para o endereco freebsd-fs@freebsd.org. Se voce estiver com problemas para visualizar esta mensagem, copie o endereco abaixo e cole no seu navegador. http://e.trindadeimoveis.com.br/v3/TRC/P/3513/408638/202922759/ Para garantir o recebimento desta mensagem em sua caixa de entrada, = adicione contato@trindadeimoveis.com.br a lista de remetentes confiaveis ou ao seu catalogo de enderecos. Caso voce queira cancelar o recebimento das mensagens e/ou denunciar abuso = deste remetente, solicitamos que copie o endereco abaixo e cole no seu navegador. http://e.trindadeimoveis.com.br/v3/TRC/R/3513/408638/202922759/ From owner-freebsd-fs@freebsd.org Thu Dec 20 17:33:05 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C3FA134DFC9 for ; Thu, 20 Dec 2018 17:33:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5E16C8A53B for ; Thu, 20 Dec 2018 17:33:04 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x729.google.com with SMTP id w204so1478023qka.2 for ; Thu, 20 Dec 2018 09:33:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fdLQinfKSdDl0dO1eawoSHXN+APGk5nRCkgu6oZ/3Lw=; b=GJYPs4+E3uZoMJ3KToT4vAfSAukXNV/7UUnpPceZ65XYWqYYn645cTFe6ZfnXoEROS ikL86MJlgDh/UmwDmJqItZEVG7C+uPDjQTKQFORGZgfpXCAoi1J0+49/fwDZfcGLDNE1 Mw9Ji/rTILeLLHL6/NLovcXKSOCquQ+JOxWl8CtPaBEtwuTZSJTKQY6H+gO0fS/pL47V 2bX0FvG5JlK5SSWRyHhN+LXZHcm/9BDxlrMQKSKNmK/JAMmLKPInX/DZvrlR6r1Wlspk PBxyfwjO+GmF66muH9DqFZhJjj7ZHdD0c4P1IDg0bc+/z3err+Y/eAT8LchASGyCARuO DYwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fdLQinfKSdDl0dO1eawoSHXN+APGk5nRCkgu6oZ/3Lw=; b=DM3mVLdyPf80cAhxSidtvZCEDgRYMiXXGhQ+PXxKDnl9w9qSSQ+HhbKBGfwUD3O+0X TK1Ytbn2ilmvJ6Y8C0030HjL6UunvfomDv7HlVaxbRpTR2CJz0pMcffp5MS+fHv1DOA1 ZwXAKsrqs2Wtn0OueK8xgukNmkDPTdqkQGVsl8frA/nFlWyG3t4GZGO/EA1pZbA688kF Xa4MWlldkZe2dB1UJKSRPkZ2jCxPlDzb5idvu5sQSUDOK34mmKKnMqtrmIHyPD0k6N58 FJY9NjdEfZMoxzU/+wFr5bI9o6aPQa57Js4iGJZj/iDSujP7E0Nh2TkJchxZ6bksfkdJ vLgA== X-Gm-Message-State: AA+aEWaol9yLl+LzIAIHZ9VoIjZ+0VgNUE/LBkWoCgp5tD3iTIJMMaK7 Hw9RxgYNGUJnbRswO9do8ZyGMNhpNIOkVrTrGVO6/w== X-Google-Smtp-Source: AFSGD/W4U/MVTuvRUh6HwJWMwJ3gk455a6nC1OzIGJetLH1qPhDG4KNiILTouM3HqiWBsxQnKSZp9H4+WLOxuMnnSrk= X-Received: by 2002:a37:6e86:: with SMTP id j128mr26410675qkc.46.1545327183717; Thu, 20 Dec 2018 09:33:03 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Thu, 20 Dec 2018 10:32:52 -0700 Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Matthew Macy Cc: freebsd-fs , freebsd-current X-Rspamd-Queue-Id: 5E16C8A53B X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=GJYPs4+E X-Spamd-Result: default: False [-4.84 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; RCVD_IN_DNSWL_NONE(0.00)[9.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.94)[-0.942,0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-1.89)[ip: (-6.15), ipnet: 2607:f8b0::/32(-1.78), asn: 15169(-1.41), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 17:33:05 -0000 Matt, This is a fairly comprehensive plan. Kudos for putting it together. The big question here is do you have a complete list of FreeBSD-specific changes that will be lost in the cut-over? We've heard about TRIM support and maybe NFSv4, but are there others that can be identified? Once you have that list, it wouldn't be hard to throw the initial email with some tweaks from the replies into a FCP so everybody knows the plan, and we have it ratified in case people come along later and 'forget'. Warner From owner-freebsd-fs@freebsd.org Thu Dec 20 19:58:56 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1295E13524E2; Thu, 20 Dec 2018 19:58:56 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ABECB6ADC6; Thu, 20 Dec 2018 19:58:55 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: from mail-it1-f176.google.com (mail-it1-f176.google.com [209.85.166.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) (Authenticated sender: mmacy) by smtp.freebsd.org (Postfix) with ESMTPSA id 80C94E799; Thu, 20 Dec 2018 19:58:55 +0000 (UTC) (envelope-from mmacy@freebsd.org) Received: by mail-it1-f176.google.com with SMTP id g85so3807078ita.3; Thu, 20 Dec 2018 11:58:55 -0800 (PST) X-Gm-Message-State: AA+aEWZryGy8f7KjeQhhQgVskCY9ywn6Hcb0uBwcZREoc6U2nf15BO9O FR1rd5eT95gKg8Go4bNd01POe9gwrLdcJz6Efgs= X-Google-Smtp-Source: ALg8bN4O7b0j6vR1+AkhII0UFSXDVmhFTywoC06WTavp+8HhZ7fr4ThuoFHI04anvt+Qm2vcH8Uehi/NTDmTA5JDIf4= X-Received: by 2002:a24:9005:: with SMTP id x5mr45809itd.102.1545335934972; Thu, 20 Dec 2018 11:58:54 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Matthew Macy Date: Thu, 20 Dec 2018 11:58:43 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: The future of ZFS in FreeBSD To: Warner Losh Cc: freebsd-fs , freebsd-current Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: ABECB6ADC6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.97)[-0.965,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 19:58:56 -0000 On Thu, Dec 20, 2018 at 9:33 AM Warner Losh wrote: > > > Matt, > > This is a fairly comprehensive plan. Kudos for putting it together. > > The big question here is do you have a complete list of FreeBSD-specific changes that will be lost in the cut-over? We've heard about TRIM support and maybe NFSv4, but are there others that can be identified? > > Once you have that list, it wouldn't be hard to throw the initial email with some tweaks from the replies into a FCP so everybody knows the plan, and we have it ratified in case people come along later and 'forget'. The general intent is that we not lose anything. TRIM is the only piece of code that really needs resolution upstream causing us to lose it temporarily. Failure to upstream FreeBSD's TRIM support has been a recurring source of bugs when integrating new features. As for NFSv4 I've forked the ACL bits in to the OS dependent code - I'll make a note to make sure that nothing regresses there. This is yet another example of why there will be a 3 month period for users to try the port before we cut over. -M From owner-freebsd-fs@freebsd.org Thu Dec 20 23:03:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75AE61357269 for ; Thu, 20 Dec 2018 23:03:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670067.outbound.protection.outlook.com [40.107.67.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 399AE727D0 for ; Thu, 20 Dec 2018 23:02:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM (10.169.142.146) by YQBPR01MB0497.CANPRD01.PROD.OUTLOOK.COM (10.169.143.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1446.18; Thu, 20 Dec 2018 23:02:58 +0000 Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c]) by YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c%8]) with mapi id 15.20.1446.022; Thu, 20 Dec 2018 23:02:58 +0000 From: Rick Macklem To: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Thread-Topic: Suggestion for hardware for ZFS fileserver Thread-Index: AQHUkzl4CLTPDjfxZEms4Ul8AJXDZ6WIQ5Vm Date: Thu, 20 Dec 2018 23:02:58 +0000 Message-ID: References: , <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> In-Reply-To: <2CB9CF77-DBC4-4452-8FC1-0A302884E71B@ifm.liu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR01MB0497; 6:hBGoSfpE+xJAC/NAuU6+Qt2GkWvV6ySDyX4v4XoHxleRvdkyYG9recJis3G09TcIrO3rwzrAaf5v7CvV2nJrQpWd87LhAT4xs+9/c7vQOO1E7/JpB3JsNF8r2H/nn40dtT7tUuigBI0E8lQB8g1ielsmHp3A3c4913caNnaW7tZlbIEXlk6PaY5BAaU3ZSbQ0G88UN7/dgVDR9DNb1pfu/aF3dpOxMY1GD6fOEK5pLYwrn03k3K/2rIKC077OrzJw151rPAvcKCf6sVAJ3aVo+8ZR5vbTXgwGMtfSiyXltLFkR0jJr0mlsjRLTEcfrv55KcmjNNd5Gb59Qr+CCYxv1gcTAmzgsm/G4kgol0PzEilRFNIwlqTkUpdteokbDD+sxseDJmTgJTyKFzk3fVk4vjUcSVCS6oNxP70Z6eSm9/h96DCIo/t8c07KpGVklwVK38wHcwhtnWFanbKM40UGA==; 5:6I05Xxlbr1CrBibwGy7OrYsd8dFZHXl4G53sqn+ceGvSAd4VaphQ+63RzyWLmqd0yVhvNuvZN5HxFyMWuCQ94yi/WIY96p7rjYUf+6BkgewYCbpfvRRcjhnozjJzmeaJJ8ZYZVhS1kS4hUevx4QhPHUa4j97Otpaya2K72bAseU=; 7:qHD5MhXxPkjeOk77+qUfDsvY6H+RUslhZ2X8SlzCjmAhiBJjem/JEw61RomfVijVBXkxlkgaGejTrHC5uyRwdJAYmMuDD/Prehb0OCxVdNQdd0+WUtWHdjoRHTZHKJdhKgEpKgdN2XftBGbRqWtIMA== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 854fc07a-f022-40d0-ca9a-08d666cf48b6 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YQBPR01MB0497; x-ms-traffictypediagnostic: YQBPR01MB0497: x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(3230021)(999002)(5005026)(6040522)(2401047)(8121501046)(93006095)(93001095)(3231475)(944501520)(52105112)(10201501046)(3002001)(149066)(150057)(6041310)(20161123562045)(20161123558120)(20161123564045)(20161123560045)(201703131423095)(201703031522075)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YQBPR01MB0497; BCL:0; PCL:0; RULEID:; SRVR:YQBPR01MB0497; x-forefront-prvs: 0892FA9A88 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(346002)(376002)(136003)(396003)(366004)(189003)(199004)(9686003)(25786009)(55016002)(106356001)(105586002)(6246003)(6436002)(316002)(786003)(296002)(8936002)(81156014)(81166006)(8676002)(256004)(11346002)(186003)(476003)(486006)(86362001)(71200400001)(7696005)(76176011)(71190400001)(446003)(74482002)(6506007)(53936002)(229853002)(102836004)(46003)(97736004)(5660300001)(74316002)(33656002)(2501003)(305945005)(14454004)(478600001)(2906002)(110136005)(99286004)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR01MB0497; H:YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: z5p1VClxJbeuWQ7eQQye1E2bU6BIBDSnibXmHepszaaP73KhaMx80/QG8K9JZBFkOHCjTskaAuYxw8fCoPCntnWh7ICXzr2J0FNtn+EUJ6tLMzTXTQ3jD6lGJZF40vIlfzHqBRFRUsVQx3I4eZdgF+Uwunztfqotg+yfYZWW3T3kKtBqcRXzpNidjynEBUOKKabOcXiT/tNAsacXCtht2SiKJd23dcebM88O/wL4ge1B2+L3u1bEp4SQp6gV9JxytO4lpNOeXrTiOKKg35jF9mEElYB+YRB1wx/KlzQjLYGq8XM0ReEYhUol/oZxyucL spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 854fc07a-f022-40d0-ca9a-08d666cf48b6 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2018 23:02:58.2675 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR01MB0497 X-Rspamd-Queue-Id: 399AE727D0 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.67 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.02 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[67.67.107.40.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.87)[-0.865,0]; IP_SCORE(-0.84)[ipnet: 40.64.0.0/10(-2.08), asn: 8075(-2.04), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Dec 2018 23:03:02 -0000 Peter Eriksson wrote: >I can give you the specs for the servers we use here for our FreeBSD-based= >fileservers - which have been working really well for us serving Home dir= ectors [good stuff snipped] >NFS (NFSv4 only, Kerberos/GSS authentication) > More or less the only thing we=92ve tuned for NFS so far is: > nfsuserd_flags=3D"-manage-gids -domain OURDOMAIN -usertimeout 10 -use= rmax >100000 16=94 > As more clients start using NFS I assume we will have to adjust other st= uff too.. >Suggestions are welcome :-) I am not the best person to suggest values for these tunables because I nev= e run an NFS server under heavy load, but at least I can mention possible val= ues. (I'll assume a 64bit arch with more than a few Gbytes of RAM that can be de= dicated to serving NFS.) For NFSv3 and NFSv4.0 clients: - The DRC (which improves correctness and not performance) is enabled for T= CP. (Some NFS server vendors only use the DRC for UDP.) This can result in si= gnificant CPU overheads and RPC RTT delays. You have two alternatives: 1 - set vfs.nfsd.cachetcp =3D 0 to disable use of the DRC for TCP. 2 - Increase vfs.nfsd.tcphighwater to something like 100000. You can also decrease vfs.nfsd.tcpcachetimeo, but that reduces the effectiveness of the DRC for TCP, since the timeout needs to be larg= er than the longest time it is likely for a client to take to do a TCP = reconnect and retry RPCs after a server crash or network partitioning. For NFSv4.1, you don't need to do the above, because it uses something ca= lled sessions instead of the DRC. For NFSv4.1 clients you will, however, want = to increase vfs.nfsd.sessionhashsize to something like 1000. For NFSv4.0 and NFSv4.1 clients, you will want to increase the state relate= d stuff to something like: vfs.nfsd.fhhashsize=3D10000 vfs.nfsd.statehashsize=3D100 vfs.nfsd.clienthashsize=3D1000 (or 1/10th of the number of client mounts up= to something like 10000) As you can see, it depends upon which NFS version your clients are using. ("nfsstat -m" should tell you that on both FreeBSD and Linux clients.) If your exported file systems are UFS, you might consider increasing your b= uffer cache size, but not for ZFS exports. Most/all of these need to be set in your /boot/loader.conf, since they need to be statically configured. vfs.nfsd.cachetcp can be cleared at any time, = I think? For your case of mostly non-NFS usage, it is hard to say if/when you want t= o do the above, but these changes probably won't hurt when you have 256Gbytes of RAM. Good luck with it, rick [more good stuff snipped] From owner-freebsd-fs@freebsd.org Fri Dec 21 03:18:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 744E2133AC9B for ; Fri, 21 Dec 2018 03:18:43 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: from mail-ua1-x92b.google.com (mail-ua1-x92b.google.com [IPv6:2607:f8b0:4864:20::92b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A7A79832B3 for ; Fri, 21 Dec 2018 03:18:42 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: by mail-ua1-x92b.google.com with SMTP id z11so1262014uaa.10 for ; Thu, 20 Dec 2018 19:18:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=h+jQvy10BbXjSnAsuXIacKJZIpM9PC7CttA6zJOghBI=; b=FjMPgr41MO2WPTDAdt/io5W9/y2ThVQoSEyHn3qYpyvoI6KyamiFAWKtc0d6uNYktr RFd2p8JT2I+rC0gaDFU8VbN7RqKc2ze6Fbb3UloPu5HEYkTI+M0cVfiJ38enCjiHZzAx ayChnTZaTgxfmo2vAm7sHANprzHupT2T32jQ7dIaAajRZuNsIA10R/n68i5u8HPygS5P wv9vsFAlDo6Btr7wZgt4JcbchhszwuNUWDgiB/gdrfFUUMpquXY6/i+r3wCxkBu4FTK1 qpK0SleqEgDvkppFk818AGy9+dIlrqp/1zUqcDhONp8hANpoOqm5G/iXK9Om7JqQTG9x Ohjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=h+jQvy10BbXjSnAsuXIacKJZIpM9PC7CttA6zJOghBI=; b=PqgdQQkt19oibHZZi7YnGP/Dn+TAEj43/PNwCJV2psuye8U3M1MIIsRBGBK2g8cOVA 1WSOiv7NrCa9MWwzPvv7pEfEXwey9seOGeNhPUzRPszbVC0LnqZb2+vJYKiOIKd3feGY p2+YMEOUYddcTMbflK00Yei4nDSKVUP5DNl0si9m7XEapbMQoPIxrfEE/jE7g6Foc0FO wi100YV3jVD2hf9RnB7ajw6n+Qsizq2cfdaCXCCSv86wyYgcyz5pJ0QHsUlQQ9QeXz9O LfeJHO3KA3nDaUp9YOfuQ9aLe94uWSpIX/pccI7Okt2i1HejpgluID1DO599Wb9F8C4K mK/w== X-Gm-Message-State: AJcUukfiqxTMAbg766llFGa9TDqJ1bxodrSyXevgtl/b5Ij/m4/o5z/K Mmz7B683sUFl4Mg53RBhT4mAcKvM9PSPL44WE3qTp6gw X-Google-Smtp-Source: ALg8bN79nT8QrWXnlmv5+AUzenQCae00awaalDrdD3uUPZR5lIh1iK0u6jiCLENiJIsfxW7Viu9LZrw/N9vMkO8vGN0= X-Received: by 2002:ab0:21c6:: with SMTP id u6mr290461uan.97.1545362321410; Thu, 20 Dec 2018 19:18:41 -0800 (PST) MIME-Version: 1.0 From: Stilez Stilezy Date: Fri, 21 Dec 2018 03:18:15 +0000 Message-ID: Subject: Dev help requested: short DTrace instrumentation script to narrow down cause of a longstanding ZFS write issue To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: A7A79832B3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=FjMPgr41; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of stilezy@gmail.com designates 2607:f8b0:4864:20::92b as permitted sender) smtp.mailfrom=stilezy@gmail.com X-Spamd-Result: default: False [-3.79 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.67)[ipnet: 2607:f8b0::/32(-1.82), asn: 15169(-1.43), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.11)[-0.114,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 03:18:43 -0000 I'm experiencing a persistent timeout problem that only occurs during network writes across a 10G LAN to a ZFS pool. Work to date confirms it's a real server-side issue, arising within the ZFS side not the client, network hardware, or FreeBSD's network I/O handling, and it involves ZFS mishandling dirty data free-up (there are bugs logged for this), or ZFS not correctly delaying too-fast write source. Of cours eit could also be mis-set sysctls, but I've replicated it with a default FreeNAS setup so it would have to be mis-configured in that as well, if so. To save time, can I skip the detailed debugging done to date, and just say that is where I've got to, so far. I have asked previously, long ago, and it was suggested at the time it could be a dirty data bug. With more experience under the belt, I'd like to dig deeper and try to narrow down the point which is bottlenecking, using dtrace. Specifically, I'd like help with a short instrumentation script to capture four key events within the ZFS write data pathway, in order to pin down which part of the |FS data flow is implicated in the bottleneck. I need help because it requires detailed knowledge of the ZFS codebase. I get the impression it could be a very short script - if I can narrow it down, maybe I can find a fix, or else, better evidence of the exact point where the bug/issue is arising. *Symptoms:* I'm using ZFS on enterprise hardware (Supermicro, Xeon E5-1620 v4, 128GB ECC, Intel P3700 NVMe SLOG/ZIL, 7200 mirrored enterprise SAS3 HDD pool, 9311 SAS3 HBA IT mode, Chelsio NICs, 10G fibre LAN, Netgear managed SFP+ switch). With a quiesced "known good" 10G network, and just a single SMB, iSCSI or netcat client (same happens on all 3 protocols), and that client doing a single 100GB file write, I can see the network RCV buffer gradually being driven down to zero without getting time to recover much, causing the TCP window to go down to << 80 bytes in tcpdump as it tries to cope, and eventually (when TCP window is driven to zero often enough) the file write session fails due to the persistant zero window condition. Jumps in the network RCV queue seems to coincide in time just before disk activity in the HDD pool, adding to evidence it's related to the ZFS write pathway. All the evidence suggests the network and client sides aren't the issue. What's happening is, ZFS is misconfigured or hitting a bug, it's not pulling new data into the next TXG in a timely manner, or delaying elegantly. File data is coming into the NIC RCV buffer at 10G speed, faster than ZFS is allowing it to be removed, and ZFS isn't effectively signalling the client to slow down or delay. *dtrace:* The most direct route to narrow it down, would involve instrumenting to log the timestamps, sizes, and time taken, for (I think!) 4 events. I don't know the exact names used for these calls, so I have to describe their effect. Help with identifying appropriate functions/instrumentation points, even if it's not clear how to instrument them, would be great! - There should be a function that gets called/polled when ZFS has space in its write pipeline, it checks to see if any ongoing write has a chunk of data to be written, and if so, pulls data from the source stream/pipe/descriptor (in this case the network RCV buffer, could be any source of an ongoing file write). This will let me instrument the events, when ZFS requests/gets source data from any ongoing file write, in this case pulling it from the network buffer. * To instrument: timestamp, max bytes to read, actual bytes read* - Within the ZFS code, the write pathway involves (simplistically) newly received write data and any related metadata/consequent updates being added to the current TXG and SLOG/ZIL, and ultimately, written out to the pool. *To instrument for those 3 events: timestamp, bytes written, write-out time taken. (Ideally also, dirty data cache size before/after)* Although the output file will be huge, it should be easy to analyse the timestamped list of events+dataflow and see which of the last 3 processes is causing the bottleneck. I would enjoy doing the work, and tracking this down somewhat, but need a little help with the instrumentation aspect which really needs developer competences. Can someone kind, please help with an appropriate short instrumentation script to capture the raw data for these 4 items :) (And any other data that in your experience, would be useful) Thank you! Stilez From owner-freebsd-fs@freebsd.org Fri Dec 21 10:33:22 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6A691344912 for ; Fri, 21 Dec 2018 10:33:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B51B1904F6 for ; Fri, 21 Dec 2018 10:33:21 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr1-x435.google.com with SMTP id l9so4675063wrt.13 for ; Fri, 21 Dec 2018 02:33:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=KIUwBMefgH01nstJXu0WQZbg6u+wWihgpQh0M2xWV/E=; b=UpBe67+5/gRA8W4dDCALvlaIvr0kVLqq7UIwMvPlCJVNB5+pSPNYRKaOw9nqr9qs17 0FfQ/xAhofyNtHGGQLgijPp5VYdjgdQ4S04xFsTNqkw0nWzA3ZnfnDaBR8JNdbzsu1/M eHek8megdnrU5dEmazNbXDswAHyRDaWU604V0tRhKkxj9+b2UOOkYc7LPHBzGqY6K/je fC3D1tLYKIyol00SHvBUhDRyTH7rdlbLFNNG/FN/n8/Ofb6VYVVKsHgV57TpF5jcAcu/ 5gbuEUJUUL5OLCt2SOo9qckdV+jUb4EJY+FJv2hvcD/u07M5hwdeYpt2FlPR1uJgtN6s lEKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=KIUwBMefgH01nstJXu0WQZbg6u+wWihgpQh0M2xWV/E=; b=p7aEgEWCB9j+ztZsZpm/m9RCxC/JTl/r8UBovuAPNYYBW8uNbIMQeSBI0bG7u32rg/ ZdTsEHW/HMnarZ1muRgV7PgMl93ZsN6LI5/40UN9kS405Z7n2Bm9BLairIYmC9caABG+ 4YKGfBnFvLnpCC6kuhoAYRl6oQk5kudD4MK7s89ojZEarxI/SBHcGGSp40heN89/cM7z PtBRxpzQSP15FetMGs51l46vkORxF87CaP4KvBy1jvfcJrxwWLkvW0YbPJQ7G/ZscpPX vFcsoUzex3vwwgQq03l5sEUnnxK+wYA9/g+GNqbmjCe4aJzGke6qrAFKVEH/m5kbQ+HY NWoA== X-Gm-Message-State: AJcUukchJ5XCsc1Ce+m90ZDZ59xEek2lDeb8TXJqhYJ0MT8sNsNUbcR/ 9La50y7b6+iA5QLCtG/PCfjuzSe/ X-Google-Smtp-Source: ALg8bN7Fsvo+sJ8kiAEOTxyuPkbk+Rn3C/l5DDHR8sMklsi37Y4uKgf9aoXrSdsMVyiF+cjJynnWHg== X-Received: by 2002:adf:fb4c:: with SMTP id c12mr1914874wrs.297.1545388400286; Fri, 21 Dec 2018 02:33:20 -0800 (PST) Received: from [192.168.0.138] (amontpellier-652-1-72-124.w109-210.abo.wanadoo.fr. [109.210.55.124]) by smtp.gmail.com with ESMTPSA id 124sm4051701wmh.22.2018.12.21.02.33.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Dec 2018 02:33:19 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Suggestion for hardware for ZFS fileserver From: Ben RUBSON In-Reply-To: Date: Fri, 21 Dec 2018 11:33:18 +0100 Cc: freebsd-fs@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: References: To: Sami Halabi X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: B51B1904F6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UpBe67+5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of benrubson@gmail.com designates 2a00:1450:4864:20::435 as permitted sender) smtp.mailfrom=benrubson@gmail.com X-Spamd-Result: default: False [-6.07 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[5.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.57)[ip: (-9.69), ipnet: 2a00:1450::/32(-1.64), asn: 15169(-1.43), country: US(-0.08)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 10:33:23 -0000 > On 13 Dec 2018, Sami Halabi wrote: > > 2. Hardware recommendations in terms of: > HBA SAS2008 / SAS3008 based card, with IT firmware. And SAS only disks (both HDD and SSD). One tunable to increase SASx008 log level, which could be useful : dev.mps.0.debug_level Ben From owner-freebsd-fs@freebsd.org Fri Dec 21 10:38:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE3AF1344B23 for ; Fri, 21 Dec 2018 10:38:59 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78D2C90833 for ; Fri, 21 Dec 2018 10:38:58 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wr1-x442.google.com with SMTP id t27so4713493wra.6 for ; Fri, 21 Dec 2018 02:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:date:references :to:in-reply-to:message-id; bh=0X99foft0nhWLSsSQTUPvajIPGwt3lVLQIBDNuKEG4Q=; b=HUnHeRKBBwyrenoWmHiqfvmms2ISlSIG1n+iwFdfnj2HsCOF8bkrzPry4HMH3fL/un Ak3eL3GxJ71LeP3GEkPNwFAkc4W4qQ/2A6A/WOTYpYskcHujGc4GRIT5HM69L9pXBUiY oF7mOLTPqLtv+OhKNLKL6p4OMcUWPbd9QU9UgYt5+Y54hE/IUbf4csFbK6Z/LONZ8o3z 3sBRkZ6jK6jLLh4m8fVEDadxWbJlqwJBxMtSIN32kplCyCJqHPiK/jXmxLi8KLN+nEWy wj5J01Fd4ykMRfTHAEVeFo2bB8rA0TQRIb6rUcP3u3/vUhfZVyPUB2O4qXjqF3RfzwIw xwAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:date:references:to:in-reply-to:message-id; bh=0X99foft0nhWLSsSQTUPvajIPGwt3lVLQIBDNuKEG4Q=; b=X3VFpair//ZUTBgkjvmugZQdEoX7TutmNMzxhuEI3gdnwgz9QEtASowbsK0/c6ltSt Ul4+d6IXr2jRcBqNBdPNTzGMHzq4vZJVSWCv6L3W2iPymQ/N7/aOYXxfQ4zZbMD4vsQ3 jhYfjYy3Jt8bPVIswU9xSEiV9Wtaok9tv/B+1su8znN4/lyJgw/WMbaSYBnsKmfJ/xaY PQvS4q1j0wXrJp2uJfLc7/DJ9KbM1x7ApUvRjc5BDVFFHJt98IUva2BW4l4cYC/EtF1q G4EbcunXR39qYThMeU86nqY6wIvepDjIUPv6jKA0u0+Zd5Br2dsBDXeTHGGTQbxZTJXZ 8JTQ== X-Gm-Message-State: AJcUukd54K0zjM3VGQwn+pIKEE4lMbldwbHumNJWTOZsfP4ZdjOW51xb dvholqNHvo9vt88cjwVXwfJwEnnI X-Google-Smtp-Source: ALg8bN41Aa2DHhqf2OhYiNjJb3p3O0H06yg640fGAcGp0/4K+IyUhMXC/B/PM7AOTGPq/P9txn7k+g== X-Received: by 2002:a5d:524b:: with SMTP id p11mr1934602wrv.147.1545388737111; Fri, 21 Dec 2018 02:38:57 -0800 (PST) Received: from [192.168.0.138] (amontpellier-652-1-72-124.w109-210.abo.wanadoo.fr. [109.210.55.124]) by smtp.gmail.com with ESMTPSA id k128sm13510657wmd.37.2018.12.21.02.38.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Dec 2018 02:38:56 -0800 (PST) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: [ZFS] Metadata Allocation Classes Date: Fri, 21 Dec 2018 11:38:55 +0100 References: <82E338C0-09E3-4648-B49B-C028A03B71D7@gmail.com> To: freebsd-fs@freebsd.org In-Reply-To: <82E338C0-09E3-4648-B49B-C028A03B71D7@gmail.com> Message-Id: X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 78D2C90833 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=HUnHeRKB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of benrubson@gmail.com designates 2a00:1450:4864:20::442 as permitted sender) smtp.mailfrom=benrubson@gmail.com X-Spamd-Result: default: False [-4.01 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.54)[ip: (0.44), ipnet: 2a00:1450::/32(-1.64), asn: 15169(-1.43), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[2.4.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 10:39:00 -0000 > On 22 Nov 2018, Ben RUBSON wrote: >=20 > Hi, >=20 > Intel has pushed the "Metadata Allocation Classes" feature to ZFS on = Linux. > This allows to store metadata on a so called "special" VDEV, made of = SSD for instance. >=20 > This looks really promising. > I wrote a patch 2 years ago which pins metadata in L2ARC, at eviction = time. > This really helps with some specific workloads. > But "Metadata Allocation Classes" should be one step further in terms = of performance. >=20 > Simple question, do you plan to backport this feature to FreeBSD ? > Or do you first wait for it to reach Illumos (does not seem to be = there yet) ? >=20 > ZoL discussion : https://github.com/zfsonlinux/zfs/issues/3779 > ZoL PR : https://github.com/zfsonlinux/zfs/pull/5182 I answer to myself, well "The future of ZFS in FreeBSD" topic in this = list does :-) Ben From owner-freebsd-fs@freebsd.org Fri Dec 21 13:19:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4024F1349520 for ; Fri, 21 Dec 2018 13:19:25 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: from mail-ua1-x942.google.com (mail-ua1-x942.google.com [IPv6:2607:f8b0:4864:20::942]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0C1595F31 for ; Fri, 21 Dec 2018 13:19:23 +0000 (UTC) (envelope-from stilezy@gmail.com) Received: by mail-ua1-x942.google.com with SMTP id d21so1660957uap.9 for ; Fri, 21 Dec 2018 05:19:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=SqSV2QEU7YU0EfwW3zVxtJMybWWoao1z2rH+DfD4uu4=; b=ArleT+p7iHk/1iHuGSykpRwJ568hPP7bUROSpNnZ0g/ityAmi0NQpDAR4pCcE9Ev1F vvUHtki/Q5B3p0JQ8cekJ/YHGPSpdSoMbxrgSNy00yGqZseHmI8l+5xAmpz83Uz4bbK/ Uimg9Oo5P6OKKfAbtBXOMockF25KwEFdIJ4znxxXDssvEkpt9dBquYw7fKAXRxex+0lF sTX3i7TjWLnhxoxn2qVWpXBoFmUWqLOS+b/cCEf2hu7U8zgzajDuIpl/9kuUKNTzA3vc g+PTMjgIA6SdU6n8YqRVo13DjKWE00L2STgXmoxvGC5yPPexfRPRXtB/F9jwz5mg5tfB KAmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=SqSV2QEU7YU0EfwW3zVxtJMybWWoao1z2rH+DfD4uu4=; b=eTc1EpClZ/po4xRhEa9B5MjqoIP61RAMjvnCg1Q/OQWTfdRKFmkoe//Kz9e1njDlnw PVXbaMIpLwdWOIB+mzTNeCSCsle0l1sQkhtrjIlQV9YXrD4/DdG2tU8jCUpcrqC4q1wu EbrHcPzj/TJP+pZgC8D9WN+8aVxpCy0SaAeJEo9iF4DxN+Q8wJxGrjqt4gblvwDyFfzJ tnIFrghkI+H7rBi6EcDxcOkU3wnJ2YmvQOMiy7F1OD1WJ8LWOrQDPo6392CXUsNWO6/8 /ilp9duuG/i0prbU0M+ms5z9yKL3Ng8zTWlSp4Kavrf/puYeHeMtD7tKmLNDFJAhShSq PE4w== X-Gm-Message-State: AJcUukfMkxBFf+U0EIrrxPaeVTxFUVaUn/vV+AsPJfF1yf67spmM5Ks4 wTrT4p7/W8gLMrmCRxxtxaftxu0nRetMNib62ZuwgLlW X-Google-Smtp-Source: ALg8bN5ovjJW0zNBCY1TTa65mgcSfT3VmfblNZnSYkWNHF9wAB4JlXBVLPVNcLR1nEuWRDKKsWUGWFlaujKWxiSkdS4= X-Received: by 2002:ab0:4121:: with SMTP id j30mr848221uad.65.1545398362532; Fri, 21 Dec 2018 05:19:22 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Stilez Stilezy Date: Fri, 21 Dec 2018 13:18:55 +0000 Message-ID: Subject: Re: Dev help requested: short DTrace instrumentation script to narrow down cause of a longstanding ZFS write issue To: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: D0C1595F31 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=ArleT+p7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of stilezy@gmail.com designates 2607:f8b0:4864:20::942 as permitted sender) smtp.mailfrom=stilezy@gmail.com X-Spamd-Result: default: False [-1.21 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.71)[-0.706,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.60)[-0.603,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(0.32)[ip: (4.94), ipnet: 2607:f8b0::/32(-1.82), asn: 15169(-1.44), country: US(-0.08)]; NEURAL_SPAM_SHORT(0.69)[0.692,0]; RCVD_IN_DNSWL_NONE(0.00)[2.4.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 13:19:25 -0000 To add some more data about this issue. I just tried the suggestions offered by Delphix, for write throttle problems (specifically the last part, about when ZFS genuinely can't keep up). I used the same setup as above (same hardware, clean FreeNAS 11.2, single SMB client writing a single 100GB to the pool, with an otherwise-idle 10G LAN). I modifyied only a few sysctls (otherwise left at default for FreeNAS 11.2), in line with the guide. The sysctls I modified were: (vfs.zfs.)*delay_scale, delay_min_dirty_percent, async_write_active_min/max_dirty_percent*. The usual sysctl for chelsio max MTU 4096 under 11.2-REL was in use, and RNG data harvesting from 10G NIC was disabled.Apart from that, it was default settings for ZFS sysctls. Hopefully this may give ideas what's up. I kept it very simple. I started by looking at dirty data writeout patterns before making any sysctl changes. This is the output of the well-known script for combined dirty.d + duration.d, when the file transfer began on the client. It seems to show matters almost immediately getting out of hand with dirty data writeout, by the 4th TXG sync: dtrace -s dirty_and_duration.d tank dtrace: script 'dirty_and_duration.d' matched 2 probes CPU ID FUNCTION:NAME 6 63989 none:txg-synced 0MB of 4096MB synced in 0.47 seconds 6 63989 none:txg-synced 0MB of 4096MB synced in 0.00 seconds 6 63989 none:txg-synced 0MB of 4096MB synced in 0.00 seconds 5 63989 none:txg-synced 64MB of 4096MB synced in 0.76 seconds <-- copy begins 1 63989 none:txg-synced 305MB of 4096MB synced in 0.32 seconds 2 63989 none:txg-synced 438MB of 4096MB synced in 2.13 seconds 1 63989 none:txg-synced 819MB of 4096MB synced in 7.88 seconds <-- 4th sync is already taking too long for 5 second TXGs to keep up. And 7.9 seconds to write out 820MB! A write-out rate of 104MB/sec, when it's got a 43% empty pool and can write to 4 7200 mirros simultaneously, without any need to calculate parity data? Something's got to be wrong. Having no idea why writeout is so incredibly slow, I first tried to get it to write consistently, without backlogging/stalling the file transfer. Following the guide, I first tried increasing *delay_scale *(current value 500K). The Delphix guide said to try doubling it or more. I took it up to 1M, then 2M, then 5M, then 50M - a factor of 100x - and the dtrace script showed that even at that scaling level, ZFS was * still* needing to invoke delays of 50M - 100M most of the time - and despite doing so, was *still *unable to prevent the issue: # dtrace -n delay-mintime'{ @ = quantize(arg2); }' dtrace: description 'delay-mintime' matched 1 probe ^C value ------------- Distribution ------------- count 128 | 0 256 | 1 512 | 0 1024 | 1 2048 | 1 4096 | 4 8192 | 6 16384 | 13 32768 | 26 65536 | 51 131072 | 102 262144 | 203 524288 |@ 399 1048576 |@ 775 2097152 |@@ 1453 4194304 |@@@@ 2603 8388608 |@@@@@@ 4243 16777216 |@@@@@ 4048 33554432 |@@@@@@ 4485 67108864 |@@@@@@@@@@@@@@@ 11208 134217728 | 0 Next, I tried the following part of the Delphix guide. I took *delay_scale *back down to "sane" values" and reducing the dirty data write-out percentages. I finally got past the problem that way - but I had to take *delay_min_dirty_percent *and *vdev.async_write_active_max_dirty_percent* down to 15%, and *delay_scale *up to 2M to do it. Those are insanely limited values for an enterprise server with a single file transfer. I checked disk IO. As shown below, all disks seemed to be performing roughly equally, suggesting the issue isn't the HBA or "bad" HDDs: # zpool iostat -T d -v tank 1 capacity operations bandwidth pool alloc free read write read write -------------------------------------- ----- ----- ----- ----- ----- ----- tank 14.5T 10.9T 0 52.3K 0 210M mirror 4.12T 3.13T 0 8.60K 0 34.7M da6 - - 0 778 0 35.1M da5 - - 0 945 0 35.8M da7 - - 0 1.01K 0 36.8M mirror 3.83T 1.61T 0 20.7K 0 82.9M da8 - - 0 5.01K 0 83.0M da9 - - 0 5.11K 0 83.0M da10 - - 0 3.14K 0 82.9M mirror 3.81T 1.62T 0 12.2K 0 48.8M da4 - - 0 2.17K 0 48.8M da11 - - 0 2.13K 0 50.7M da2 - - 0 2.73K 0 50.7M mirror 2.69T 4.57T 0 10.8K 0 43.6M da0 - - 0 1.26K 0 45.5M da1 - - 0 1.29K 0 43.7M da3 - - 0 1.50K 0 45.8M I gather the stats here aren't reliable, but if that's any indication of the average bytes per write, it also suggests something odd is going on. The vdevs comprise matched enterprise 7200 HDDs, all new, and the top 2 vdevs have slightly faster HDDs than the bottom two vdevs. Writing a single large seq file in a 57% full pool, data should be written 4 ways at a time, hence topping out at about 4 x single HDD speed as an ideal ceiling, say about 400 - 800MB/sec (the HDDs probably benchmark at around 180 - 200 MB/sec seq write individually). I double checked by testing local write speed for a non-cached 5GB local file (pre-created on a different device therefore also in ZFS ARC/L2ARC and won't slow down writing). I copied the file to three destinations: */dev/null * (to isolate the time required by /dev/random with no disk writes), *the boot pool* (ZFS on mirrored slow-ish SATA SSDs 54% full) and *the same pool as above*, to allow effects to be distinguished. I also used dtrace to watch writeouts, as above, to compare with LAN file tfr writeout patterns. I can't decide if this shows the entire time taken is due to /dev/random, or whether it shows the problem is still there and hasn't actually gone away, despite the apparently smooth LAN dataflow achieved above: # dd if=/dev/random of=/TEST0.DAT bs=1048576 count=5120 <-- saved to boot pool # dd if=/dev/random of=TEST1.DAT bs=1048576 count=5120 <-- saved to tank # /usr/bin/time -h cp /TEST0.DAT /dev/null <-- read speed for boot pool file 1.25s real 0.00s user 1.25s sys # /usr/bin/time -h cp /TEST0.DAT TEST0-COPY1.DAT <-- copy boot pool file to tank 2m30.34s real 0.00s user 2.29s sys 1 63989 none:txg-synced [At timestamp: 9507491.794 sec] 0MB of 476MB synced in 0.00 seconds 2 63989 none:txg-synced [At timestamp: 9507496.057 sec] 64MB of 476MB synced in 0.88 seconds 7 63989 none:txg-synced [At timestamp: 9507499.424 sec] 476MB of 476MB synced in 3.36 seconds 2 63989 none:txg-synced [At timestamp: 9507515.465 sec] 476MB of 476MB synced in 16.04 seconds 4 63989 none:txg-synced [At timestamp: 9507532.055 sec] 476MB of 476MB synced in 16.59 seconds 4 63989 none:txg-synced [At timestamp: 9507548.635 sec] 476MB of 476MB synced in 16.58 seconds 5 63989 none:txg-synced [At timestamp: 9507564.808 sec] 476MB of 476MB synced in 16.17 seconds 4 63989 none:txg-synced [At timestamp: 9507580.779 sec] 476MB of 476MB synced in 15.97 seconds 7 63989 none:txg-synced [At timestamp: 9507598.004 sec] 476MB of 476MB synced in 17.22 seconds 2 63989 none:txg-synced [At timestamp: 9507614.247 sec] 476MB of 476MB synced in 16.24 seconds 4 63989 none:txg-synced [At timestamp: 9507630.392 sec] 476MB of 476MB synced in 16.14 seconds 4 63989 none:txg-synced [At timestamp: 9507646.723 sec] 476MB of 476MB synced in 16.33 seconds 3 63989 none:txg-synced [At timestamp: 9507657.185 sec] 300MB of 476MB synced in 10.46 seconds 3 63989 none:txg-synced [At timestamp: 9507657.218 sec] 0MB of 476MB synced in 0.03 seconds [completed] # /usr/bin/time -h cp TEST1.DAT /dev/null <-- read speed for tank file 1.21s real 0.00s user 1.20s sys # /usr/bin/time -h cp TEST1.DAT /TEST1-COPY1.DAT <-- copy tank file to boot pool 2m0.98s real 0.00s user 2.57s sys 2 63989 none:txg-synced [At timestamp: 9507827.131 sec] 0MB of 476MB synced in 0.01 seconds 0 63989 none:txg-synced [At timestamp: 9507831.391 sec] 64MB of 476MB synced in 1.59 seconds 2 63989 none:txg-synced [At timestamp: 9507840.045 sec] 384MB of 476MB synced in 8.65 seconds 2 63989 none:txg-synced [At timestamp: 9507840.045 sec] 62MB of 476MB synced in 0.00 seconds 0 63989 none:txg-synced [At timestamp: 9507841.743 sec] 64MB of 476MB synced in 1.69 seconds 2 63989 none:txg-synced [At timestamp: 9507850.117 sec] 368MB of 476MB synced in 8.37 seconds 2 63989 none:txg-synced [At timestamp: 9507850.118 sec] 60MB of 476MB synced in 0.00 seconds 1 63989 none:txg-synced [At timestamp: 9507851.751 sec] 62MB of 476MB synced in 1.63 seconds 7 63989 none:txg-synced [At timestamp: 9507859.728 sec] 351MB of 476MB synced in 7.97 seconds 7 63989 none:txg-synced [At timestamp: 9507859.728 sec] 58MB of 476MB synced in 0.00 seconds 2 63989 none:txg-synced [At timestamp: 9507861.298 sec] 61MB of 476MB synced in 1.56 seconds 5 63989 none:txg-synced [At timestamp: 9507868.970 sec] 336MB of 476MB synced in 7.67 seconds 5 63989 none:txg-synced [At timestamp: 9507868.971 sec] 56MB of 476MB synced in 0.00 seconds 2 63989 none:txg-synced [At timestamp: 9507870.507 sec] 60MB of 476MB synced in 1.53 seconds 6 63989 none:txg-synced [At timestamp: 9507877.644 sec] 322MB of 476MB synced in 7.13 seconds 6 63989 none:txg-synced [At timestamp: 9507877.664 sec] 55MB of 476MB synced in 0.02 seconds 4 63989 none:txg-synced [At timestamp: 9507879.197 sec] 96MB of 476MB synced in 1.53 seconds [same pattern until completed] Hmm. 16-17 seconds average to write out 476MB that's already in ARC/L2ARC from a previous write? 150 secs to do the writeout of a simple 5GB file? That's about 34 MB/sec effective copying speed, to copy a file locally from ARC/L2ARC, to a pool of 4 fast striped mirrors, that should be able to write at 600-800MB/sec. It does confirm, however, that the issue isn't LAN related, since it happens even with local file writing. The boot pool isn't much faster, although it's SSD it only has one mirror, so it's at a 4x speed disadvantage from the start. Very, very wrong-looking..... Finally, I returned to dirty.d/duration.d, to compare the later TXG writeout patterns (using the new, much tighter settings that didn't stall LAN writes) to those at the initial testing run. This time we can see it is handling the tfr fine - but it also shows again how much it had to strangulate performance in order to do so: # dtrace -s dirty_and_duration.d tank dtrace: script 'dirty_and_duration.d' matched 2 probes CPU ID FUNCTION:NAME 3 63989 none:txg-synced [At timestamp: 9503423.244 sec] 0MB of 476MB synced in 0.00 seconds 4 63989 none:txg-synced [At timestamp: 9503429.056 sec] 0MB of 476MB synced in 0.50 seconds 3 63989 none:txg-synced [At timestamp: 9503431.789 sec] 64MB of 476MB synced in 0.69 seconds 7 63989 none:txg-synced [At timestamp: 9503433.114 sec] 280MB of 476MB synced in 1.32 seconds 0 63989 none:txg-synced [At timestamp: 9503435.103 sec] 476MB of 476MB synced in 1.98 seconds 2 63989 none:txg-synced [At timestamp: 9503436.962 sec] 476MB of 476MB synced in 1.85 seconds 6 63989 none:txg-synced [At timestamp: 9503438.810 sec] 470MB of 476MB synced in 1.84 seconds 3 63989 none:txg-synced [At timestamp: 9503440.791 sec] 474MB of 476MB synced in 1.98 seconds 0 63989 none:txg-synced [At timestamp: 9503442.726 sec] 476MB of 476MB synced in 1.93 seconds 2 63989 none:txg-synced [At timestamp: 9503445.287 sec] 476MB of 476MB synced in 2.56 seconds 2 63989 none:txg-synced [At timestamp: 9503447.193 sec] 476MB of 476MB synced in 1.90 seconds 2 63989 none:txg-synced [At timestamp: 9503449.566 sec] 476MB of 476MB synced in 2.37 seconds 3 63989 none:txg-synced [At timestamp: 9503451.476 sec] 476MB of 476MB synced in 1.91 seconds 2 63989 none:txg-synced [At timestamp: 9503453.207 sec] 476MB of 476MB synced in 1.73 seconds 4 63989 none:txg-synced [At timestamp: 9503454.983 sec] 411MB of 476MB synced in 1.77 seconds 3 63989 none:txg-synced [At timestamp: 9503456.860 sec] 476MB of 476MB synced in 1.87 seconds 3 63989 none:txg-synced [At timestamp: 9503458.820 sec] 473MB of 476MB synced in 1.95 seconds 3 63989 none:txg-synced [At timestamp: 9503460.682 sec] 476MB of 476MB synced in 1.86 seconds 4 63989 none:txg-synced [At timestamp: 9503462.676 sec] 476MB of 476MB synced in 1.99 seconds 0 63989 none:txg-synced [At timestamp: 9503464.346 sec] 476MB of 476MB synced in 1.66 seconds Crunching the numbers, the average write-out transfer rate is 242-245 MB/sec (411-476MB in ~1.8 sec for individual TXGs, and 7064 MB in 29.2 sec overall). It looks like something has also hit a limit - TXGs are being written out about every 1.8 sec, and also taking 1.8 sec to writeout. Put another way, to get to a point where ZFS was inserting enough delay that the file transfer was consistent and no longer backlogging and stalling the NIC buffer for tens of seconds or minutes at a time, I had to push the overall file tansfer down to about 240 MB/sec, which for a single client/single file write on an enterprise server able to write to 4 vdevs in parallel, is crazy low. I don't know if this helps, but I feel it might be useful to convey how badly my server's ZFS async write pathway is writing out, even under AFAIK reasonable/default config and fast appropriate HW for the load, and fast striped mirrors, and why I think it's indicative of something deeper I'm not seeing, whether bad sysctls or a dirty data bug as per previous replies when the issue was earlier raised on this list. So this is why I want to dig "under the hood" with dtrace, but I need help in writing the necessary short instrumenting scripts required to narrow down where it's going wrong internally.. Help kindly appreciated! Stilez On Fri, 21 Dec 2018 at 03:18, Stilez Stilezy wrote: > I'm experiencing a persistent timeout problem that only occurs during > network writes across a 10G LAN to a ZFS pool. Work to date confirms it's a > real server-side issue, arising within the ZFS side not the client, network > hardware, or FreeBSD's network I/O handling, and it involves ZFS > mishandling dirty data free-up (there are bugs logged for this), or ZFS not > correctly delaying too-fast write source. Of cours eit could also be > mis-set sysctls, but I've replicated it with a default FreeNAS setup so it > would have to be mis-configured in that as well, if so. > > To save time, can I skip the detailed debugging done to date, and just say > that is where I've got to, so far. I have asked previously, long ago, and > it was suggested at the time it could be a dirty data bug. With more > experience under the belt, I'd like to dig deeper and try to narrow down > the point which is bottlenecking, using dtrace. > > Specifically, I'd like help with a short instrumentation script to capture > four key events within the ZFS write data pathway, in order to pin down > which part of the |FS data flow is implicated in the bottleneck. I need > help because it requires detailed knowledge of the ZFS codebase. I get the > impression it could be a very short script - if I can narrow it down, maybe > I can find a fix, or else, better evidence of the exact point where the > bug/issue is arising. > > *Symptoms:* > > I'm using ZFS on enterprise hardware (Supermicro, Xeon E5-1620 v4, 128GB > ECC, Intel P3700 NVMe SLOG/ZIL, 7200 mirrored enterprise SAS3 HDD pool, > 9311 SAS3 HBA IT mode, Chelsio NICs, 10G fibre LAN, Netgear managed SFP+ > switch). > > With a quiesced "known good" 10G network, and just a single SMB, iSCSI or > netcat client (same happens on all 3 protocols), and that client doing a > single 100GB file write, I can see the network RCV buffer gradually being > driven down to zero without getting time to recover much, causing the TCP > window to go down to << 80 bytes in tcpdump as it tries to cope, and > eventually (when TCP window is driven to zero often enough) the file write > session fails due to the persistant zero window condition. Jumps in the > network RCV queue seems to coincide in time just before disk activity in > the HDD pool, adding to evidence it's related to the ZFS write pathway. > > All the evidence suggests the network and client sides aren't the issue. > What's happening is, ZFS is misconfigured or hitting a bug, it's not > pulling new data into the next TXG in a timely manner, or delaying > elegantly. File data is coming into the NIC RCV buffer at 10G speed, faster > than ZFS is allowing it to be removed, and ZFS isn't effectively signalling > the client to slow down or delay. > > > *dtrace:* > > The most direct route to narrow it down, would involve instrumenting to > log the timestamps, sizes, and time taken, for (I think!) 4 events. I don't > know the exact names used for these calls, so I have to describe their > effect. Help with identifying appropriate functions/instrumentation points, > even if it's not clear how to instrument them, would be great! > > - There should be a function that gets called/polled when ZFS has > space in its write pipeline, it checks to see if any ongoing write has a > chunk of data to be written, and if so, pulls data from the source > stream/pipe/descriptor (in this case the network RCV buffer, could be any > source of an ongoing file write). This will let me instrument the events, > when ZFS requests/gets source data from any ongoing file write, in this > case pulling it from the network buffer. > * To instrument: timestamp, max bytes to read, actual bytes read* > > - Within the ZFS code, the write pathway involves (simplistically) > newly received write data and any related metadata/consequent updates being > added to the current TXG and SLOG/ZIL, and ultimately, written out to the > pool. > *To instrument for those 3 events: timestamp, bytes written, > write-out time taken. (Ideally also, dirty data cache size > before/after)* > > > Although the output file will be huge, it should be easy to analyse the > timestamped list of events+dataflow and see which of the last 3 processes > is causing the bottleneck. > > I would enjoy doing the work, and tracking this down somewhat, but need a > little help with the instrumentation aspect which really needs developer > competences. > > Can someone kind, please help with an appropriate short instrumentation > script to capture the raw data for these 4 items :) > > (And any other data that in your experience, would be useful) > > Thank you! > > Stilez > From owner-freebsd-fs@freebsd.org Fri Dec 21 16:51:55 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95361135013D for ; Fri, 21 Dec 2018 16:51:55 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B6C11728DE for ; Fri, 21 Dec 2018 16:51:54 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qt1-x834.google.com with SMTP id d19so6347030qtq.9 for ; Fri, 21 Dec 2018 08:51:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oMkoiCanOa44u1yqmhIoVIgMIvdPxdqu9LtVTlrEomo=; b=eeYzlE+MXLFsq1AYhBHzL/RFjgdEcHLnvcvWVQsg4ZB2Q0fvlxpiUrluDSnBc4eCTA DR48+QN1Kri7VlH9Pbv0mux/1IEprxXuqbFfuaQlrCjDDgf+M2yoStkqJXiB0cIqnjtS RYPtJ1pKTJfN5ubNuPQBXfAifNqMUmwPvOYEzTptZPoJlGldiO2HAzLJa+0kpFnR8rHN z0q5FVIejeqsDUsSv4a4MGQr+7Q34nu1SD4GxcG09Jzi6FgXS5lpKMmHkC7qj0Pyz3Ad KaY31XQnk/YFQsrbKt+ng82stvo3dryG4c5j8CxYMC3k+KVrQ1p7CdYw6Zmkqyyh6eeF GDDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oMkoiCanOa44u1yqmhIoVIgMIvdPxdqu9LtVTlrEomo=; b=cpYclr5O2mH5g8BBH2cHbrV44DJo4WIT60S4av/8r+r09jhR7Z/TkrL25cpx8Mydau sK7WRl3hP5oZG3iI8pmnHNVoWL+incp02fU/9Nn6m01iaBxJR1OVaGJrBPrQG8ztSvPL CrKooLlqswk5+nSUv/DwHs4HRvRGULmMI8cc+/j31bO4IYT6ns+T/8QdLsC3oAfXVwHW OUwf3pklM3xq9/vtSbDpuFm/MY89Ry0goxDAsq0j6vh4TSKdy9H3x9q7OASyXRfTT26k rjPpcJkbWCMLMlqdhUjIi5HPBcg10pIOKj2ToLb9LtAe0djIGAXKATAsuFeG2zq1oeMD l0ww== X-Gm-Message-State: AJcUukfPlMoFJDCswPH5gXmARYNijVYAdoKDEjoqt4vW7oPiZS8zfY2/ ShpZq0ep3fCssszCW2XDuJdLWm2KZxHS1XkBSzM= X-Google-Smtp-Source: AFSGD/WNjFAqDZ6M8/DtH1ty8V4zoo64QCNLBRYSbFevN2f6xnNuXjI5cJDNndNpfSfn227l3x7lxzjUsAzsNz4VcVY= X-Received: by 2002:aed:356d:: with SMTP id b42mr3250601qte.186.1545411114047; Fri, 21 Dec 2018 08:51:54 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Sami Halabi Date: Fri, 21 Dec 2018 18:51:42 +0200 Message-ID: Subject: Re: Suggestion for hardware for ZFS fileserver To: Ben RUBSON Cc: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: B6C11728DE X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eeYzlE+M; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::834 as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-5.90 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.92)[-0.918,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.97)[ip: (-6.51), ipnet: 2607:f8b0::/32(-1.82), asn: 15169(-1.44), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 16:51:55 -0000 Hi, First thanks for everyone who responded, very helpful. One 2 more questions/clarifications: 1. As many mentioned 2008/3008 LSI HBA I searched (for lsi sas hba 24 ports) and found many but with lsi i/o controller other than 2008/3008, for example 3224)..it seems 2008/3008 are more popular in 8port hba cards. 2. I got That as much ram is better however I saw almost no docs / recommendations and how to estimate CPU.. So I need more clarification on how to size CPU with the following configurations ZFS for raidz2, all without dedup: A) lz4 compression. B) encryption (I must say I still not quiet know what are my options) C) A+B Thanks, Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, 21 = =D7=91=D7=93=D7=A6=D7=9E=D7=B3 2018, 12:33, =D7=9E=D7=90=D7=AA Ben RUBSON <= ben.rubson@gmail.com>: > > On 13 Dec 2018, Sami Halabi wrote: > > > > 2. Hardware recommendations in terms of: > > HBA > > SAS2008 / SAS3008 based card, with IT firmware. > And SAS only disks (both HDD and SSD). > > One tunable to increase SASx008 log level, which could be useful : > dev.mps.0.debug_level > > Ben > > From owner-freebsd-fs@freebsd.org Fri Dec 21 17:17:13 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A44D01351245 for ; Fri, 21 Dec 2018 17:17:13 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9EE073D0A for ; Fri, 21 Dec 2018 17:17:12 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id B57A12115B0 for ; Fri, 21 Dec 2018 12:16:41 -0500 (EST) Received: from [192.168.10.25] (D15.Denninger.Net [192.168.10.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 217CD461244 for ; Fri, 21 Dec 2018 11:16:41 -0600 (CST) Subject: Re: Suggestion for hardware for ZFS fileserver To: freebsd-fs@freebsd.org References: From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> Date: Fri, 21 Dec 2018 11:16:40 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020902050406030709040007" X-Rspamd-Queue-Id: B9EE073D0A X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.03 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.83)[-0.835,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.98)[ip: (-9.83), ipnet: 104.236.64.0/18(-3.19), asn: 14061(3.19), country: US(-0.08)]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 17:17:14 -0000 This is a cryptographically signed message in MIME format. --------------ms020902050406030709040007 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 12/21/2018 10:51, Sami Halabi wrote: > Hi, > First thanks for everyone who responded, very helpful. > > One 2 more questions/clarifications: > 1. As many mentioned 2008/3008 LSI HBA I searched (for lsi sas hba 24 > ports) and found many but with lsi i/o controller other than 2008/3008,= for > example 3224)..it seems 2008/3008 are more popular in 8port hba cards. > 2. I got That as much ram is better however I saw almost no docs / > recommendations and how to estimate CPU.. So I need more clarification = on > how to size CPU with the following configurations ZFS for raidz2, all > without dedup: > A) lz4 compression. > B) encryption (I must say I still not quiet know what are my options) > C) A+B > > Thanks, > Sami > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=99=D7=95=D7=9D =D7=95=D7=B3, 2= 1 =D7=91=D7=93=D7=A6=D7=9E=D7=B3 2018, 12:33, =D7=9E=D7=90=D7=AA IMHO -- 1. RAM *must* be ECC.=C2=A0 No wiggle room here.=C2=A0 Undetected RAM cor= ruption on a ZFS-heavy system is EXTREMELY likely to get onto the disk with a correct checksum, which means permanent, undetectable corruption of the data and possibly a pool that, when certain operations are performed on it, immediately panics the system.=C2=A0 In other words it's entirely possible to get into a situation where the only "remedy" is to destroy the pool impacted and re-create it.=C2=A0 Incidentally this means that /backups are not optional; you must have them and develop something that results in a PROVABLE good copy that can be restored in the event of disaster because of not only this risk but also human error that the operating system and hardware executes exactly as you requested, either of which results in unrecoverable data loss./ 2. More RAM is better, up to a point, in that cache is faster than disk I/O in all cases as operations are avoided.=C2=A0 HOWEVER, there are pathologies in both the FreeBSD VFS and the ARC when considered as a group.=C2=A0 I and others have tried to eliminate the pathological behavi= or under certain workloads (and some of us have had long-running debates on same.)=C2=A0 Therefore, your workload must be considered -- simply saying= "more is better" may not be correct for your particular circumstances. 3. LZ4 is good for compressible data.=C2=A0 It is worthless for non-compressible (or already-compressed) data, such as for example MP3s, MP4s (video), etc. and in fact makes performance worse since the ZFS code has to detect that the compression is futile and that takes cycles. 4. On FreeBSD I prefer GELI on the base partition to which ZFS is then pointed as a pool member for encryption at the present time.=C2=A0 It's proven, uses AES hardware acceleration on modern processors and works.=C2= =A0 Read the documentation carefully and understand your options for keying (e.g. password only, password + key file, etc) and how you will manage the security of the key component(s). --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020902050406030709040007 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgxMjIxMTcxNjQw WjBPBgkqhkiG9w0BCQQxQgRAnq1LVp5SdhEuUNp1i8FZFPmHXD8tphP7t2TDjYqW5fqBmGt9 2YRaDsi9yy2OrG0MWEUuLxMqT0Iy4frk2hPvrTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAPcaoZ8fjjpvX09NyJ47zdwDrHx5dYhTv7WNwb8l2IACjOs5HevWsQsj865cdIZbCS bcDPSmR6n8MI1jou9c45lIeSTa6mzsrxQMGAnBFcuIPrP0WhGReAHdw+I5dFuq7SIwmwGUi5 pdVEYCLUD4rO22ymMpvFFgm3qu7vRKpL5HlvfVcoVK6fT3RX0D4svtYSm+cVbQOvlNYIrLoU QkPNs7sjEKZApy+60YwLbAfKX1uqDxclEULu3ld1cQCBO6POMGGx6dAlwSvXR6ec6qLKSwu2 DvYuBAHs++LFUijrum3/F4/xlZjMCOmb+Mv8LTudrvSf8soDIp+Lp15vY/hQEG2RJpRsbezw rhne1CA/q5S+9YtDRUdLlB7EYbO2UweXfHk6AAQYEjAK8VD3ZD1YBbssE8oTXDJYUgas8NwH KaDtA71OiRa3jHiS9maURD9vasiyKbdZUkIxeU/TAhzp4UXYm43OEifu4Qa6SjBOBgTv/5bF Wch8u0HN+q5LwEuWvbnJdWqouRIST45GLBUZ1mYP0a6BXpa/cR2GNzDs95s56vn2K3w1gCtq fKiKQvxwAzo12MLL878wL1i81teyL/t32CMN8aD6UWRnKik94QnTDwQaGewvAlmlDXy7dKZm 8OVSp3P9MIKppZRFaHcXevF73kT+yihOon0KyKDg/QAAAAAAAA== --------------ms020902050406030709040007-- From owner-freebsd-fs@freebsd.org Fri Dec 21 19:25:07 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8F3F13547A0 for ; Fri, 21 Dec 2018 19:25:06 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from mailout.ifm.liu.se (mailout.ifm.liu.se [130.236.160.13]) by mx1.freebsd.org (Postfix) with ESMTP id 276BF8093B for ; Fri, 21 Dec 2018 19:25:04 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from [192.168.1.79] (h-99-23.A785.priv.bahnhof.se [158.174.99.23]) (authenticated bits=0) by mail.ifm.liu.se (8.15.2/8.14.4) with ESMTPSA id wBLIs4Be020741 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 21 Dec 2018 19:54:05 +0100 (CET) From: Peter Eriksson Message-Id: <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Suggestion for hardware for ZFS fileserver Date: Fri, 21 Dec 2018 19:53:58 +0100 In-Reply-To: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> To: freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> X-Mailer: Apple Mail (2.3445.102.3) X-IFMLiUSE-MailScanner-Information: Please contact postmaster@ifm.liu.se more information X-IFMLiUSE-MailScanner-ID: wBLIs4Be020741 X-IFMLiUSE-MailScanner: Found to be clean X-IFMLiUSE-MailScanner-From: peter@ifm.liu.se X-Spam-Status: No X-Rspamd-Queue-Id: 276BF8093B X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of peter@ifm.liu.se designates 130.236.160.13 as permitted sender) smtp.mailfrom=peter@ifm.liu.se X-Spamd-Result: default: False [-3.21 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.986,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-0.00)[country: SE(-0.02)]; NEURAL_HAM_LONG(-0.99)[-0.993,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[13.160.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MX_GOOD(-0.01)[e-mailfilter04.sunet.se,e-mailfilter03.sunet.se]; NEURAL_HAM_SHORT(-0.82)[-0.820,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 19:25:07 -0000 Just to add a few pointers based on our experience from our = FreeBSD-based filers. > 1. RAM *must* be ECC. No wiggle room here. Undetected RAM corruption 100% agreed. ECC is definitely the way to go! > 2. More RAM is better, up to a point, in that cache is faster than = disk > I/O in all cases as operations are avoided. HOWEVER, there are > pathologies in both the FreeBSD VFS and the ARC when considered as a > group. I and others have tried to eliminate the pathological behavior > under certain workloads (and some of us have had long-running debates = on > same.) Therefore, your workload must be considered -- simply saying > "more is better" may not be correct for your particular circumstances. Yes, our servers all have 256GB RAM and we=E2=80=99ve been having some = performance issues every now and then that has forced us to adjust a = couple of kernel settings in order to minimise the impact for the users. = I=E2=80=99m sure we=E2=80=99ll run into more in the future. Our setup: A number of Dell servers with 256GB RAM each, =E2=80=9CHBA330=E2=80=9D = (LSI 3008) controllers and 14 10TB 7200rpm SAS drives, dual SLOG SSDs = and dual L2ARC SSDs connected to the network via dual 10Gbps ethernet, = serving users via SMB (Samba), NFS and SFTP. Managed via Puppet. Every user get their own ZFS filesystem with a ref quota set -> about = 20000 zfs filesystems per frontend server (we have around 110K users = (students & staff) - and around 3000 active (around 500 per server) at = the same time currently (mostly SMB for now, but NFS is growing). LZ4 = compression enabled on all. Every filesystem gets a snapshot taken every hour (accessible via = Windows =E2=80=9Cprevious versions=E2=80=9D).=20 1st level backups is done via rsync to secondary servers (HPs with big = SAS disk cabinets (70 disks/cab)) so around 100K filesystems on the = biggest one right now. And snapshots on those too. No users have direct = access to them. We decided against using zfs send/recv since we wanted some better = =E2=80=9Cfault=E2=80=9D isolation between the primary and secondary = servers in case of ZFS corruption on the primary frontend servers. = Considering the panic-causing bugs with zfs send+recv that has been = reported this was probably a good choice :-) This has caused some interesting problems=E2=80=A6=20 First thing we noticed was that booting would take forever=E2=80=A6 = Mounting the 20-100k filesystems _and_ enabling them to be shared via = NFS is not done efficient at all (for each filesystem it re-reads = /etc/zfs/exports (a couple of times) befor appending one line to the = end. Repeat 20-100,000 times=E2=80=A6 Not to mention the big kernel lock = for NFS =E2=80=9Chold all NFS activity while we flush and reinstalls all = sharing information per filesystem=E2=80=9D being done by mountd=E2=80=A6 Wish list item #1: A BerkeleyDB-based =E2=80=99sharetab=E2=80=99 that = replaces the horribly slow /etc/zfs/exports text file. Wish list item #2: A reimplementation of mountd and the kernel interface = to allow a =E2=80=9Cdiff=E2=80=9D between the contents of the DB-based = sharetab above be input into the kernel instead of the brute-force way = it=E2=80=99s done now.. (I=E2=80=99ve written some code that implements item #1 above and it = helps quite a bit. Nothing near production quality yet though. I have = looked at item #2 a bit too but not done anything about it.) And then we have the Puppet =E2=80=9Cfacter=E2=80=9D process that does = an inventory of the systems. Doing things like =E2=80=9Czfs list=E2=80=9D = (to list all filesystems and then try to upload it to the PuppetDB (and = fail due to too much data) and =E2=80=9Czfs upgrade=E2=80=9D (to get the = first line of output about ZFS version - which has the side effect of = also doing a recursive walk thru all filesystems - taking something like = 6 hours on the main backup server=E2=80=A6 Solved that one with some = binary patching and a wrapper script around /sbin/zfs :-) Wish list item #3: A =E2=80=9Czfs version=E2=80=9D command that just = prints the ZFS version instead that Puppet =E2=80=98facter=E2=80=99 = could use. Wish list item #4: A better (we currently binary-patch /sbin/zfs into = /lbin/zfs (which doesn=E2=80=99t exist) in the libfacter shared = library=E2=80=A6) way to disable the ZFS filesystem enumeration that = facter does=E2=80=A6. =20 Another thing we noticed was that when the rsync backups were running = lot=E2=80=99s of things would start to go really slow. Things one at = first didn=E2=80=99t think would be affected - like everything that = stat:s /etc/nsswitch.conf (or other files/devices) on the (separate) = root disks (mirrored ZFS). Access time would inflate 100x times or more. = Turns out the rsync processes that would stat() all the users file would = cause the kernel =E2=80=98vnodes=E2=80=99 table to get full (and older = entries would get flushed out - like the vnode entry for = /etc/nsswitch.conf (and basically everything). So we increased the = kern.maxvnodes setting a number of times=E2=80=A6 Currently we=E2=80=99re = running at a vnode table size of 20 million (but we probably should = increase it even more). Uses a number of GBs of RAM but for us it=E2=80=99= s worth it. Wish list item #5: Separate vnode tables per ZFS pool, or some way to = =E2=80=9Cprotect=E2=80=9D =E2=80=9Cimportant=E2=80=9D vnodes... Another thing we noticed recently was that the ARC - which had a default = cap of about 251GB - would use all that and then we we would have a = three-way fight of memory between the ARC, the VNode table and the 500+ = =E2=80=99smbd=E2=80=99 user processes that uses quite a lot of RAM to = per process (and all the others) - causing the kernel pagedaemon to work = a lot causing the Samba smbd & Winbindd daemons to become really slow = (causing multi second login times for users). Solved that one by capping the ARC to 128GB=E2=80=A6 (Can probably = increase it a bit but 128GB seems to be plenty enough for us right now). = Now ARC gets=E2=80=99 50% of the machine and the rest have plenty of RAM = to play with. Dunno what to wish for here though :-) > 4. On FreeBSD I prefer GELI on the base partition to which ZFS is then > pointed as a pool member for encryption at the present time. It's > proven, uses AES hardware acceleration on modern processors and works.=20= > Read the documentation carefully and understand your options for = keying > (e.g. password only, password + key file, etc) and how you will manage > the security of the key component(s). Yep, we use GELI+ZFS too on one server. Works fine! - Peter From owner-freebsd-fs@freebsd.org Fri Dec 21 23:50:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C481E1337669 for ; Fri, 21 Dec 2018 23:50:02 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660040.outbound.protection.outlook.com [40.107.66.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19BF3898E5 for ; Fri, 21 Dec 2018 23:50:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM (10.169.142.146) by YQBPR01MB0178.CANPRD01.PROD.OUTLOOK.COM (10.169.141.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1446.18; Fri, 21 Dec 2018 23:49:58 +0000 Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c]) by YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c%8]) with mapi id 15.20.1446.022; Fri, 21 Dec 2018 23:49:58 +0000 From: Rick Macklem To: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Thread-Topic: Suggestion for hardware for ZFS fileserver Thread-Index: AQHUmU1yCLTPDjfxZEms4Ul8AJXDZ6WJb0QAgAAbMACAAE3yOQ== Date: Fri, 21 Dec 2018 23:49:58 +0000 Message-ID: References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net>, <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> In-Reply-To: <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR01MB0178; 6:bimqE2c7+hsAxrU6jueLWI/re4ENRIa6PVzXOVQmv3TCW3r0AJaMBzrT2N6qhYitXRiNb7+Tc2hj4VAzrxbP0xpycAl6acpoVkGVIpUjJC3XmWlpLFUnyfIqYuM2GSnAr7KbN99JZDt9BCYOtSMy6EeOFiNrwaFxomuobWKnemsayoD+/e6dIbAs1kBzlHYPX/rCq8aTa9TNA4h6f8NrwEG0D7ZAsY3j9y+uy/R6oQyxu0LRInQSvfR67lAslVTwosmWmbS18+a5R301DiUtPlJPkBpSGjeLb9pahUq929dQCO2GvHthwlGF009b3Qa4jNnKFvemqMi1PALNGRYrAqN0/uWhvRwfDWwSBmS1hU1tb9uCaN5ZlOxzGtnE5S/Hb2xpVxQoLfm259isOPjU9lxae0ZRq48qhWl0auNBph2pkD0f8vaLrql35ykmX9ctKJbS+rRO09wBisYOUPxZfKjPAp0MsQaCsjm8PXEl49I=; 5:7IBnnk2onT70sUOE7zTm4Z7gC2uLjdO4ELUXbUzKXSq8TAUEKdx7U5E8HkSCUWGGZ6qnA3ZQjqal34kFPoaF2uA3fls1piQ+FXCQwPfGVPLnkyDm8W0tUnFvA1uI2CHzRlzj4d8SRP5SzJItqPWGgqlcXziMsqIBT5+zmMHYVo0=; 7:RcphuH/cOLrU1JwoPt+cQeheqwTbZHbuqpXkPx8XYWm09Dcs8Di5wwVZJ8pVP4qckQACsQdtcMcmiUODhVDiiT2DzUVWQq6rXOry3lIaRc8goFHPqu41zLm0vjDIaCfIVsOOr42Q1zTvNhHjzYwqDQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: fc29ec1f-443e-4a06-38cd-08d6679f0423 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YQBPR01MB0178; x-ms-traffictypediagnostic: YQBPR01MB0178: x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(3230021)(999002)(5005026)(6040522)(2401047)(8121501046)(3002001)(10201501046)(93006095)(93001095)(3231475)(944501520)(52105112)(149066)(150057)(6041310)(20161123558120)(20161123564045)(20161123562045)(20161123560045)(201703131423095)(201703031522075)(201702281529075)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(201708071742011)(7699051)(76991095); SRVR:YQBPR01MB0178; BCL:0; PCL:0; RULEID:; SRVR:YQBPR01MB0178; x-forefront-prvs: 0893636978 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(366004)(136003)(39860400002)(396003)(346002)(189003)(199004)(476003)(486006)(74482002)(110136005)(446003)(11346002)(305945005)(33656002)(478600001)(99286004)(74316002)(2501003)(97736004)(8936002)(6246003)(6436002)(5660300001)(68736007)(316002)(46003)(93886005)(786003)(296002)(105586002)(6506007)(81166006)(81156014)(55016002)(9686003)(86362001)(53936002)(102836004)(76176011)(8676002)(229853002)(186003)(2906002)(106356001)(25786009)(71200400001)(71190400001)(7696005)(14444005)(256004)(14454004); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR01MB0178; H:YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-microsoft-antispam-message-info: uO+uw8mQ2E4BMcVLa3ZpqgvXkBcUUIBLqknLnMz1PNgVQZP2qDJYEij1fL1AvNJkyPwYVACjEEBmU+bQxMXE2jYm5LMpdEiS6wx2T1owDWG8tCDJOnjws6ZsNcKOSkd/iXTGwnQ38CNDNtUWAeRxTjSkpTsmhH/BEEqkocKfMt5kNnRCFpW72ySfgEqRjLwNn0tqmTU2CukrB61njt+SDcH6gtgPRGYBroelFPFxVJmt94hFlqwdwlhmhjxowpvxvEdjLwgOjlgoi3ggOUbmD8U69ZiLhaoMPZNq08H5qaPpryPtsBf7088ZGEmrkBMN spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: fc29ec1f-443e-4a06-38cd-08d6679f0423 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Dec 2018 23:49:58.5000 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR01MB0178 X-Rspamd-Queue-Id: 19BF3898E5 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.40 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-3.97 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.66.107.40.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.81)[-0.807,0]; IP_SCORE(-0.86)[ipnet: 40.64.0.0/10(-2.17), asn: 8075(-2.04), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.66.107.40.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Dec 2018 23:50:03 -0000 Peter Eriksson wrote: [good stuff snipped] >This has caused some interesting problems=85 > >First thing we noticed was that booting would take forever=85 Mounting the= 20-100k >filesystems _and_ enabling them to be shared via NFS is not done = efficient at all (for >each filesystem it re-reads /etc/zfs/exports (a coup= le of times) befor appending one >line to the end. Repeat 20-100,000 times= =85 Not to mention the big kernel lock for >NFS =93hold all NFS activity wh= ile we flush and reinstalls all sharing information per >filesystem=94 bein= g done by mountd=85 Yes, /etc/exports and mountd were implemented in the 1980s, when a dozen file systems would have been a large server. Scaling to 10,000 or more file systems wasn't even conceivable back then. >Wish list item #1: A BerkeleyDB-based =92sharetab=92 that replaces the hor= ribly >slow /etc/zfs/exports text file. >Wish list item #2: A reimplementation of mountd and the kernel interface t= o allow >a =93diff=94 between the contents of the DB-based sharetab above b= e input into the >kernel instead of the brute-force way it=92s done now.. The parser in mountd for /etc/exports is already an ugly beast and I think implementing a "diff" version will be difficult, especially figuring out wh= at needs to be deleted. I do have a couple of questions related to this: 1 - Would your case work if there was an "add these lines to /etc/exports"? (Basically adding entries for file systems, but not trying to delete a= nything previously exported. I am not a ZFS guy, but I think ZFS just generat= es another exports file and then gets mountd to export everything again.) 2 - Are all (or maybe most) of these ZFS file systems exported with the sam= e arguments? - Here I am thinking that a "default-for-all-ZFS-filesystems" line co= uld be put in /etc/exports that would apply to all ZFS file systems not e= xported by explicit lines in the exports file(s). This would be fairly easy to implement and would avoid trying to hand= le 1000s of entries. In particular, #2 above could be easily implemented on top of what is alrea= dy there, using a new type of line in /etc/exports and handling that as a spec= ial case by the NFS server code, when no specific export for the file system to= the client is found. >(I=92ve written some code that implements item #1 above and it helps quite= a bit. >Nothing near production quality yet though. I have looked at item = #2 a bit too but >not done anything about it.) [more good stuff snipped] Btw, although I put the questions here, I think a separate thread discussin= g how to scale to 10000+ file systems might be useful. (On freebsd-fs@ or freebsd-current@. The latter sometimes gets the attention of more developer= s.) rick From owner-freebsd-fs@freebsd.org Sat Dec 22 00:46:55 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2944133A6EE for ; Sat, 22 Dec 2018 00:46:55 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from mailout.ifm.liu.se (mailout.ifm.liu.se [130.236.160.13]) by mx1.freebsd.org (Postfix) with ESMTP id AC32B8BFC4 for ; Sat, 22 Dec 2018 00:46:53 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from [192.168.1.79] (h-99-23.A785.priv.bahnhof.se [158.174.99.23]) (authenticated bits=0) by mail.ifm.liu.se (8.15.2/8.14.4) with ESMTPSA id wBM0kEEI006704 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 22 Dec 2018 01:46:15 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Suggestion for hardware for ZFS fileserver From: Peter Eriksson In-Reply-To: Date: Sat, 22 Dec 2018 01:46:06 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> To: "freebsd-fs@freebsd.org" X-Mailer: Apple Mail (2.3445.102.3) X-IFMLiUSE-MailScanner-Information: Please contact postmaster@ifm.liu.se more information X-IFMLiUSE-MailScanner-ID: wBM0kEEI006704 X-IFMLiUSE-MailScanner: Found to be clean X-IFMLiUSE-MailScanner-From: peter@ifm.liu.se X-Spam-Status: No X-Rspamd-Queue-Id: AC32B8BFC4 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of peter@ifm.liu.se designates 130.236.160.13 as permitted sender) smtp.mailfrom=peter@ifm.liu.se X-Spamd-Result: default: False [-1.92 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.00)[country: SE(-0.02)]; NEURAL_HAM_LONG(-0.99)[-0.994,0]; NEURAL_SPAM_SHORT(0.48)[0.481,0]; NEURAL_HAM_MEDIUM(-0.99)[-0.989,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[13.160.236.130.list.dnswl.org : 127.0.11.2]; MX_GOOD(-0.01)[cached: e-mailfilter04.sunet.se]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2018 00:46:55 -0000 > On 22 Dec 2018, at 00:49, Rick Macklem wrote: >=20 > Peter Eriksson wrote: > [good stuff snipped] >> This has caused some interesting problems=E2=80=A6 >>=20 >> First thing we noticed was that booting would take forever=E2=80=A6 = Mounting the 20-100k >filesystems _and_ enabling them to be shared via = NFS is not done efficient at all (for >each filesystem it re-reads = /etc/zfs/exports (a couple of times) befor appending one >line to the = end. Repeat 20-100,000 times=E2=80=A6 Not to mention the big kernel lock = for >NFS =E2=80=9Chold all NFS activity while we flush and reinstalls = all sharing information per >filesystem=E2=80=9D being done by mountd=E2=80= =A6 > Yes, /etc/exports and mountd were implemented in the 1980s, when a = dozen > file systems would have been a large server. Scaling to 10,000 or more = file > systems wasn't even conceivable back then. Yeah, for a normal user with non-silly amounts of filesystems this is a = non-issue. Anyway it=E2=80=99s the kind of issues that I kind of like to = think about how to solve. It=E2=80=99s fun :-) >> Wish list item #1: A BerkeleyDB-based =E2=80=99sharetab=E2=80=99 that = replaces the horribly >slow /etc/zfs/exports text file. >> Wish list item #2: A reimplementation of mountd and the kernel = interface to allow >a =E2=80=9Cdiff=E2=80=9D between the contents of the = DB-based sharetab above be input into the >kernel instead of the = brute-force way it=E2=80=99s done now.. > The parser in mountd for /etc/exports is already an ugly beast and I = think > implementing a "diff" version will be difficult, especially figuring = out what needs > to be deleted. Yeah, I tried to decode it (this summer) and I think I sort of got the = hang of it eventually.=20 > I do have a couple of questions related to this: > 1 - Would your case work if there was an "add these lines to = /etc/exports"? > (Basically adding entries for file systems, but not trying to = delete anything > previously exported. I am not a ZFS guy, but I think ZFS just = generates another > exports file and then gets mountd to export everything again.) Yeah, the ZFS library that the zfs commands use just reads and updates = the separate /etc/zfs/exports text file (and have mountd read both = /etc/exports and /etc/zfs/exports). The problem is that basically what = it does when you tell it to =E2=80=9Czfs mount -a=E2=80=9D (mount all = filesystems in all zpools) is a big (pseudocode): For P in ZPOOLS; do For Z in ZFILESYSTEMS-AND-SNAPSHOTS in $P; do Mount $Z If $Z Have =E2=80=9Csharenfs=E2=80=9D option; Then Open /etc/zfs/exports Read until you find a matching line, replace with the options, = else if not found, Append options Close /etc/zfs/exports Signal mountd (Which then opens /etc/exports and /etc/zfs/exports and does = it=E2=80=99s magic) End End End All wrapped up in a Solaris compatibility layer I libzfs. Actually I = think it even reads the /etc/zfs/exports file twice for each loop = iteration due to some abstractions. Btw things got really =E2=80=9Cfun=E2=80= =9D when the hourly snapshots we were taking (adding 10-20k new = snapshots every hour, and we didn=E2=80=99t expire them fast enough in = the beginning) triggered the code above and that code took longer than 1 = hour to execute - mountd was 100% busy getting signalled and rereading, = flushing and reinstalling exports into the kernel all the time) and = basically never finished. Luckily we didn=E2=80=99t have an NFS clients = accessing the servers at that time :-) This summer I wrote some code to instead use a Btree BerkeleyDB file and = modified the libzfs code and mountd daemon to instead use that database = for much faster lookups (no need to read the whole /etc/zfs/exports file = all the time) and additions. Worked pretty well actually and wasn=E2=80=99= t that hard to add. Wanted to also add a possibility to add = =E2=80=9Cexports=E2=80=9D arguments =E2=80=9CSolaris=E2=80=9D-style so = one could say things like: /export/staff = vers=3D4,sec=3Dkrb5:krb5i:krb5p,rw=3D130.236.0.0/16,sec=3Dsys,ro=3D130.236= .160.0/24:10.1.2.3 But I never finished that (solaris-style exports options) part=E2=80=A6. We=E2=80=99ve lately been toying with putting the NFS sharing stuff into = separate =E2=80=9Cprivate" ZFS attribute (separate from official = =E2=80=9Csharenfs=E2=80=9D one) and have another tool to read them = instead and generate another =E2=80=9Cexports=E2=80=9D file so that file = can be generated in =E2=80=9Cone go=E2=80=9D and just signal mountd once = after all filesystems have been mounted. Unfortunately that would mean = that they won=E2=80=99t be shared until after all of them have been = mounted but we think it would take less time all-in-all. We also modified the FreeBSD boot scripts so that we make sure to first = mount all most important ZFS filesystems that is needed on the boot = disks (not just /) and then we mount (and share via NFS the rest in the = background so we can login to the machine as root early (no need for = everything to have been mounted before giving us a login prompt). (Right now a reboot of the bigger servers take an hour or two before all = filesystems are mounted and exported). =20 > 2 - Are all (or maybe most) of these ZFS file systems exported with = the same > arguments? > - Here I am thinking that a "default-for-all-ZFS-filesystems" = line could be > put in /etc/exports that would apply to all ZFS file systems = not exported > by explicit lines in the exports file(s). > This would be fairly easy to implement and would avoid trying to = handle > 1000s of entries. For us most have exactly the same exports arguments. (We set options on = the top level filsystems (/export/staff, /export/students etc) and then = all home dirs inherit those. > In particular, #2 above could be easily implemented on top of what is = already > there, using a new type of line in /etc/exports and handling that as a = special > case by the NFS server code, when no specific export for the file = system to the > client is found. >=20 >> (I=E2=80=99ve written some code that implements item #1 above and it = helps quite a bit. >Nothing near production quality yet though. I have = looked at item #2 a bit too but >not done anything about it.) > [more good stuff snipped] > Btw, although I put the questions here, I think a separate thread = discussing > how to scale to 10000+ file systems might be useful. (On freebsd-fs@ = or > freebsd-current@. The latter sometimes gets the attention of more = developers.) Yeah, probably a good idea! - Peter > rick >=20 >=20 From owner-freebsd-fs@freebsd.org Sat Dec 22 14:49:36 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C6E31355B47 for ; Sat, 22 Dec 2018 14:49:36 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DEFB8A88E for ; Sat, 22 Dec 2018 14:49:35 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-qt1-x835.google.com with SMTP id v11so9061833qtc.2 for ; Sat, 22 Dec 2018 06:49:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jmE9D8os0NUpT2tv7MQNiwHZdUKRRzW8h0o/crb6fXA=; b=mCTQJ5WtgAzqfit67XcHbfv5uqgpY0v/TfisqeBpOClzJ52XtwyC48KyQdz99s58+J DB1v/mapuSR/VL1GnzCUhV5ntxQpHqNVpxSmpWBAjgo8qXq9WYkIuKt4IhFtGjt2COzu KK5A/O/ToYRy4LtXzXZuHZUC8Qts4pdGmtIxKnzLCPR5nEvo0kfSGX2ibmjvZg0Dj3dN 5/mRUegm4jFJWnmvd4TXoTXoJgaTpVqyZyrPn0OBw+rSrT5XR5pV/E7OvXowg1MoYvFm 7iyMw3tuXyNvJ7pmN2OZWzUwLA/sWgdABdGAJSGQsInPdxIqpzVe/Bb6eqPt8Ocw9zG8 PlmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jmE9D8os0NUpT2tv7MQNiwHZdUKRRzW8h0o/crb6fXA=; b=kKe3YGW62BwyurguQjLZnRopxe8haUMcs1+uT+yh15Jleej+NNbPmGayGEKbzzXzAm nczWZeZUM0lPJJUlVQSzpFBHxpWnxXF4E8eSzh6n/jiD8MTMHP1Wv3AEoHBAIhlLQUh9 419TSzbMCDvSAMl2kqCFCEF6f81iVpfI25PUgexPmSVSLE7T2uLIL3BD9rqItFdZYkR5 QHVxFdc9Y/A/sVTrHLcs6jJNDLyorLb/GiAQ4d8KCcUSn83GAF3w0/4L5L4Up6IvJJoP JC1OqpKd/Ys0d/4mvCLU9CELmRJ3wY+Dw0LYuF7p/yCoShhRGYhp79gib7JXfDz0z1yd LCOA== X-Gm-Message-State: AJcUukeWxIXi6J92MyuxWs0jr3+MOYbbht+7abpPNbZzwAwNQXT30XCI j5CJg7qNBjuSZMxzzj/Y3iNoniDaP4UcmrWz8/Jaqw== X-Google-Smtp-Source: AFSGD/VzcwgBYPOsdolNZw6fEin8w5iK6LeQtHcO/L6zitUnDv6c1LonNjesmh6fir1CPK1TJ6Rq3LmQUL89HMzQim8= X-Received: by 2002:aed:356d:: with SMTP id b42mr6510091qte.186.1545490174621; Sat, 22 Dec 2018 06:49:34 -0800 (PST) MIME-Version: 1.0 References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> In-Reply-To: From: Sami Halabi Date: Sat, 22 Dec 2018 16:49:22 +0200 Message-ID: Subject: Re: Suggestion for hardware for ZFS fileserver To: Peter Eriksson Cc: freebsd-fs@freebsd.org X-Rspamd-Queue-Id: 5DEFB8A88E X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=mCTQJ5Wt; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of sodynet1@gmail.com designates 2607:f8b0:4864:20::835 as permitted sender) smtp.mailfrom=sodynet1@gmail.com X-Spamd-Result: default: False [-3.68 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.72)[-0.723,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[5.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; HTTP_TO_IP(1.00)[]; IP_SCORE(-1.94)[ip: (-6.36), ipnet: 2607:f8b0::/32(-1.83), asn: 15169(-1.45), country: US(-0.08)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2018 14:49:36 -0000 Hi, What sas hba card do you recommend for 16/24 internal ports and 2 external that are recognized and work well with freebsd ZFS. Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=A9=D7=91=D7=AA, 22 =D7=91=D7=93=D7= =A6=D7=9E=D7=B3 2018, 2:48, =D7=9E=D7=90=D7=AA Peter Eriksson : > > > > On 22 Dec 2018, at 00:49, Rick Macklem wrote: > > > > Peter Eriksson wrote: > > [good stuff snipped] > >> This has caused some interesting problems=E2=80=A6 > >> > >> First thing we noticed was that booting would take forever=E2=80=A6 Mo= unting > the 20-100k >filesystems _and_ enabling them to be shared via NFS is not > done efficient at all (for >each filesystem it re-reads /etc/zfs/exports = (a > couple of times) befor appending one >line to the end. Repeat 20-100,000 > times=E2=80=A6 Not to mention the big kernel lock for >NFS =E2=80=9Chold = all NFS activity > while we flush and reinstalls all sharing information per >filesystem=E2= =80=9D > being done by mountd=E2=80=A6 > > Yes, /etc/exports and mountd were implemented in the 1980s, when a doze= n > > file systems would have been a large server. Scaling to 10,000 or more > file > > systems wasn't even conceivable back then. > > Yeah, for a normal user with non-silly amounts of filesystems this is a > non-issue. Anyway it=E2=80=99s the kind of issues that I kind of like to = think > about how to solve. It=E2=80=99s fun :-) > > > >> Wish list item #1: A BerkeleyDB-based =E2=80=99sharetab=E2=80=99 that = replaces the > horribly >slow /etc/zfs/exports text file. > >> Wish list item #2: A reimplementation of mountd and the kernel > interface to allow >a =E2=80=9Cdiff=E2=80=9D between the contents of the = DB-based sharetab > above be input into the >kernel instead of the brute-force way it=E2=80= =99s done > now.. > > The parser in mountd for /etc/exports is already an ugly beast and I > think > > implementing a "diff" version will be difficult, especially figuring ou= t > what needs > > to be deleted. > > Yeah, I tried to decode it (this summer) and I think I sort of got the > hang of it eventually. > > > > I do have a couple of questions related to this: > > 1 - Would your case work if there was an "add these lines to > /etc/exports"? > > (Basically adding entries for file systems, but not trying to delet= e > anything > > previously exported. I am not a ZFS guy, but I think ZFS just > generates another > > exports file and then gets mountd to export everything again.) > > Yeah, the ZFS library that the zfs commands use just reads and updates th= e > separate /etc/zfs/exports text file (and have mountd read both /etc/expor= ts > and /etc/zfs/exports). The problem is that basically what it does when yo= u > tell it to =E2=80=9Czfs mount -a=E2=80=9D (mount all filesystems in all z= pools) is a big > (pseudocode): > > For P in ZPOOLS; do > For Z in ZFILESYSTEMS-AND-SNAPSHOTS in $P; do > Mount $Z > If $Z Have =E2=80=9Csharenfs=E2=80=9D option; Then > Open /etc/zfs/exports > Read until you find a matching line, replace with the options, els= e > if not found, Append options > Close /etc/zfs/exports > Signal mountd > (Which then opens /etc/exports and /etc/zfs/exports and does it= =E2=80=99s > magic) > End > End > End > > All wrapped up in a Solaris compatibility layer I libzfs. Actually I thin= k > it even reads the /etc/zfs/exports file twice for each loop iteration due > to some abstractions. Btw things got really =E2=80=9Cfun=E2=80=9D when th= e hourly snapshots > we were taking (adding 10-20k new snapshots every hour, and we didn=E2=80= =99t > expire them fast enough in the beginning) triggered the code above and th= at > code took longer than 1 hour to execute - mountd was 100% busy getting > signalled and rereading, flushing and reinstalling exports into the kerne= l > all the time) and basically never finished. Luckily we didn=E2=80=99t hav= e an NFS > clients accessing the servers at that time :-) > > This summer I wrote some code to instead use a Btree BerkeleyDB file and > modified the libzfs code and mountd daemon to instead use that database f= or > much faster lookups (no need to read the whole /etc/zfs/exports file all > the time) and additions. Worked pretty well actually and wasn=E2=80=99t t= hat hard > to add. Wanted to also add a possibility to add =E2=80=9Cexports=E2=80=9D= arguments > =E2=80=9CSolaris=E2=80=9D-style so one could say things like: > > /export/staff vers=3D4,sec=3Dkrb5:krb5i:krb5p,rw=3D > 130.236.0.0/16,sec=3Dsys,ro=3D130.236.160.0/24:10.1.2.3 > > But I never finished that (solaris-style exports options) part=E2=80=A6. > > We=E2=80=99ve lately been toying with putting the NFS sharing stuff into = separate > =E2=80=9Cprivate" ZFS attribute (separate from official =E2=80=9Csharenfs= =E2=80=9D one) and have > another tool to read them instead and generate another =E2=80=9Cexports= =E2=80=9D file so > that file can be generated in =E2=80=9Cone go=E2=80=9D and just signal mo= untd once after > all filesystems have been mounted. Unfortunately that would mean that the= y > won=E2=80=99t be shared until after all of them have been mounted but we = think it > would take less time all-in-all. > > We also modified the FreeBSD boot scripts so that we make sure to first > mount all most important ZFS filesystems that is needed on the boot disks > (not just /) and then we mount (and share via NFS the rest in the > background so we can login to the machine as root early (no need for > everything to have been mounted before giving us a login prompt). > > (Right now a reboot of the bigger servers take an hour or two before all > filesystems are mounted and exported). > > > > 2 - Are all (or maybe most) of these ZFS file systems exported with the > same > > arguments? > > - Here I am thinking that a "default-for-all-ZFS-filesystems" line > could be > > put in /etc/exports that would apply to all ZFS file systems no= t > exported > > by explicit lines in the exports file(s). > > This would be fairly easy to implement and would avoid trying to > handle > > 1000s of entries. > > For us most have exactly the same exports arguments. (We set options on > the top level filsystems (/export/staff, /export/students etc) and then a= ll > home dirs inherit those. > > > In particular, #2 above could be easily implemented on top of what is > already > > there, using a new type of line in /etc/exports and handling that as a > special > > case by the NFS server code, when no specific export for the file syste= m > to the > > client is found. > > > >> (I=E2=80=99ve written some code that implements item #1 above and it h= elps > quite a bit. >Nothing near production quality yet though. I have looked a= t > item #2 a bit too but >not done anything about it.) > > [more good stuff snipped] > > Btw, although I put the questions here, I think a separate thread > discussing > > how to scale to 10000+ file systems might be useful. (On freebsd-fs@ or > > freebsd-current@. The latter sometimes gets the attention of more > developers.) > > Yeah, probably a good idea! > > - Peter > > > rick > > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@freebsd.org Sat Dec 22 16:54:48 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90DF51358348 for ; Sat, 22 Dec 2018 16:54:48 +0000 (UTC) (envelope-from mboyaci@ciu.edu.tr) Received: from haspolat.ciu.edu.tr (karpaz.ciu.edu.tr [212.175.150.25]) by mx1.freebsd.org (Postfix) with ESMTP id 0A9A68EC04 for ; Sat, 22 Dec 2018 16:54:46 +0000 (UTC) (envelope-from mboyaci@ciu.edu.tr) Received: from localhost (localhost [127.0.0.1]) by haspolat.ciu.edu.tr (Postfix) with ESMTP id 4C8813A06A0C for ; Sat, 22 Dec 2018 18:48:41 +0200 (EET) X-Virus-Scanned: amavisd-new at ciu.edu.tr Received: from haspolat.ciu.edu.tr ([127.0.0.1]) by localhost (haspolat.ciu.edu.tr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BEIRImnPg6Zg for ; Sat, 22 Dec 2018 18:48:41 +0200 (EET) Received: from localhost (localhost [127.0.0.1]) by haspolat.ciu.edu.tr (Postfix) with ESMTP id C18383A21CD6 for ; Sat, 22 Dec 2018 18:27:26 +0200 (EET) X-Virus-Scanned: amavisd-new at ciu.edu.tr Received: from haspolat.ciu.edu.tr ([127.0.0.1]) by localhost (haspolat.ciu.edu.tr [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dZp5nZe4GMhU for ; Sat, 22 Dec 2018 18:27:26 +0200 (EET) Received: from ciu.edu.tr (arf.ciu.edu.tr [10.0.0.17]) by haspolat.ciu.edu.tr (Postfix) with ESMTP id 8220B3A0969A for ; Sat, 22 Dec 2018 17:57:26 +0200 (EET) Reply-To: bendannan@bendannanesq.org From: "Barrister Ben D Annan Esq" To: freebsd-fs@freebsd.org Subject: Re: Privileged And Confidential For You - freebsd-fs@freebsd.org Date: 22 Dec 2018 18:58:16 +0300 Message-ID: <20181222185816.5CA04724A88F92F8@ciu.edu.tr> X-Rspamd-Queue-Id: 0A9A68EC04 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=ciu.edu.tr; spf=pass (mx1.freebsd.org: domain of mboyaci@ciu.edu.tr designates 212.175.150.25 as permitted sender) smtp.mailfrom=mboyaci@ciu.edu.tr X-Spamd-Result: default: False [-0.86 / 15.00]; ARC_NA(0.00)[]; FAKE_REPLY(1.00)[]; RCVD_COUNT_FIVE(0.00)[6]; HAS_REPLYTO(0.00)[bendannan@bendannanesq.org]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:212.175.150.25]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.986,0]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_MEDIUM(-0.98)[-0.979,0]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; NEURAL_SPAM_SHORT(0.58)[0.579,0]; MX_GOOD(-0.01)[karpaz.ciu.edu.tr]; DMARC_POLICY_ALLOW(-0.50)[ciu.edu.tr,none]; MIME_HTML_ONLY(0.20)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; IP_SCORE(-0.06)[asn: 9121(-0.40), country: TR(0.08)]; ASN(0.00)[asn:9121, ipnet:212.175.150.0/24, country:TR]; MID_RHS_MATCH_FROM(0.00)[] MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Dec 2018 16:54:48 -0000