From owner-freebsd-fs@freebsd.org Tue Sep 8 07:43:08 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E32773E66A9; Tue, 8 Sep 2020 07:43:08 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Blxvw5Xsrz4Nk9; Tue, 8 Sep 2020 07:43:08 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MBP-WLAN.fritz.box (p200300cd5f16b9007cacd1235d1cc1e5.dip0.t-ipconnect.de [IPv6:2003:cd:5f16:b900:7cac:d123:5d1c:c1e5]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 88A9921D76; Tue, 8 Sep 2020 07:43:07 +0000 (UTC) (envelope-from se@freebsd.org) To: FreeBSD CURRENT , freebsd-fs@freebsd.org From: Stefan Esser Subject: OpenZFS and L2ARC Message-ID: Date: Tue, 8 Sep 2020 09:43:03 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 07:43:09 -0000 OpenZFS seems to work quite well for me, in general, but I have questions regarding the L2ARC statistics. The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that are not relevant here, since without level 2 ARC) and an 1 TB M.2 SSD with a 256 GB partition for the L2ARC (and most of it currently unused, else). The L2ARC seems to have filled to the limit of 256 GB, but after several reboots, sysctl reports a L2ARC size of nearly twice the allocated space: kstat.zfs.misc.arcstats.l2_size: 534620858880 That is 497 GiB, and might be possible with a lz4 compression factor of 2 - if the value reported is not the space allocated, but the actual (uncompressed) data held by the L2ARC. The sysutils/zfs-stats port reports the following values for this system, BTW: ------------------------------------------------------------------------ ZFS Subsystem Report Tue Sep 8 09:02:46 2020 ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Passed Headroom: 0 Tried Lock Failures: 0 IO In Progress: 0 Low Memory Aborts: 7 Free on Write: 123 Writes While Full: 0 R/W Clashes: 0 Bad Checksums: 0 IO Errors: 0 SPA Mismatch: 0 L2 ARC Size: (Adaptive) 497.91 GiB Header Size: 0.11% 558.83 MiB L2 ARC Evicts: Lock Retries: 6 Upon Reading: 0 L2 ARC Breakdown: 5.75 m Hit Ratio: 81.94% 4.71 m Miss Ratio: 18.06% 1.04 m Feeds: 235.04 k L2 ARC Buffer: Bytes Scanned: 0 Bytes Buffer Iterations: 0 List Iterations: 0 NULL List Iterations: 0 L2 ARC Writes: Writes Sent: 100.00% 22.67 k ------------------------------------------------------------------------ With the FreeBSD ZFS (without persistent L2ARC) I never got more than 20% hit ratio on the L2ARC between reboots. Quite a number of sysctl variable names have changed, and the port needs to be adapted to the new names (therefore there are lots of 0 values in the -L output). The following names used by zfs-stats do not exist in OpenZFS: kstat.zfs.misc.arcstats.recycle_miss kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned kstat.zfs.misc.arcstats.l2_write_buffer_iter kstat.zfs.misc.arcstats.l2_write_buffer_list_iter kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter kstat.zfs.misc.arcstats.l2_write_full kstat.zfs.misc.arcstats.l2_write_in_l2 kstat.zfs.misc.arcstats.l2_write_io_in_progress kstat.zfs.misc.arcstats.l2_write_not_cacheable kstat.zfs.misc.arcstats.l2_write_passed_headroom kstat.zfs.misc.arcstats.l2_write_pios kstat.zfs.misc.arcstats.l2_write_spa_mismatch kstat.zfs.misc.arcstats.l2_write_trylock_fail kstat.zfs.misc.arcstats.l2_writes_hdr_miss vfs.zfs.vdev.cache.size The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size can be used to detect OpenZFS, and is easily fixed. But the above listed L2ARC values seem to have been removed from or have never existed in OpenZFS, and I did not find any substitutes. Are there any plans to re-create them in OpenZFS on FreeBSD or are they gone for good? I'd like to update the zfs-stats port for compatibilíty with OpenZFS ... From owner-freebsd-fs@freebsd.org Tue Sep 8 17:06:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 99CA73CFC6E for ; Tue, 8 Sep 2020 17:06:36 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmBQ36Wc9z42nq for ; Tue, 8 Sep 2020 17:06:35 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: by mail-ej1-x642.google.com with SMTP id p9so23594172ejf.6 for ; Tue, 08 Sep 2020 10:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=IYM6iJdqqYhINX0fwgEzq4yf7PEghQh6BisS+AyKgZQ=; b=Bv+GBn8Nju0TZM1fgdpQqRKmOCryXj9oAN9fAhZiPYCVFTxku+WlFDQEi7txtwdzkZ gZGGPJXBxOGhndem5uo1DmDqFxiXE4wuWeU2d964RSvAiadX1IiSbcs2kf446y8QBHaO ymdIdO0asUbgcXuUzEQ6xmpirjwM5PvZcTha/Xn5mqKWSzD6aT1x2Ex2hCVrkrxnhX17 gix3MRRSB2HXNGH9BneAdEnaxte9kKLC49rG5ampDxcVXGBDs43ZGjvSivcaxi2KM3AT 8jmTanH4LQ882MrgDclV/yceid3ORnRxem3SOfm4mfW5JWdeRhQC4vWBouXQ0MsyxplA Y9gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=IYM6iJdqqYhINX0fwgEzq4yf7PEghQh6BisS+AyKgZQ=; b=RqyDN//GT11Hr1mV/vGXKAZSLyy77RV9PlsQ5kk0NqSQ07SF9gsS3bUagM+bySe/KP CwWXMcLvuLB2txfFmTuqi+2uUJMptdCNfW2FI30eGomHLDSEv1bDtwYeGclka6aYpRsS 8WvJLNLC/fIDws1avMshUmkifZRjkS7ECZ7xWgv6GNPPxRzQMZ+jH3Cwl9aPlJFjDGFs k8KuYp7QEvagXXXF/8lrNk3PEuVFIP7WUZvRMIl4EXPUE60WMnMOD/GcBKkHJKTrDBfB nboGUWMNOayCK+mGf4py/0n1F/XJA1NJffhJZ4neHCsH1raH9QtxPiUXWydehVdURQzS Z2sQ== X-Gm-Message-State: AOAM532NzAQYmGe0FtI7tuSNuw7974jqoAWigi44rzNmLFX9vdTNmyEn 44D3rRRNz3CkjDTtkHeFKh7dwaGN0Q/ca0OEVOwMLw== X-Google-Smtp-Source: ABdhPJwjkS4VnbtXEJJD7pVF86l1O7tbK9Q16qZkXwOES08yVNyjhHf6eVl2FIwTcdwkUre3x3jLk5cFXY/lSzorneE= X-Received: by 2002:a17:906:4e54:: with SMTP id g20mr27898481ejw.252.1599584794190; Tue, 08 Sep 2020 10:06:34 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ryan Moeller Date: Tue, 8 Sep 2020 13:06:23 -0400 Message-ID: Subject: Re: OpenZFS and L2ARC To: Stefan Esser Cc: FreeBSD CURRENT , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4BmBQ36Wc9z42nq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ixsystems.com header.s=google header.b=Bv+GBn8N; dmarc=pass (policy=none) header.from=ixsystems.com; spf=pass (mx1.freebsd.org: domain of ryan@ixsystems.com designates 2a00:1450:4864:20::642 as permitted sender) smtp.mailfrom=ryan@ixsystems.com X-Spamd-Result: default: False [-3.43 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[ixsystems.com:s=google]; FREEFALL_USER(0.00)[ryan]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-0.99)[-0.993]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[ixsystems.com:+]; DMARC_POLICY_ALLOW(-0.50)[ixsystems.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::642:from]; NEURAL_HAM_SHORT(-0.46)[-0.456]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:06:36 -0000 On Tue, Sep 8, 2020 at 3:43 AM Stefan Esser wrote: > > OpenZFS seems to work quite well for me, in general, but I have > questions regarding the L2ARC statistics. > > The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that > are not relevant here, since without level 2 ARC) and an 1 TB M.2 > SSD with a 256 GB partition for the L2ARC (and most of it currently > unused, else). > > The L2ARC seems to have filled to the limit of 256 GB, but after > several reboots, sysctl reports a L2ARC size of nearly twice the > allocated space: > > kstat.zfs.misc.arcstats.l2_size: 534620858880 > > That is 497 GiB, and might be possible with a lz4 compression > factor of 2 - if the value reported is not the space allocated, > but the actual (uncompressed) data held by the L2ARC. > > > The sysutils/zfs-stats port reports the following values for > this system, BTW: > > ------------------------------------------------------------------------ > ZFS Subsystem Report Tue Sep 8 09:02:46 2020 > ------------------------------------------------------------------------ > > L2 ARC Summary: (HEALTHY) > Passed Headroom: 0 > Tried Lock Failures: 0 > IO In Progress: 0 > Low Memory Aborts: 7 > Free on Write: 123 > Writes While Full: 0 > R/W Clashes: 0 > Bad Checksums: 0 > IO Errors: 0 > SPA Mismatch: 0 > > L2 ARC Size: (Adaptive) 497.91 GiB > Header Size: 0.11% 558.83 MiB > > L2 ARC Evicts: > Lock Retries: 6 > Upon Reading: 0 > > L2 ARC Breakdown: 5.75 m > Hit Ratio: 81.94% 4.71 m > Miss Ratio: 18.06% 1.04 m > Feeds: 235.04 k > > L2 ARC Buffer: > Bytes Scanned: 0 Bytes > Buffer Iterations: 0 > List Iterations: 0 > NULL List Iterations: 0 > > L2 ARC Writes: > Writes Sent: 100.00% 22.67 k > > ------------------------------------------------------------------------ > > With the FreeBSD ZFS (without persistent L2ARC) I never got more > than 20% hit ratio on the L2ARC between reboots. > > Quite a number of sysctl variable names have changed, and the port > needs to be adapted to the new names (therefore there are lots of 0 > values in the -L output). > > The following names used by zfs-stats do not exist in OpenZFS: > > kstat.zfs.misc.arcstats.recycle_miss > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned > kstat.zfs.misc.arcstats.l2_write_buffer_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter > kstat.zfs.misc.arcstats.l2_write_full > kstat.zfs.misc.arcstats.l2_write_in_l2 > kstat.zfs.misc.arcstats.l2_write_io_in_progress > kstat.zfs.misc.arcstats.l2_write_not_cacheable > kstat.zfs.misc.arcstats.l2_write_passed_headroom > kstat.zfs.misc.arcstats.l2_write_pios > kstat.zfs.misc.arcstats.l2_write_spa_mismatch > kstat.zfs.misc.arcstats.l2_write_trylock_fail > kstat.zfs.misc.arcstats.l2_writes_hdr_miss > vfs.zfs.vdev.cache.size > > The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size > can be used to detect OpenZFS, and is easily fixed. > > But the above listed L2ARC values seem to have been removed from or > have never existed in OpenZFS, and I did not find any substitutes. > > Are there any plans to re-create them in OpenZFS on FreeBSD or are > they gone for good? > > I'd like to update the zfs-stats port for compatibil=C3=ADty with OpenZFS= ... Improved L2ARC stats is something we would like to look into, yes. There is currently one PR open to add a few more arcstats for L2: * evict_l2_eligible_mfu * evict_l2_eligible_mru * l2_prefetch_asize * l2_mru_asize * l2_mfu_asize * l2_bufc_data_asize * l2_bufc_metadata_asize Adding the other L2ARC arcstats from FreeBSD would be a welcome change, as well. It's *somewhere* on the todo list. -- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com From owner-freebsd-fs@freebsd.org Tue Sep 8 17:30:56 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 653683D0A6B; Tue, 8 Sep 2020 17:30:56 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmBy80dt3z452l; Tue, 8 Sep 2020 17:30:55 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x744.google.com with SMTP id d20so16088002qka.5; Tue, 08 Sep 2020 10:30:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Y24IO2upEyHZ2JiazVFuiBDXPC081SKq/UdzR8dAB1M=; b=Qxjgm87yk2XAW4BoWOklt2NZ6qc/8pbkCTAjeq4ZFr0TLtNF+ofPIg23DaOZKFqL5n MODjpKN0A1QPpPrn8nihtC1Qmh4nPyGIPxrMlAkvXBS8d8ixl1lvIEvoBETCPpB8lc6u vjTzQJIETNlsgoTHWu237Oad98vIksmBFBxnzW1CX18NhJjnAhWjlTluGT23Na9yDwze O+DUNvr81wauC5mOp61/DVlRrsMrw+FyP7B21slpXBRBUdT/PypdtmhVPXtrEXREAU0h zUKL/Etk9nJ61Tv1FboL2BnYh/UsF6dyMr78H1KqpaMdzbLWCQdqFKym4sunj1cVHVAC V4kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=Y24IO2upEyHZ2JiazVFuiBDXPC081SKq/UdzR8dAB1M=; b=aCQ3QN0qGM6omfKxueAlFPACHakbZudSegRQPf0lVk4gMWaOngl39ShKTiktdaJwF0 AUPIyasb4OQ/s+nqxISSxue+bloyAfRkrVGJ/JOEX340rWs8lKL2IImvtVAtVGMvvhCs KloMeFiO0qbetpKq/rnCUhMogn3z2FJgAiot6Er6g2Cu/SOWd8saqtDCU1eTqcN4pqk7 TIExIumNiq9PISPwAHoK5BBkzEGqKmg3KwwsceWWMIyV8LdeOpdq4P/AjTlBGX8X8Dvp t5r3Xx7RuN/6wEyNdh1jTOL3Mx+rFV4yoWFAE5MljIfbXN+HnS5aCB5THuze2SilMgAX ZdqA== X-Gm-Message-State: AOAM5307S7sSOauQ6hbZDg/XMzjjZSrgyf5dSNx0ozYlMR8KGlaa8TOp hoxF18EQHcPo2qJlXgH76DZDDo4mCC1sMg== X-Google-Smtp-Source: ABdhPJwhBSLkP4fBuoZVVPN6hEqDRkGX8ELvaZsz68mTtF25vMRMT+mBiSbZ9KyfqJ7nU8Hd9bzqSw== X-Received: by 2002:a37:a64a:: with SMTP id p71mr1091924qke.389.1599586254906; Tue, 08 Sep 2020 10:30:54 -0700 (PDT) Received: from raichu (toroon0560w-lp130-08-67-71-176-35.dsl.bell.ca. [67.71.176.35]) by smtp.gmail.com with ESMTPSA id v42sm102826qth.35.2020.09.08.10.30.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Sep 2020 10:30:53 -0700 (PDT) Sender: Mark Johnston Date: Tue, 8 Sep 2020 13:30:51 -0400 From: Mark Johnston To: Guido Falsi Cc: freebsd-current@freebsd.org, freebsd-fs@freebsd.org, Matt Macy Subject: Re: Boot error with OpenZFS Message-ID: <20200908173051.GB5058@raichu> References: <364bc35c-3930-2ce5-a3cf-3039bf671a92@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <364bc35c-3930-2ce5-a3cf-3039bf671a92@FreeBSD.org> X-Rspamd-Queue-Id: 4BmBy80dt3z452l X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:30:56 -0000 On Tue, Sep 08, 2020 at 07:24:38PM +0200, Guido Falsi wrote: > On 08/09/20 10:01, Guido Falsi wrote: > > Hi, > > > > I'm trying to update to recent head, but I can't boot my system with the > > compiled kernel. > > > > The system has ZFS on root and was working with previous kernel (before > > OpenZFS migration). > > > > I'm trying to boot r365437. > > > > I load zfs from loader with zfs_load="YES" as usual and get this error > > message at the start of kernel output: > > > > link_elf_obj: symbol lockstat_enabled undefined > > KLD file zfs.ko - could not finalize loading > > > > The zfs.ko file is aligned with the kernel and I did not observe errors > > while compiling. > > > > I am able to make the machine using the kernel from the most recent head > > snapshot on ftp.freebsd.org [1]. I'm also going to try with a kernel > > with debug symbols and one with GENERIC config. > > A locally built GENERIC works fine, so this is my fault. I clearly have > something wrong in my kernel config. > > I'll report anyway as soon as I discover what it is. Can you verify that adding "options KDTRACE_HOOKS" fixes the problem? I note that the zfs.ko Makefile has -DKTRACE_HOOKS in CFLAGS, among a few other surprising things. From owner-freebsd-fs@freebsd.org Tue Sep 8 17:52:49 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5391E3D1FCE for ; Tue, 8 Sep 2020 17:52:49 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmCRL0dtzz4B8N for ; Tue, 8 Sep 2020 17:52:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 9E1C014AA7 for ; Tue, 8 Sep 2020 17:52:35 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 9E1C014AA7 Subject: Re: OpenZFS and L2ARC To: freebsd-fs@freebsd.org References: From: Allan Jude Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Message-ID: <4f30a8a5-b7fb-e4bb-a5e6-b59fa4ec10c2@freebsd.org> Date: Tue, 8 Sep 2020 13:52:34 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD" X-Rspamd-Queue-Id: 4BmCRL0dtzz4B8N X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:52:50 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD Content-Type: multipart/mixed; boundary="m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA"; protected-headers="v1" From: Allan Jude To: freebsd-fs@freebsd.org Message-ID: <4f30a8a5-b7fb-e4bb-a5e6-b59fa4ec10c2@freebsd.org> Subject: Re: OpenZFS and L2ARC References: In-Reply-To: --m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2020-09-08 03:43, Stefan Esser wrote: > OpenZFS seems to work quite well for me, in general, but I have > questions regarding the L2ARC statistics. >=20 > The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that > are not relevant here, since without level 2 ARC) and an 1 TB M.2 > SSD with a 256 GB partition for the L2ARC (and most of it currently > unused, else). >=20 > The L2ARC seems to have filled to the limit of 256 GB, but after > several reboots, sysctl reports a L2ARC size of nearly twice the > allocated space: >=20 > kstat.zfs.misc.arcstats.l2_size: 534620858880 >=20 > That is 497 GiB, and might be possible with a lz4 compression > factor of 2 - if the value reported is not the space allocated, > but the actual (uncompressed) data held by the L2ARC. >=20 >=20 > The sysutils/zfs-stats port reports the following values for > this system, BTW: >=20 > -----------------------------------------------------------------------= - > ZFS Subsystem Report=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Tue S= ep=A0 8 09:02:46 2020 > -----------------------------------------------------------------------= - >=20 > L2 ARC Summary: (HEALTHY) > =A0=A0=A0=A0Passed Headroom:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0Tried Lock Failures:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0IO In Progress:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0 > =A0=A0=A0=A0Low Memory Aborts:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 7 > =A0=A0=A0=A0Free on Write:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 123 > =A0=A0=A0=A0Writes While Full:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0R/W Clashes:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0= > =A0=A0=A0=A0Bad Checksums:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0 > =A0=A0=A0=A0IO Errors:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0SPA Mismatch:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 0 >=20 > L2 ARC Size: (Adaptive)=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 49= 7.91=A0=A0=A0 GiB > =A0=A0=A0=A0Header Size:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0.11%=A0=A0=A0= 558.83=A0=A0=A0 MiB >=20 > L2 ARC Evicts: > =A0=A0=A0=A0Lock Retries:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 6 > =A0=A0=A0=A0Upon Reading:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 0 >=20 > L2 ARC Breakdown:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 5.75=A0=A0= =A0 m > =A0=A0=A0=A0Hit Ratio:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 81.94%=A0=A0=A0= 4.71=A0=A0=A0 m > =A0=A0=A0=A0Miss Ratio:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 18.06%=A0=A0=A0= 1.04=A0=A0=A0 m > =A0=A0=A0=A0Feeds:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 235.04=A0=A0=A0 k >=20 > L2 ARC Buffer: > =A0=A0=A0=A0Bytes Scanned:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0=A0=A0=A0 Bytes > =A0=A0=A0=A0Buffer Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0List Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0NULL List Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 >=20 > L2 ARC Writes: > =A0=A0=A0=A0Writes Sent:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 100.00%=A0=A0= =A0 22.67=A0=A0=A0 k >=20 > -----------------------------------------------------------------------= - >=20 > With the FreeBSD ZFS (without persistent L2ARC) I never got more > than 20% hit ratio on the L2ARC between reboots. >=20 > Quite a number of sysctl variable names have changed, and the port > needs to be adapted to the new names (therefore there are lots of 0 > values in the -L output). >=20 > The following names used by zfs-stats do not exist in OpenZFS: >=20 > kstat.zfs.misc.arcstats.recycle_miss > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned > kstat.zfs.misc.arcstats.l2_write_buffer_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter > kstat.zfs.misc.arcstats.l2_write_full > kstat.zfs.misc.arcstats.l2_write_in_l2 > kstat.zfs.misc.arcstats.l2_write_io_in_progress > kstat.zfs.misc.arcstats.l2_write_not_cacheable > kstat.zfs.misc.arcstats.l2_write_passed_headroom > kstat.zfs.misc.arcstats.l2_write_pios > kstat.zfs.misc.arcstats.l2_write_spa_mismatch > kstat.zfs.misc.arcstats.l2_write_trylock_fail > kstat.zfs.misc.arcstats.l2_writes_hdr_miss > vfs.zfs.vdev.cache.size >=20 > The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size > can be used to detect OpenZFS, and is easily fixed. >=20 > But the above listed L2ARC values seem to have been removed from or > have never existed in OpenZFS, and I did not find any substitutes. >=20 > Are there any plans to re-create them in OpenZFS on FreeBSD or are > they gone for good? >=20 > I'd like to update the zfs-stats port for compatibil=EDty with OpenZFS = =2E.. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" I think the kstats you are looking for are: kstat.zfs.misc.arcstats.l2_asize: 0 kstat.zfs.misc.arcstats.l2_size: 0 l2_asize is the 'allocated' size, that will be how much actual disk space is used. And l2_size will be the amount of data contained, so compression may make this quite a bit larger than the size of the physical device. --=20 Allan Jude --m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA-- --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfV8TjAAoJEBmVNT4SmAt+UsAQAMHOPL/VVqRg1UlNzlHWR5MB aAALic6r+kRG1uI2sFHuLJFEW0KDNmSV0wS2yZGIoXdmEmGSGtk2vPZtdMGW8e4Y BCNY36zCaC9HPc5R6nL+PFdV2U0CU46WEtCQy51ESotjQxNR+EkVnETuRLyaDTMY vqAgs33ue/fVKSot8UfgYedjJlT8F1CtRGuby3cClZUr7HmBesJrstZoIv+0NJ8H MQhGhOvuSeeLQz81LOVsC9nj1PcAtAHBJbbffAhC0wLh6rQC92OaFNUQFj+4BXpm 7w3JzAbUi/+MpY8xIFNkxuYAQ9ve3KBAy7EtHFWY/vCNZIkoM8XSuCPilFFXiNLE paqZ2WCutckXpc5J3lYZIFWiEMgjuSUhut5IZaGd0CBa4AqGT2q1BPFxMndLehaP b5Aq8cAKIKRxL3++6vQgO7KUVAF2QBcrkI6GSUd5stE7NKPqi+5panCkQwd5AGFP NFSAfNIQrKQ3bl0pVq3oyPnxjTsnKvlpvRLXsJZhZC99jxx7qa+eXyUdr1HC9kLy VnP5sd3Xk37lce5UnQKa+pLHkR5JrWmvPiHXT24+aQJxhZfI546qfdOFnztzs68Z iJAYXAy7qINXRIscc0ytiaSabr77dlvgBAIkCu6VGrhAZMLQtgnh5nr6fXeND81W IG3TavEXUNFKfXSByYwn =YTk9 -----END PGP SIGNATURE----- --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD-- From owner-freebsd-fs@freebsd.org Tue Sep 8 20:18:56 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 335543D658F; Tue, 8 Sep 2020 20:18:56 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmGgy4j8zz4P8X; Tue, 8 Sep 2020 20:18:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 088KId9N044002 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 8 Sep 2020 23:18:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 088KId9N044002 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 088KIde6044001; Tue, 8 Sep 2020 23:18:39 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 8 Sep 2020 23:18:39 +0300 From: Konstantin Belousov To: freebsd-current@freebsd.org, freebsd-fs@freebsd.org, freebsd-ppc@freebsd.org, mmacy@freebsd.org Subject: Re: Last ZFS upgrade (r365347) breaks booting Message-ID: <20200908201839.GR94807@kib.kiev.ua> References: <20200908193918.GB21279@KGPE-D16> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200908193918.GB21279@KGPE-D16> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4BmGgy4j8zz4P8X X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [1.45 / 15.00]; ARC_NA(0.00)[]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; NEURAL_SPAM_MEDIUM(0.36)[0.360]; NEURAL_SPAM_SHORT(0.63)[0.631]; NEURAL_SPAM_LONG(0.45)[0.454]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-current,freebsd-fs,freebsd-ppc]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 20:18:56 -0000 On Tue, Sep 08, 2020 at 09:39:18PM +0200, Piotr Kubaj via freebsd-ppc wrote: > I'm currently on r365449 on powerpc64. I use ZFS on / with zfs.ko compiled in kernel (because there's no loader on PowerNV systems). > > There seems to be a regression that happened recently, probably in r365347 (although I can't bisect it). Booting on my system, both with older kernels and the newest I have, I'm getting: > exec /sbin/init: error 8 > exec /sbin/init.bak: error 8 > exec /rescue/init: error 8 > init: not found in path /sbin/init:/sbin/oinit:/sbin/init.bak:/rescue/init > panic: no init > > This would suggest that the regression happened in the userspace. I can confirm when running livecd that there is /sbin/init installed. > ZFS, when loaded from that image (https://download.freebsd.org/ftp/snapshots/powerpc/powerpc64/ISO-IMAGES/13.0/FreeBSD-13.0-CURRENT-powerpc-powerpc64-20200903-c122cf32f2a-disc1.iso) seems to work fine. Just in case, check r365433. From owner-freebsd-fs@freebsd.org Wed Sep 9 02:12:05 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 30D893E0327 for ; Wed, 9 Sep 2020 02:12:05 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from echo.brtsvcs.net (echo.brtsvcs.net [208.111.40.118]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmQWS2sH7z4nRn for ; Wed, 9 Sep 2020 02:12:04 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from chombo.houseloki.net (unknown [IPv6:2602:41:642b:600::6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "chombo.houseloki.net", Issuer "brtsvcs.net CA" (verified OK)) by echo.brtsvcs.net (Postfix) with ESMTPS id A8F0938D25 for ; Wed, 9 Sep 2020 02:12:02 +0000 (UTC) Received: from [IPv6:2602:41:642b:630:dd2a:ba9e:2d4a:7c5f] (unknown [IPv6:2602:41:642b:630:dd2a:ba9e:2d4a:7c5f]) by chombo.houseloki.net (Postfix) with ESMTPSA id 612111E7 for ; Tue, 8 Sep 2020 19:12:01 -0700 (PDT) From: Mel Pilgrim Subject: Is an HP P812 (ciss(4) RAID) usable for ZFS? To: freebsd-fs@freebsd.org Message-ID: Date: Tue, 8 Sep 2020 19:12:00 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BmQWS2sH7z4nRn X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of list_freebsd@bluerosetech.com designates 208.111.40.118 as permitted sender) smtp.mailfrom=list_freebsd@bluerosetech.com X-Spamd-Result: default: False [-0.90 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.88)[-0.885]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; NEURAL_HAM_SHORT(-0.30)[-0.297]; DMARC_NA(0.00)[bluerosetech.com]; NEURAL_HAM_MEDIUM(-0.42)[-0.415]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:36236, ipnet:208.111.40.0/24, country:US]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 02:12:05 -0000 [Posting this here as this list gets seen by many ZFS-aware eyeballs, but if there is a better list please tell me.] I've gotten my hands on a couple of HP Smart Array P812 RAID controllers, but I can't tell if they'll work for ZFS. I know they're supported by ciss(4), but queries have gotten me conflicting answers about HBA/JBOD capability, including that HBA mode might be OS-dependent. These do support creating single-disk RAID0 plexes, but I'd rather not have RAID logic in the path at all if I can avoid it. Does anyone know if these will work as HBAs for ZFS on FreeBSD? If not, has anyone used the stated workaround and gotten good results? From owner-freebsd-fs@freebsd.org Wed Sep 9 06:46:12 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F1883E6863; Wed, 9 Sep 2020 06:46:12 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmXbm2MSgz3YxY; Wed, 9 Sep 2020 06:46:12 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MacBook-Pro-449.fritz.box (p200300cd5f236a00615dc138489ae2a2.dip0.t-ipconnect.de [IPv6:2003:cd:5f23:6a00:615d:c138:489a:e2a2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 6D05B2C490; Wed, 9 Sep 2020 06:46:11 +0000 (UTC) (envelope-from se@freebsd.org) To: FreeBSD CURRENT , freebsd-fs@freebsd.org Cc: Matthew Macy , Allan Jude , Graham Perrin References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> From: Stefan Esser Subject: Re: OpenZFS and L2ARC Message-ID: Date: Wed, 9 Sep 2020 08:46:07 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.2.1 MIME-Version: 1.0 In-Reply-To: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ARtKBvQ3PD8ZctsHDZ0dtbb5dRgH8eFbC" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 06:46:12 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ARtKBvQ3PD8ZctsHDZ0dtbb5dRgH8eFbC Content-Type: multipart/mixed; boundary="QLWn3nEBcMkqHZ3Q91QTkhW669IrrfHWb"; protected-headers="v1" From: Stefan Esser To: FreeBSD CURRENT , freebsd-fs@freebsd.org Cc: Matthew Macy , Allan Jude , Graham Perrin Message-ID: Subject: Re: OpenZFS and L2ARC References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> In-Reply-To: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> --QLWn3nEBcMkqHZ3Q91QTkhW669IrrfHWb Content-Type: multipart/mixed; boundary="------------251E0D5F4C0D1E7299068885" Content-Language: de-DE This is a multi-part message in MIME format. --------------251E0D5F4C0D1E7299068885 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Am 09.09.20 um 00:45 schrieb Graham Perrin: > On 08/09/2020 08:43, Stefan Esser wrote: >> OpenZFS seems to work quite well for me, in general, but I have=20 >> questions regarding the L2ARC statistics. >> > =85 >=20 >> The sysutils/zfs-stats port reports the following values for >> this system, BTW: >> >> ----------------------------------------------------------------------= -- >> ZFS Subsystem Report=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Tue = Sep=A0 8 09:02:46 2020 >> ----------------------------------------------------------------------= -- >> > =85 >=20 >> >> Quite a number of sysctl variable names have changed, and the port >> needs to be adapted to the new names (therefore there are lots of 0 >> values in the -L output). >> >> The following names used by zfs-stats do not exist in OpenZFS: >> >> kstat.zfs.misc.arcstats.recycle_miss >> kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned >> kstat.zfs.misc.arcstats.l2_write_buffer_iter >> kstat.zfs.misc.arcstats.l2_write_buffer_list_iter >> kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter >> kstat.zfs.misc.arcstats.l2_write_full >> kstat.zfs.misc.arcstats.l2_write_in_l2 >> kstat.zfs.misc.arcstats.l2_write_io_in_progress >> kstat.zfs.misc.arcstats.l2_write_not_cacheable >> kstat.zfs.misc.arcstats.l2_write_passed_headroom >> kstat.zfs.misc.arcstats.l2_write_pios >> kstat.zfs.misc.arcstats.l2_write_spa_mismatch >> kstat.zfs.misc.arcstats.l2_write_trylock_fail >> kstat.zfs.misc.arcstats.l2_writes_hdr_miss >> vfs.zfs.vdev.cache.size >> >> The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size >> can be used to detect OpenZFS, and is easily fixed. >> >> But the above listed L2ARC values seem to have been removed from or >> have never existed in OpenZFS, and I did not find any substitutes. >> >> Are there any plans to re-create them in OpenZFS on FreeBSD or are >> they gone for good? >=20 > Recalling=20 > ,=20 > on 28/03/2020 15:17,28/03/2020 15:17, Allan Jude wrote: >=20 > >> =85 > >> > >> Basically 'arc' was converted to a subtree. > >> > >> We should add some backwards compat sysctls to cover some of > >> these renames etc so configs and scripts don't break etc. This is not possible for quite a number of sysctls, since there is no simple 1:1 mapping for many of them. And there is an annoyance that I had noticed before but now have tracked down: $ time sysctl kstat.zfs.misc.dbufs | wc 55327 2047031 16333472 real 0m16,446s user 0m0,055s sys 0m16,397s Somebody decided to put a complete list of dbufs under this sysctl and thus querying "kstat.zfs.misc" takes that long (16 seconds to generate 16 MB of output on my system), even if only a few other values in "kstat.zfs.misc" are needed. I do not know whether there is any chance to get that debug output moved out of the "misc", e.g. into a new "debug" sub-tree. I'm afraid, that on Linux there are scripts that expect it under this name. If it is not acceptable to the upstream, we should locally modify the sysctl tree to move that variable out of "misc", IMHO. (While not taking much time, "kstat.zfs.misc.dbgmsg" should also be relocted to a "debug" sub-tree, IMHO ...) zfs-stats needs tens of values from "misc", and if they are not all added individually to the Kstat array, this will limit the response time to any zfs-stats invocation. It is not too hard to add the new variables in zfs-stats and to adapt the calculations to derive meaningful values to display. But if it always takes 16 seconds to generate any output, I'm not likely to use it too often ... Regards, STefan --------------251E0D5F4C0D1E7299068885-- --QLWn3nEBcMkqHZ3Q91QTkhW669IrrfHWb-- --ARtKBvQ3PD8ZctsHDZ0dtbb5dRgH8eFbC Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAl9Yei8FAwAAAAAACgkQR+u171r99USZ UQgAthKKr1z5b3fQKi8LSvDREnf5W2r7QIbwb57u9jljbeD79jxAmJXcC950wyUbbQMMqQrqWSsi MRzo5Cnv8TyfsLMf2yPWXG6zLaYzztKAKr2mbpzoMhCLOsJhhe1cxRXB0WnosbZLg/FH/wYGwPZa /uCoGdomGJcr51AIbbdF8IvpQq3k5RdiONqVt1wXx7Z5XfF+P5eH5ju8lHEBDO8T/odmUQL+7D4W MaqCebR2q+Hy+I6RF8jn5W2LVzl6o5RLLp33ItemuvIghOuHuIkx/z3qfatTB/FvUdwA+scfw3us dQ3IR2tUhaCt06+kmlJ2vH/7it66AyQT2KviiU+ONQ== =DLIq -----END PGP SIGNATURE----- --ARtKBvQ3PD8ZctsHDZ0dtbb5dRgH8eFbC-- From owner-freebsd-fs@freebsd.org Wed Sep 9 08:30:14 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9FFCC3C9510; Wed, 9 Sep 2020 08:30:14 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmZvp3n7hz3gGh; Wed, 9 Sep 2020 08:30:14 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MacBook-Pro-449.fritz.box (p200300cd5f236a00615dc138489ae2a2.dip0.t-ipconnect.de [IPv6:2003:cd:5f23:6a00:615d:c138:489a:e2a2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id A1D612C3D8; Wed, 9 Sep 2020 08:30:13 +0000 (UTC) (envelope-from se@freebsd.org) From: Stefan Esser To: FreeBSD CURRENT , freebsd-fs@freebsd.org Cc: Matthew Macy , Allan Jude , Graham Perrin References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> Subject: Re: OpenZFS and L2ARC Message-ID: <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> Date: Wed, 9 Sep 2020 10:30:09 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="4MdWjVcNIaJybu17SUuzQTUZJiNC47eDU" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 08:30:14 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --4MdWjVcNIaJybu17SUuzQTUZJiNC47eDU Content-Type: multipart/mixed; boundary="psYTM27l8ECy8qY3oI4Tf2QWIP3Ki99ky"; protected-headers="v1" From: Stefan Esser To: FreeBSD CURRENT , freebsd-fs@freebsd.org Cc: Matthew Macy , Allan Jude , Graham Perrin Message-ID: <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> Subject: Re: OpenZFS and L2ARC References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> In-Reply-To: --psYTM27l8ECy8qY3oI4Tf2QWIP3Ki99ky Content-Type: multipart/mixed; boundary="------------0106056A6F82F876FECCBA97" Content-Language: en-US This is a multi-part message in MIME format. --------------0106056A6F82F876FECCBA97 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Am 09.09.20 um 08:46 schrieb Stefan Esser: > Am 09.09.20 um 00:45 schrieb Graham Perrin: >> Recalling=20 >> ,=20 >> on 28/03/2020 15:17,28/03/2020 15:17, Allan Jude wrote: >> >> =A0>> =85 >> =A0>> >> =A0>> Basically 'arc' was converted to a subtree. >> =A0>> >> =A0>> We should add some backwards compat sysctls to cover some of >> =A0>> these renames etc so configs and scripts don't break etc. >=20 > This is not possible for quite a number of sysctls, since there is > no simple 1:1 mapping for many of them. >=20 >=20 > And there is an annoyance that I had noticed before but now have > tracked down: >=20 > $ time sysctl kstat.zfs.misc.dbufs | wc > =A0=A0 55327 2047031 16333472 >=20 > real=A0=A0=A0 0m16,446s > user=A0=A0=A0 0m0,055s > sys=A0=A0=A0 0m16,397s >=20 > Somebody decided to put a complete list of dbufs under this sysctl > and thus querying "kstat.zfs.misc" takes that long (16 seconds to > generate 16 MB of output on my system), even if only a few other > values in "kstat.zfs.misc" are needed. >=20 > I do not know whether there is any chance to get that debug output > moved out of the "misc", e.g. into a new "debug" sub-tree. I'm afraid, > that on Linux there are scripts that expect it under this name. >=20 > If it is not acceptable to the upstream, we should locally modify the > sysctl tree to move that variable out of "misc", IMHO. (While not > taking much time, "kstat.zfs.misc.dbgmsg" should also be relocted to > a "debug" sub-tree, IMHO ...) >=20 > zfs-stats needs tens of values from "misc", and if they are not all > added individually to the Kstat array, this will limit the response > time to any zfs-stats invocation. >=20 > It is not too hard to add the new variables in zfs-stats and to > adapt the calculations to derive meaningful values to display. >=20 > But if it always takes 16 seconds to generate any output, I'm not > likely to use it too often ... Update: I have created a fork of zfs-stats to work on: https://github.com/stesser/zfs-stats Initial change is to work around the long delay mentioned above and to use the correct name for the vdev cache size variable and to display the size, data contents and the corresponding compression factor of the compressed L2ARC. I'll create pull requests to inform the upstream of these changes. --------------0106056A6F82F876FECCBA97-- --psYTM27l8ECy8qY3oI4Tf2QWIP3Ki99ky-- --4MdWjVcNIaJybu17SUuzQTUZJiNC47eDU Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFqEEo3HqZZwL7MgrcVMTR+u171r99UQFAl9YkpIFAwAAAAAACgkQR+u171r99UQi jQf/WwI9ynTm4OdJazZizYjcq3FCNO4P3usLKgWms3ogCuv8AyssrnmVDBi1/fqfpFqPmjcE7xyF AWgckWJInV40sAXUvTgoCyJDMbrKFPRks/baYJKtRRCS6O73MXHE3yxKnSLY36bXWzYxB9RO1KAP 95jvniY+xsBaLUwsKVl9cilCzxtr4GZWw7XcxWlutVzgGn2DYIoTFp+rd1TJn7Lz3os4NjCNh/6i 7H98SQ7ijrrjfVivDjT1ipQjmJAuoYEXuCDJbPU4e3oB/q3IiQlv3EUkNcdpTmUpb/hcZ0Th8nHL hDzVP1lm8dGDkl26NwKtPpKYcEx1BcgVEMzgm/s6Lw== =8eE1 -----END PGP SIGNATURE----- --4MdWjVcNIaJybu17SUuzQTUZJiNC47eDU-- From owner-freebsd-fs@freebsd.org Wed Sep 9 08:57:37 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C1DDE3C9CDD for ; Wed, 9 Sep 2020 08:57:37 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmbWN6CdKz3yRc for ; Wed, 9 Sep 2020 08:57:36 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 4BmbWF0YLPz6dRX; Wed, 9 Sep 2020 10:57:29 +0200 (CEST) Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id 0D6kSUkvkqgF; Wed, 9 Sep 2020 10:57:27 +0200 (CEST) Subject: Re: Is an HP P812 (ciss(4) RAID) usable for ZFS? To: Mel Pilgrim , freebsd-fs@freebsd.org References: From: Guido Falsi Autocrypt: addr=mad@madpilot.net; keydata= mQENBE+G+l0BCADi/WBQ0aRJfnE7LBPsM0G3m/m3Yx7OPu4iYFvS84xawmRHtCNjWIntsxuX fptkmEo3Rsw816WUrek8dxoUAYdHd+EcpBcnnDzfDH5LW/TZ4gbrFezrHPdRp7wdxi23GN80 qPwHEwXuF0X4Wy5V0OO8B6VT/nA0ADYnBDhXS52HGIJ/GCUjgqJn+phDTdCFLvrSFdmgx4Wl c0W5Z1p5cmDF9l8L/hc959AeyNf7I9dXnjekGM9gVv7UDUYzCifR3U8T0fnfdMmS8NeI9NC+ wuREpRO4lKOkTnj9TtQJRiptlhcHQiAlG1cFqs7EQo57Tqq6cxD1FycZJLuC32bGbgalABEB AAG0Hkd1aWRvIEZhbHNpIDxtYWRAbWFkcGlsb3QubmV0PokBOQQTAQgAIwIbAwIeAQIXgAUL CQgHAwUVCgkICwQWAgMBBQJS79AgAhkBAAoJEBrmhg5Wy9KTc0kH/RO64ORBlTbTHaUaOj8F Je5O5NU2Pt9Cyt5ZWBRvxntr1zPTJGKRPS9ihlIfqT4ZvEngQGp57EUyFbCpI0UWasTerImM tt5WACnGmCzUTB39UXx8Oy4b1EgWeTJQ747e/F1mQLXTNa6ijRBE9fYlTb4gAkPN88/wVV9v 3PZozKLTg16ghBzHM/P7Lk8L7clPEZChX1FTa/6eSt3nvzfCuTMZbBPJF/ph+q1KyPqRgVfh tyhu5dvgMoPz/ni41IfeSrkJTD5RXzdyGR9q4Z1NYeBsLkRjC4LxKAP5KqUsvlOUjKvO1byj ApYdMarol+IGkaSk9e3zVYAJkWKjn/ni8Xa5Ag0EUxB7QQEQAKFhrDceoPdK/IHDSmoj6SQY isvM7VdhcleS7E9DoEAVt7yMbf6HbbMVTTY6ckvwTWQssywLBXNVqxgc4WLJjzfUhgef+WE7 5M3+WFYlOVQLGZY/zEVgma1raYnOHNAOzeHLDmEXjbZP6vGAeDyBbGfQPpE7qGYZ7ubeT3Xw QO+PklcCrvOPj2ZPcAxGNS2xVU/LzONqCrJqLMJSIcCdsbiSP4G5PnDFHtMokaTY6OEr8OEQ fOAerhcHUa/z7Uu8YtmaqKH+QGkE/WEgaRqSiTnv0JOTD+DxehaqvoKPPZ++2NpCZMHB2i6A /xifmQwEiIjEXtcueBRzkNUQkxhqZyS13SrhocL9ydtaVPBzZatAEjUDDEJmAMLVFs45qfyh MiNapHJo2n3MW/E5omqCvEkDdWX/en3P7CK2TemeaDghMsgkNKax/z0wNo5UZCkOPOz0xpNi UilOVbkuezZZNg65741qee2lfXhQIaZ66yT7hphc/N/z3PIAtLeze4u1VR2EXAuZ2sWAdlKC NTlJMsaU/x70BV11Wd/ypnVzM68dfdQIIAj1iMFAD/lXGlEUmKXg5Ov2VQDlTntQoanCYrAg +8CttPzjrydgLZFq3hrtQmfc0se5yv1WHS69+BsUOG09RvvawUDZxUjW19kyeN9THaNRgow3 kSuArUp6zSmJABEBAAGJAR8EGAEIAAkFAlMQe0ECGwwACgkQGuaGDlbL0pMN5wgA4bCkX/qw EVC06ToeR6C2putmSWQMgpDaqrv65Hubo+QGmg2P4ewTYQQ4g6oYWS03qHxqVVWhKz7FjfrV +dH8qbCLfSgIcvdBha7ayGZVrsiuMLKGbw36fcmkZPpSDOfHcP0XH8Z+u9CWj0xUkTxAlZ/7 i6gYSUpG2JWNtdmE/X8VVEyXusCLwy0K0BI60A/4dRTIX3C4QKrJ3ZbUXegz70ynjHf+lQMZ 9IZKASoRMuS5FozPQh6abvmwZEPdf5I9riUElzvHrqJ8Bx0t3Pujdoth+yNHpnBxrtO8LkQd rQ58P0SwcaIX33T2U9pG8bhu5YVR88FQ8OQ0cEsPBpDncrkBDQRPhvpdAQgAsd6mrOq1GSZw lzRscNQa9W2WB/3Tj4ON4PL2e9B+hc9lT/ny2zB3agXu5wbsXTzwxgJpQT7hNHkCSckW98h3 HRjFfhZPNCgInuUGsjcNyVguQh+/47ckhph0s7U+6B4yNuIiqQZk4mo8WgCNj1YIihVmGWEs gDOwMaajbDYZ0r1/3GkKlYjOXeUuT/WgourrSR5oZJVNA/k4X2H7M3JUr1BSc32L7BJt8M7A ntul6k17J0L8GmkvLvTUtQTO+p+DYQMna2ngD3PbAvQRcbEGnkg9ABrdEF0Wp4Gx+gGGWsyF KlHvPdMtgWAy3JsS+rQapG6LoW3yUJpwpEpA86KdBwARAQABiQEfBCgBCAAJBQJTEH0NAh0B AAoJEBrmhg5Wy9KTMZcIAMSsidGF4KpjGcKzhkNK0sEpevcelQ6DzgT7kcXuq6LQ6YOrbof2 /KPgGie9/ToFZfJXH8zE5GefqkKvHZbYssWilFvkI90F9n138kG205NB/2zlaQb74/v9ZMXJ XcipnIx+T2tOMCBgHJU41IMJmB+NfRt5A6CDytJdhWxqppsEo5jjy/7tJM1Nn47G87tAV8qV NUtzbS6zdnbHB4W2BJwCObbVv8epL3hu/L5efV2j2tSbVTmyvK/ClYMBqdtUo3uPX75GF/Ku YDCOP1BTA5zzmzp4PMVd+gmHcMgCZKY6lvcEtdi5FLI0we2kcY8ffPvM2d6MNhFsGLaVI95J 0oqJAR8EGAECAAkFAk+G+l0CGwwACgkQGuaGDlbL0pM18Qf9HTNNhu8N0ISKtmR8lgPhJuu8 9rOEa8KKEatr4fQ7gL+hmYOEqZ/yHLcPQvGxbAlLR7F0SheKvAEk4B1aFwGULPo0SzuO0d/W tVMEbGa95JTm/6mfiymWMlWf8UifD1MDKzzPR7Om0ybeoPM8S/RQTboUU1WLpwd4mg9pVJlK 0xr55GOSHNf4m7S+P1kvl3xgmEj14zVMq9yJBNWFlsQK5ciifh7sFpfuxWdEVbtgIdxpzImK LXSLA0vOroKAvxFTGBrBq3vxV6eUmaKyd5HbbWejmafY1ua5dcnew9lxpWKLdqkC27Vt0Cku +LtTY3325V+BChncwNcJJS7IMmBz6w== Message-ID: <3031306c-ac32-2718-929b-209edd7fe7f8@madpilot.net> Date: Wed, 9 Sep 2020 10:57:26 +0200 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4BmbWN6CdKz3yRc X-Spamd-Bar: / X-Spamd-Result: default: False [-0.89 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[madpilot.net:s=bjowvop61wgh]; NEURAL_HAM_MEDIUM(-0.97)[-0.970]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MISSING_MIME_VERSION(2.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-0.98)[-0.975]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[madpilot.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[madpilot.net,quarantine]; NEURAL_HAM_SHORT(-0.94)[-0.942]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 08:57:37 -0000 On 09/09/20 04:12, Mel Pilgrim wrote: > [Posting this here as this list gets seen by many ZFS-aware eyeballs, > but if there is a better list please tell me.] > > I've gotten my hands on a couple of HP Smart Array P812 RAID > controllers, but I can't tell if they'll work for ZFS. > > I know they're supported by ciss(4), but queries have gotten me > conflicting answers about HBA/JBOD capability, including that HBA mode > might be OS-dependent. These do support creating single-disk RAID0 > plexes, but I'd rather not have RAID logic in the path at all if I can > avoid it. > > Does anyone know if these will work as HBAs for ZFS on FreeBSD? If not, > has anyone used the stated workaround and gotten good results? I have used such controllers around 6-8 years ago. The only viable solution I found was configure them as single drive RAIDS and then configure ZFS on those. It does work, but I can't say if it affects performance or not. It sure affects the ability to mount such a ZFS pool on another machine with a different controller. If you can get a real HBA in place of it it would be better. My experience, though, is that HP technicians and salesmen will do ANYTHING to prevent you not using their smartarray technology and have a sincere dislike for software RAID solutions (less lockin I guess). -- Guido Falsi From owner-freebsd-fs@freebsd.org Wed Sep 9 19:26:26 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9F393D9F65; Wed, 9 Sep 2020 19:26:26 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmsSy50mSz3S6Q; Wed, 9 Sep 2020 19:26:26 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-274.local (unknown [IPv6:2601:648:8681:1cb0:1577:6de4:8b12:1f92]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: jhb) by smtp.freebsd.org (Postfix) with ESMTPSA id E8DC21197D; Wed, 9 Sep 2020 19:26:25 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Subject: Re: OpenZFS and L2ARC To: Stefan Esser , FreeBSD CURRENT , freebsd-fs@freebsd.org Cc: Matthew Macy , Allan Jude , Graham Perrin References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> From: John Baldwin Autocrypt: addr=jhb@FreeBSD.org; keydata= mQGiBETQ+XcRBADMFybiq69u+fJRy/0wzqTNS8jFfWaBTs5/OfcV7wWezVmf9sgwn8TW0Dk0 c9MBl0pz+H01dA2ZSGZ5fXlmFIsee1WEzqeJzpiwd/pejPgSzXB9ijbLHZ2/E0jhGBcVy5Yo /Tw5+U/+laeYKu2xb0XPvM0zMNls1ah5OnP9a6Ql6wCgupaoMySb7DXm2LHD1Z9jTsHcAQMD /1jzh2BoHriy/Q2s4KzzjVp/mQO5DSm2z14BvbQRcXU48oAosHA1u3Wrov6LfPY+0U1tG47X 1BGfnQH+rNAaH0livoSBQ0IPI/8WfIW7ub4qV6HYwWKVqkDkqwcpmGNDbz3gfaDht6nsie5Z pcuCcul4M9CW7Md6zzyvktjnbz61BADGDCopfZC4of0Z3Ka0u8Wik6UJOuqShBt1WcFS8ya1 oB4rc4tXfSHyMF63aPUBMxHR5DXeH+EO2edoSwViDMqWk1jTnYza51rbGY+pebLQOVOxAY7k do5Ordl3wklBPMVEPWoZ61SdbcjhHVwaC5zfiskcxj5wwXd2E9qYlBqRg7QeSm9obiBCYWxk d2luIDxqaGJARnJlZUJTRC5vcmc+iGAEExECACAFAkTQ+awCGwMGCwkIBwMCBBUCCAMEFgID AQIeAQIXgAAKCRBy3lIGd+N/BI6RAJ9S97fvbME+3hxzE3JUyUZ6vTewDACdE1stFuSfqMvM jomvZdYxIYyTUpC5Ag0ERND5ghAIAPwsO0B7BL+bz8sLlLoQktGxXwXQfS5cInvL17Dsgnr3 1AKa94j9EnXQyPEj7u0d+LmEe6CGEGDh1OcGFTMVrof2ZzkSy4+FkZwMKJpTiqeaShMh+Goj XlwIMDxyADYvBIg3eN5YdFKaPQpfgSqhT+7El7w+wSZZD8pPQuLAnie5iz9C8iKy4/cMSOrH YUK/tO+Nhw8Jjlw94Ik0T80iEhI2t+XBVjwdfjbq3HrJ0ehqdBwukyeJRYKmbn298KOFQVHO EVbHA4rF/37jzaMadK43FgJ0SAhPPF5l4l89z5oPu0b/+5e2inA3b8J3iGZxywjM+Csq1tqz hltEc7Q+E08AAwUIAL+15XH8bPbjNJdVyg2CMl10JNW2wWg2Q6qdljeaRqeR6zFus7EZTwtX sNzs5bP8y51PSUDJbeiy2RNCNKWFMndM22TZnk3GNG45nQd4OwYK0RZVrikalmJY5Q6m7Z16 4yrZgIXFdKj2t8F+x613/SJW1lIr9/bDp4U9tw0V1g3l2dFtD3p3ZrQ3hpoDtoK70ioIAjjH aIXIAcm3FGZFXy503DOA0KaTWwvOVdYCFLm3zWuSOmrX/GsEc7ovasOWwjPn878qVjbUKWwx Q4QkF4OhUV9zPtf9tDSAZ3x7QSwoKbCoRCZ/xbyTUPyQ1VvNy/mYrBcYlzHodsaqUDjHuW+I SQQYEQIACQUCRND5ggIbDAAKCRBy3lIGd+N/BCO8AJ9j1dWVQWxw/YdTbEyrRKOY8YZNwwCf afMAg8QvmOWnHx3wl8WslCaXaE8= Message-ID: <1245cf2f-7e70-df12-a69f-11ff6f9ed5ac@FreeBSD.org> Date: Wed, 9 Sep 2020 12:26:24 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 19:26:26 -0000 On 9/9/20 1:30 AM, Stefan Esser wrote: > Am 09.09.20 um 08:46 schrieb Stefan Esser: >> Am 09.09.20 um 00:45 schrieb Graham Perrin: >>> Recalling >>> , >>> on 28/03/2020 15:17,28/03/2020 15:17, Allan Jude wrote: >>> >>>  >> … >>>  >> >>>  >> Basically 'arc' was converted to a subtree. >>>  >> >>>  >> We should add some backwards compat sysctls to cover some of >>>  >> these renames etc so configs and scripts don't break etc. >> >> This is not possible for quite a number of sysctls, since there is >> no simple 1:1 mapping for many of them. >> >> >> And there is an annoyance that I had noticed before but now have >> tracked down: >> >> $ time sysctl kstat.zfs.misc.dbufs | wc >>    55327 2047031 16333472 >> >> real    0m16,446s >> user    0m0,055s >> sys    0m16,397s >> >> Somebody decided to put a complete list of dbufs under this sysctl >> and thus querying "kstat.zfs.misc" takes that long (16 seconds to >> generate 16 MB of output on my system), even if only a few other >> values in "kstat.zfs.misc" are needed. >> >> I do not know whether there is any chance to get that debug output >> moved out of the "misc", e.g. into a new "debug" sub-tree. I'm afraid, >> that on Linux there are scripts that expect it under this name. >> >> If it is not acceptable to the upstream, we should locally modify the >> sysctl tree to move that variable out of "misc", IMHO. (While not >> taking much time, "kstat.zfs.misc.dbgmsg" should also be relocted to >> a "debug" sub-tree, IMHO ...) >> >> zfs-stats needs tens of values from "misc", and if they are not all >> added individually to the Kstat array, this will limit the response >> time to any zfs-stats invocation. >> >> It is not too hard to add the new variables in zfs-stats and to >> adapt the calculations to derive meaningful values to display. >> >> But if it always takes 16 seconds to generate any output, I'm not >> likely to use it too often ... > > Update: I have created a fork of zfs-stats to work on: > > https://github.com/stesser/zfs-stats > > Initial change is to work around the long delay mentioned above and to > use the correct name for the vdev cache size variable and to display > the size, data contents and the corresponding compression factor of the > compressed L2ARC. > > I'll create pull requests to inform the upstream of these changes. A simple fix might be to use CTLFLAG_SKIP so that you only invoke the expensive sysctls if you request them by name, but not if you request the 'kstat.zfs' tree. -- John Baldwin From owner-freebsd-fs@freebsd.org Wed Sep 9 19:26:59 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 084EE3DA49C for ; Wed, 9 Sep 2020 19:26:59 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmsTY2y4Kz3S6M for ; Wed, 9 Sep 2020 19:26:57 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 3DAB74004B for ; Wed, 9 Sep 2020 21:26:47 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 286FF40049; Wed, 9 Sep 2020 21:26:47 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:bd92:fe10:aefc:1466] (unknown [IPv6:2001:9b1:28ff:d901:bd92:fe10:aefc:1466]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 435C54004D; Wed, 9 Sep 2020 21:26:45 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: Is an HP P812 (ciss(4) RAID) usable for ZFS? From: Peter Eriksson In-Reply-To: Date: Wed, 9 Sep 2020 21:26:44 +0200 Cc: Mel Pilgrim Content-Transfer-Encoding: quoted-printable Message-Id: <59749CE0-756D-46C5-8B4B-0A2F49BFAF46@lysator.liu.se> References: To: FreeBSD FS X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 4BmsTY2y4Kz3S6M X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-2.04 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; ARC_NA(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.996]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.72)[-0.721]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; NEURAL_HAM_MEDIUM(-1.02)[-1.022]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2020 19:26:59 -0000 You really want the HP Smart (ciss driver) controllers to run in HBA = mode when using ZFS. However, only the newer ones does support that mode = (as far as I know). I=E2=80=99m using the HP Smart HBA H241 controller cards and they work = pretty well. You=E2=80=99ll need to patch the ciss driver if you ever = try to connect more than 48 drives to one though: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246279 This web page = https://community.spiceworks.com/topic/2188239-freenas-on-dl180-g6-and-sma= rt-array-p812-in-hba-mode claims that P812 & P410 can _not_ be put into HBA mode P822 & P420 CAN be put into HBA mode My personal experience is with the P400 (can NOT be put into HBA mode) = and the H241 (works). So your only choice is probably to create single-disk LUNs=E2=80=A6 = (What I=E2=80=99m doing on the P400 - but that=E2=80=99s a pain=E2=80=A6) =E2=80=94 Peter Eriksson > On 9 Sep 2020, at 04:12, Mel Pilgrim = wrote: >=20 > [Posting this here as this list gets seen by many ZFS-aware eyeballs, = but if there is a better list please tell me.] >=20 > I've gotten my hands on a couple of HP Smart Array P812 RAID = controllers, but I can't tell if they'll work for ZFS. >=20 > I know they're supported by ciss(4), but queries have gotten me = conflicting answers about HBA/JBOD capability, including that HBA mode = might be OS-dependent. These do support creating single-disk RAID0 = plexes, but I'd rather not have RAID logic in the path at all if I can = avoid it. >=20 > Does anyone know if these will work as HBAs for ZFS on FreeBSD? If = not, has anyone used the stated workaround and gotten good results? > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Thu Sep 10 07:09:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ACB183ECE76; Thu, 10 Sep 2020 07:09:36 +0000 (UTC) (envelope-from se@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn94J3JfHz4Z1d; Thu, 10 Sep 2020 07:09:36 +0000 (UTC) (envelope-from se@freebsd.org) Received: from Stefans-MacBook-Pro-449.fritz.box (p200300cd5f07a50069dad6b59fcbe266.dip0.t-ipconnect.de [IPv6:2003:cd:5f07:a500:69da:d6b5:9fcb:e266]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 49AAC1746A; Thu, 10 Sep 2020 07:09:35 +0000 (UTC) (envelope-from se@freebsd.org) To: John Baldwin Cc: Matthew Macy , Allan Jude , Graham Perrin , FreeBSD CURRENT , freebsd-fs@freebsd.org References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> <1245cf2f-7e70-df12-a69f-11ff6f9ed5ac@FreeBSD.org> From: Stefan Esser Subject: Re: OpenZFS and L2ARC Message-ID: <4cc21896-b7b7-e126-c0a3-f584d1c81533@freebsd.org> Date: Thu, 10 Sep 2020 09:09:32 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.2.1 MIME-Version: 1.0 In-Reply-To: <1245cf2f-7e70-df12-a69f-11ff6f9ed5ac@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="est1qLY507OfMAyXLM6UQLB79MaCG4LW7" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 07:09:36 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --est1qLY507OfMAyXLM6UQLB79MaCG4LW7 Content-Type: multipart/mixed; boundary="u1nIn2YbKjACPxZOQ3qF8KiSY91mcjWJt"; protected-headers="v1" From: Stefan Esser To: John Baldwin Cc: Matthew Macy , Allan Jude , Graham Perrin , FreeBSD CURRENT , freebsd-fs@freebsd.org Message-ID: <4cc21896-b7b7-e126-c0a3-f584d1c81533@freebsd.org> Subject: Re: OpenZFS and L2ARC References: <7d54dc30-b8b1-a127-ec39-9fb759c8a55d@gmail.com> <712aa75b-b8ce-1eb0-ea0e-db1c2b7cc0c2@freebsd.org> <1245cf2f-7e70-df12-a69f-11ff6f9ed5ac@FreeBSD.org> In-Reply-To: <1245cf2f-7e70-df12-a69f-11ff6f9ed5ac@FreeBSD.org> --u1nIn2YbKjACPxZOQ3qF8KiSY91mcjWJt Content-Type: multipart/mixed; boundary="------------B37FE6F29A17D6902E01BD5F" Content-Language: en-US This is a multi-part message in MIME format. --------------B37FE6F29A17D6902E01BD5F Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Am 09.09.20 um 21:26 schrieb John Baldwin:> A simple fix might be to use = CTLFLAG_SKIP so that you only invoke the > expensive sysctls if you request them by name, but not if you request > the 'kstat.zfs' tree. I have looked at /sys/contrib/openzfs/module/zfs/dbuf.c where I had assumed that the "kstat.zfs.misc.dbufs" sysctl node was created, but did not spot the location on a quick search. The kstat nodes are created by kstat_install() and AFAICT, there is no parameter that directly allows to create the sysctl node with CTLFLAG_SKIP, currently. This long delay affects sysctl -a and I'd really hope that it can be fixed in a way that suppresses this large debug output unless it is specifically requested by passing the full node name ... Regards, STefan --------------B37FE6F29A17D6902E01BD5F-- --u1nIn2YbKjACPxZOQ3qF8KiSY91mcjWJt-- --est1qLY507OfMAyXLM6UQLB79MaCG4LW7 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFqEEo3HqZZwL7MgrcVMTR+u171r99UQFAl9Z0SwFAwAAAAAACgkQR+u171r99UQ9 Gwf9Fs5doLd5Zk0tTIGPWgwq68NkyVcyRLtG4p03F4jv9SWYR+WkHNQPdrf00liKKiheHkpPjDwQ 3AwBDlWDHbU4LzC3Hp03yM2QzSwezWei/Uf8OFKaMGGxl3Z14tJ1BnjI9SXDeTPs0O0Axj3LuQKS 0cfLLpmOWTiKkJSQ6ts4GYLCwvd9ulEwZ8KXKWkwVZ8vL9PfqNRLjFprv0u6BRcgxALBAhMdCbsy JiaImX63109vg/aH3TxkpIWKXZpW08nNTJYBEB3Mv9zlLW+Umg4J39F7PCXrW7gGV2zEbScWDG1u MwNxUYzELpz+KewBgG3oypj/FJoN7dAUH8kOiZ2PyQ== =zK1s -----END PGP SIGNATURE----- --est1qLY507OfMAyXLM6UQLB79MaCG4LW7--