From owner-freebsd-fs@freebsd.org Sun Dec 23 01:52:01 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8FDE13441F1 for ; Sun, 23 Dec 2018 01:52:00 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from mailout.ifm.liu.se (mailout.ifm.liu.se [130.236.160.13]) by mx1.freebsd.org (Postfix) with ESMTP id D209283B04 for ; Sun, 23 Dec 2018 01:51:58 +0000 (UTC) (envelope-from peter@ifm.liu.se) Received: from [192.168.1.79] (h-99-23.A785.priv.bahnhof.se [158.174.99.23]) (authenticated bits=0) by mail.ifm.liu.se (8.15.2/8.14.4) with ESMTPSA id wBN1p8LL006630 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 23 Dec 2018 02:51:09 +0100 (CET) From: Peter Eriksson Message-Id: <3F3EC02F-B969-43E3-B9B5-342504ED0962@ifm.liu.se> Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: {Disarmed} Re: {Disarmed} Re: Suggestion for hardware for ZFS fileserver Date: Sun, 23 Dec 2018 02:51:02 +0100 In-Reply-To: To: freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> X-Mailer: Apple Mail (2.3445.102.3) X-IFMLiUSE-MailScanner-Information: Please contact postmaster@ifm.liu.se more information X-IFMLiUSE-MailScanner-ID: wBN1p8LL006630 X-IFMLiUSE-MailScanner: Found to be clean X-IFMLiUSE-MailScanner-From: peter@ifm.liu.se X-Spam-Status: No X-Rspamd-Queue-Id: D209283B04 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of peter@ifm.liu.se designates 130.236.160.13 as permitted sender) smtp.mailfrom=peter@ifm.liu.se X-Spamd-Result: default: False [-1.16 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MV_CASE(0.50)[]; URI_COUNT_ODD(1.00)[5]; RCVD_IN_DNSWL_MED(-0.20)[13.160.236.130.list.dnswl.org : 127.0.11.2]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; MX_GOOD(-0.01)[e-mailfilter04.sunet.se,e-mailfilter03.sunet.se]; NEURAL_HAM_SHORT(-0.84)[-0.840,0]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-0.00)[country: SE(-0.02)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:2843, ipnet:130.236.0.0/16, country:SE]; MIME_TRACE(0.00)[0:+,1:+]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.930,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.977,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; HTTP_TO_IP(1.00)[]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2018 01:52:01 -0000 Can=E2=80=99t really give you some generic recommendations but on our Dell = R730xd and R740xd servers we use the Dell HB330 SAS HBA card, also know as = =E2=80=9CDell Storage Controller 12GB-SASHBA=E2=80=9D that uses the =E2=80= =9Cmpr=E2=80=9D device driver. This is an LSI3008 based controller and work= s really well. Only internal drives on the Dell servers though (730xd and 7= 40xd servers). Beware that this is not the same as the =E2=80=9CH330=E2=80= =9D RAID controller that Dell normally sells you. We had to do a =E2=80=9Cs= pecial=E2=80=9D in order to get the 10TB drives with 4K sectors with the HB= A330 controller since Dell only would sell use the 10TB drives together wit= h the H330 controller at the time we bought them. So we bought the HBA330:s= separately and swapped them ourself... And then we had to do a low-level r= eformat of all the disks since Dell by default would deliver them formatted= with a nonstandard sector size (4160 bytes I think, or perhaps 4112) and = =E2=80=9CProtection Information=E2=80=9D enabled (used and understood by th= e H330 controller, but not FreeBSD when using HBAs). But that=E2=80=99s eas= y to fix (just takes an hour or so per drive to do): # sg_format =E2=80=94size=3D4096 =E2=80=94fmtpinfo=3D0 /dev/da0 On our HP servers we use the HP Smart HBA H241 controller in HBA mode (set = via the BIOS configuration page) connected to external HP D6030 SAS shelfs = (70 disks per shelf). This is a HP special one that uses the =E2=80=9Cciss= =E2=80=9D driver. Also works fine. - Peter > On 22 Dec 2018, at 15:49, Sami Halabi wrote: >=20 > Hi, >=20 > What sas hba card do you recommend for 16/24 internal ports and 2 externa= l that are recognized and work well with freebsd ZFS. > Sami >=20 > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A =D7=A9=D7=91=D7=AA, 22 =D7=91=D7=93= =D7=A6=D7=9E=D7=B3 2018, 2:48, =D7=9E=D7=90=D7=AA Peter Eriksson >: >=20 >=20 > > On 22 Dec 2018, at 00:49, Rick Macklem > wrote: > >=20 > > Peter Eriksson wrote: > > [good stuff snipped] > >> This has caused some interesting problems=E2=80=A6 > >>=20 > >> First thing we noticed was that booting would take forever=E2=80=A6 Mo= unting the 20-100k >filesystems _and_ enabling them to be shared via NFS is= not done efficient at all (for >each filesystem it re-reads /etc/zfs/expor= ts (a couple of times) befor appending one >line to the end. Repeat 20-100,= 000 times=E2=80=A6 Not to mention the big kernel lock for >NFS =E2=80=9Chol= d all NFS activity while we flush and reinstalls all sharing information pe= r >filesystem=E2=80=9D being done by mountd=E2=80=A6 > > Yes, /etc/exports and mountd were implemented in the 1980s, when a dozen > > file systems would have been a large server. Scaling to 10,000 or more = file > > systems wasn't even conceivable back then. >=20 > Yeah, for a normal user with non-silly amounts of filesystems this is a n= on-issue. Anyway it=E2=80=99s the kind of issues that I kind of like to thi= nk about how to solve. It=E2=80=99s fun :-) >=20 >=20 > >> Wish list item #1: A BerkeleyDB-based =E2=80=99sharetab=E2=80=99 that = replaces the horribly >slow /etc/zfs/exports text file. > >> Wish list item #2: A reimplementation of mountd and the kernel interfa= ce to allow >a =E2=80=9Cdiff=E2=80=9D between the contents of the DB-based = sharetab above be input into the >kernel instead of the brute-force way it= =E2=80=99s done now.. > > The parser in mountd for /etc/exports is already an ugly beast and I th= ink > > implementing a "diff" version will be difficult, especially figuring ou= t what needs > > to be deleted. >=20 > Yeah, I tried to decode it (this summer) and I think I sort of got the ha= ng of it eventually.=20 >=20 >=20 > > I do have a couple of questions related to this: > > 1 - Would your case work if there was an "add these lines to /etc/expor= ts"? > > (Basically adding entries for file systems, but not trying to delet= e anything > > previously exported. I am not a ZFS guy, but I think ZFS just gene= rates another > > exports file and then gets mountd to export everything again.) >=20 > Yeah, the ZFS library that the zfs commands use just reads and updates th= e separate /etc/zfs/exports text file (and have mountd read both /etc/expor= ts and /etc/zfs/exports). The problem is that basically what it does when y= ou tell it to =E2=80=9Czfs mount -a=E2=80=9D (mount all filesystems in all = zpools) is a big (pseudocode): >=20 > For P in ZPOOLS; do > For Z in ZFILESYSTEMS-AND-SNAPSHOTS in $P; do > Mount $Z > If $Z Have =E2=80=9Csharenfs=E2=80=9D option; Then > Open /etc/zfs/exports > Read until you find a matching line, replace with the options, els= e if not found, Append options > Close /etc/zfs/exports > Signal mountd > (Which then opens /etc/exports and /etc/zfs/exports and does it= =E2=80=99s magic) > End > End > End >=20 > All wrapped up in a Solaris compatibility layer I libzfs. Actually I thin= k it even reads the /etc/zfs/exports file twice for each loop iteration due= to some abstractions. Btw things got really =E2=80=9Cfun=E2=80=9D when the= hourly snapshots we were taking (adding 10-20k new snapshots every hour, a= nd we didn=E2=80=99t expire them fast enough in the beginning) triggered th= e code above and that code took longer than 1 hour to execute - mountd was = 100% busy getting signalled and rereading, flushing and reinstalling export= s into the kernel all the time) and basically never finished. Luckily we di= dn=E2=80=99t have an NFS clients accessing the servers at that time :-) >=20 > This summer I wrote some code to instead use a Btree BerkeleyDB file and = modified the libzfs code and mountd daemon to instead use that database for= much faster lookups (no need to read the whole /etc/zfs/exports file all t= he time) and additions. Worked pretty well actually and wasn=E2=80=99t that= hard to add. Wanted to also add a possibility to add =E2=80=9Cexports=E2= =80=9D arguments =E2=80=9CSolaris=E2=80=9D-style so one could say things li= ke: >=20 > /export/staff vers=3D4,sec=3Dkrb5:krb5i:krb5p,rw=3DMailScanner = warning: numerical links are often malicious: 130.236.0.0/16,sec=3Dsys,ro= =3D130.236.160.0/24:10.1.2.3 >=20 > But I never finished that (solaris-style exports options) part=E2=80=A6. >=20 > We=E2=80=99ve lately been toying with putting the NFS sharing stuff into = separate =E2=80=9Cprivate" ZFS attribute (separate from official =E2=80=9Cs= harenfs=E2=80=9D one) and have another tool to read them instead and genera= te another =E2=80=9Cexports=E2=80=9D file so that file can be generated in = =E2=80=9Cone go=E2=80=9D and just signal mountd once after all filesystems = have been mounted. Unfortunately that would mean that they won=E2=80=99t be= shared until after all of them have been mounted but we think it would tak= e less time all-in-all. >=20 > We also modified the FreeBSD boot scripts so that we make sure to first m= ount all most important ZFS filesystems that is needed on the boot disks (n= ot just /) and then we mount (and share via NFS the rest in the background = so we can login to the machine as root early (no need for everything to hav= e been mounted before giving us a login prompt). >=20 > (Right now a reboot of the bigger servers take an hour or two before all = filesystems are mounted and exported). >=20 >=20 > > 2 - Are all (or maybe most) of these ZFS file systems exported with the= same > > arguments? > > - Here I am thinking that a "default-for-all-ZFS-filesystems" line= could be > > put in /etc/exports that would apply to all ZFS file systems no= t exported > > by explicit lines in the exports file(s). > > This would be fairly easy to implement and would avoid trying to h= andle > > 1000s of entries. >=20 > For us most have exactly the same exports arguments. (We set options on t= he top level filsystems (/export/staff, /export/students etc) and then all = home dirs inherit those. >=20 > > In particular, #2 above could be easily implemented on top of what is a= lready > > there, using a new type of line in /etc/exports and handling that as a = special > > case by the NFS server code, when no specific export for the file syste= m to the > > client is found. > >=20 > >> (I=E2=80=99ve written some code that implements item #1 above and it h= elps quite a bit. >Nothing near production quality yet though. I have looke= d at item #2 a bit too but >not done anything about it.) > > [more good stuff snipped] > > Btw, although I put the questions here, I think a separate thread discu= ssing > > how to scale to 10000+ file systems might be useful. (On freebsd-fs@ or > > freebsd-current@. The latter sometimes gets the attention of more devel= opers.) >=20 > Yeah, probably a good idea! >=20 > - Peter >=20 > > rick > >=20 > >=20 >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org " From owner-freebsd-fs@freebsd.org Sun Dec 23 08:46:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94A9813517CB for ; Sun, 23 Dec 2018 08:46:25 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 388C08FA9F for ; Sun, 23 Dec 2018 08:46:24 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm1-x331.google.com with SMTP id y185so17237062wmd.1 for ; Sun, 23 Dec 2018 00:46:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=t57qbnSVQ1zjeNgg9wPsLzxIC7jzqZUOarruzpy5wYU=; b=re3nXMSV/7ft+/FXp1H/6wHtDsPXXVpPjCb36KhHn8P6kvuapHpl3J7nEz75oMxCiY +CX45XsDvyDbVa6HaTbZHNJpZmFBaSPoT9zf6U3IkoNdJvRf1rO+Jb0X2tnYUuHYm5gg D4FxEygw+hbDxyfLKhrzxciP6VTVaR+QEVBpjBKex6GaC5/M39h8Jys+FSPAdW72+Rmn ojTfPKDcV70d+NiZ8djXSg4WlblE5JcODxvDf1VlAJXtw+aVHCgcmgYKvyfe7x1gf0NB iw8CclHI0a/g9aVB0GjUetotH6FUIyikgVpTwYXGp03/F8IOFbwHUwoefJlfJEL/qA8Z Eqiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=t57qbnSVQ1zjeNgg9wPsLzxIC7jzqZUOarruzpy5wYU=; b=IqPVCiypJkxdlJ7jH947knIoQJLkILFdx08aXRTBmm1s3m8x9z2jZy+rpkULz4RRR8 fGaJPTkJZICT97FhFsKJm6WetBaNR29OYKUScSTAxXVHKsHhTR1uOl/4MPgtkmuHcezK ql8OS75CjbnLj4AIJXCT1+JYCA+HSu9laAvlDKip4GcLwZubbC6ZVp9RIFM//Ff7eLkU OYRnmGIaraW6QA+bPagnAKmJ02PNcripyzZ1141Lm4DQj/brJuFHR9mxTpggfpU/mERs NFiPK6vNba6rNTta6PZDnsN4N6Xj+w05pRUZRTNm372iYxkgmqlGy6cadRs4RjObb9dJ gsQw== X-Gm-Message-State: AJcUukcEDQPK4/z2Fl7rEyLjmwGNY4Mxq14itQndp9VyyUANKkhGUxkz sODnkr5JQKHjAlRKXUbwj09U8Zhz X-Google-Smtp-Source: AFSGD/U6MQsUljtwxxtqw5ke0VFnP/Lm758KZhKBhXi59HQ1ygeogpchGwxzEqIEyN0/wtgnpEAf+Q== X-Received: by 2002:a1c:128b:: with SMTP id 133mr9085290wms.22.1545554782595; Sun, 23 Dec 2018 00:46:22 -0800 (PST) Received: from bens-mac.local (amontpellier-652-1-72-124.w109-210.abo.wanadoo.fr. [109.210.55.124]) by smtp.gmail.com with ESMTPSA id y34sm46554840wrd.68.2018.12.23.00.46.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 23 Dec 2018 00:46:21 -0800 (PST) From: Ben RUBSON Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: Suggestion for hardware for ZFS fileserver Date: Sun, 23 Dec 2018 09:46:20 +0100 References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> To: freebsd-fs@freebsd.org In-Reply-To: Message-Id: <6964F9CA-F3C8-488A-9C52-1E1676AAD175@gmail.com> X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 388C08FA9F X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=re3nXMSV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of benrubson@gmail.com designates 2a00:1450:4864:20::331 as permitted sender) smtp.mailfrom=benrubson@gmail.com X-Spamd-Result: default: False [-5.45 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.42)[-0.416,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.52)[ip: (-9.38), ipnet: 2a00:1450::/32(-1.68), asn: 15169(-1.46), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[1.3.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2018 08:46:25 -0000 > On 22 Dec 2018, Sami Halabi wrote: > > Hi, > > What sas hba card do you recommend for 16/24 internal ports and 2 external > that are recognized and work well with freebsd ZFS. > Sami No issue here with some LSI/Broadcom 9211-8i (LSI2008 based controller). Ben From owner-freebsd-fs@freebsd.org Sun Dec 23 10:30:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E34D135583A for ; Sun, 23 Dec 2018 10:30:59 +0000 (UTC) (envelope-from alexader@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0D1426D1AC for ; Sun, 23 Dec 2018 10:30:55 +0000 (UTC) (envelope-from alexader@leidinger.net) Received: from localhost (IO.Leidinger.net [192.168.1.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id 65CEBCD80; Sun, 23 Dec 2018 11:30:33 +0100 (CET) Date: Sun, 23 Dec 2018 11:30:31 +0100 From: Alexander Leidinger To: Rick Macklem Cc: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Message-ID: <20181223113031.00005150@Leidinger.net> In-Reply-To: References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Leidinger-net-MailScanner-Information: Please contact the ISP for more information X-Leidinger-net-MailScanner-ID: CAB3315507.A50D9 X-Leidinger-net-MailScanner: Found to be clean X-Leidinger-net-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-3, required 6, autolearn=not spam, ALL_TRUSTED -1.00, BAYES_00 -1.90, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10) X-Leidinger-net-MailScanner-From: alexader@leidinger.net X-Leidinger-net-MailScanner-Watermark: 1546165840.18832@pOB+g+IpQ6LyTfkIjRt1gQ X-Leidinger-net-Spam-Status: No X-Rspamd-Queue-Id: 0D1426D1AC X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.91 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; NEURAL_SPAM_SHORT(0.78)[0.784,0]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[Leidinger.net,quarantine]; MX_GOOD(-0.01)[mailgate.leidinger.net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.69)[asn: 34240(-3.43), country: DE(-0.01)]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2018 10:30:59 -0000 On Fri, 21 Dec 2018 23:49:58 +0000 Rick Macklem wrote: > Peter Eriksson wrote: > [good stuff snipped] > >This has caused some interesting problems… > > > >First thing we noticed was that booting would take forever… Mounting > >the 20-100k >filesystems _and_ enabling them to be shared via NFS is > >not done efficient at all (for >each filesystem it > >re-reads /etc/zfs/exports (a couple of times) befor appending one > >>line to the end. Repeat 20-100,000 times… Not to mention the big > >>kernel lock for >NFS “hold all NFS activity while we flush and > >>reinstalls all sharing information per >filesystem” being done by > >>mountd… > Yes, /etc/exports and mountd were implemented in the 1980s, when a > dozen file systems would have been a large server. Scaling to 10,000 > or more file systems wasn't even conceivable back then. There was an effort in 2012 to improve on that. https://sourceforge.net/projects/nfse/ I can't find code for this. In GNATS this was as kern/136865, but I can't find it in bugzilla. It was not picked up by us, no idea why (maybe nobody bothered back then). Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From owner-freebsd-fs@freebsd.org Sun Dec 23 21:00:32 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 922F31347AF5 for ; Sun, 23 Dec 2018 21:00:32 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2E6598DDA1 for ; Sun, 23 Dec 2018 21:00:32 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id E6B1D1347AF1; Sun, 23 Dec 2018 21:00:31 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C43141347AEF for ; Sun, 23 Dec 2018 21:00:31 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 586C28DD9B for ; Sun, 23 Dec 2018 21:00:31 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 9E13613163 for ; Sun, 23 Dec 2018 21:00:30 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBNL0UKt023537 for ; Sun, 23 Dec 2018 21:00:30 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBNL0UOH023527 for fs@FreeBSD.org; Sun, 23 Dec 2018 21:00:30 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201812232100.wBNL0UOH023527@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: fs@FreeBSD.org Subject: Problem reports for fs@FreeBSD.org that need special attention Date: Sun, 23 Dec 2018 21:00:30 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Dec 2018 21:00:32 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 203492 | mount_unionfs -o below causes panic New | 217062 | for file systems mounted with -o noexec, exec=off Open | 140068 | [smbfs] [patch] smbfs does not allow semicolon in Open | 144447 | [zfs] sharenfs fsunshare() & fsshare_main() non f Open | 211491 | System hangs after "Uptime" on reboot with ZFS Open | 221909 | [ZFS] Add a sysctl to toggle send_corrupt_data 6 problems total for which you should take action. From owner-freebsd-fs@freebsd.org Mon Dec 24 00:21:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BD932134FD00 for ; Mon, 24 Dec 2018 00:21:25 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660040.outbound.protection.outlook.com [40.107.66.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C19409578F for ; Mon, 24 Dec 2018 00:21:23 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM (10.169.142.146) by YQBPR01MB0529.CANPRD01.PROD.OUTLOOK.COM (10.169.143.135) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1446.24; Mon, 24 Dec 2018 00:21:21 +0000 Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c]) by YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c%8]) with mapi id 15.20.1446.026; Mon, 24 Dec 2018 00:21:14 +0000 From: Rick Macklem To: Alexander Leidinger CC: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Thread-Topic: Suggestion for hardware for ZFS fileserver Thread-Index: AQHUmU1yCLTPDjfxZEms4Ul8AJXDZ6WJb0QAgAAbMACAAE3yOYACSg6AgADmYwI= Date: Mon, 24 Dec 2018 00:21:13 +0000 Message-ID: References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> , <20181223113031.00005150@Leidinger.net> In-Reply-To: <20181223113031.00005150@Leidinger.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR01MB0529; 6:02y+JLlG+OkzTBOZupRCuzuaiIeMs1ywAwfBWIekX/2tTiXVevtlD7ZvYzCTe/mcxn7ILrp1rExlgRaefgJTWWsi2h0DKkn/dz1QGJVVzo1nFZ9PzoH0XQ640Lya5sDsquysxJ5VTnvaW1bXnz4fHX1BLSUWguVyE3PSxyNxIWBaTVnfoPJRn/oA5LC9YtouV+Q2rLOn3t6PEPdOxfkQxFAsfZlGISqBo6+itCW+Si4CApV4pJQ4i8W67dMiQscirsA7QEBe3oUJEiWXjlUo/oYU8dEFp+C8ibt6UBEHDKsb5LGz24r8TQTwJfj4fxqJXFGRnWCTxf8Rj0jcJs3C9DorBqOXyIx82J0fwuq3TX1orAeYmuysmENdvNhbt9snyAVtGKnONqZxgRhA04hXCDRKOwQOFfLEb2yKsAUq6yVpnWhyivXnYKVgHQjSDh9aDGjN23eag0G03qzTmcE01pTNyCvq5VRqiC2qyL4CehQ=; 5:gZya0TJ8L/nONDptbHcYXJzaNrkjT+rrHcJxVrnPq///lW4Eg9vpCgbp7iffoYhrHcuwAOYe+q0t8gusaX4W+31kTdm9RaMP+vn17PnCOBzJUCSEEcx0P9wgkYcB7G37TSZNO4cmpiW60jinZT55qSu+2W583zfGhmOD6MYrdEA=; 7:j1nazsoYVTHfNLnvwzTyvbvSBWS947wQY7QYJMg8ezH9xj7xLHz2vVvfOVK8dnWJbBifwL0L7iAxAhpbAgxrq0uu7fHuRbDm/T8uNb0/jZ08vC84plW1eVpowYFfU7QSWlj3EV/ggUflLJDGKrgnkQ== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: be09f2af-2f7d-42aa-e5d7-08d66935b6cb x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(2017052603328)(7153060)(7193020); SRVR:YQBPR01MB0529; x-ms-traffictypediagnostic: YQBPR01MB0529: x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(3230021)(908002)(999002)(5005026)(6040522)(2401047)(8121501046)(10201501046)(93006095)(93001095)(3002001)(3231475)(944501520)(52105112)(6041310)(201703131423095)(201703031522075)(201702281528075)(201702281529075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123562045)(20161123560045)(20161123558120)(201708071742011)(7699051)(76991095); SRVR:YQBPR01MB0529; BCL:0; PCL:0; RULEID:; SRVR:YQBPR01MB0529; x-forefront-prvs: 0896BFCE6C x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(136003)(366004)(346002)(39860400002)(376002)(189003)(199004)(105586002)(186003)(102836004)(6916009)(14454004)(99286004)(74316002)(5660300001)(6506007)(305945005)(93886005)(6436002)(478600001)(106356001)(11346002)(446003)(33656002)(786003)(54906003)(229853002)(316002)(296002)(46003)(9686003)(55016002)(6306002)(25786009)(256004)(2906002)(8936002)(4326008)(6246003)(53936002)(68736007)(97736004)(71200400001)(8676002)(476003)(71190400001)(86362001)(486006)(81156014)(76176011)(74482002)(81166006)(7696005); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR01MB0529; H:YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: omRCCf+CeELAztS5ArnEypzeINqnMHNjNvM5Tyvtt1ZkmzP6Y72dzqGSzCmQ8TVcBONygxh9hIm7FWy3pqoRyF671Oy+OZDZBQcGFBzd+VKInofO55UumWshbGWGT5kcYuQnQ4fhO5EVNZpbryXWM8ndbAR3vXu98/cts5PFLbeP5XiA38rhvfi6Woy1dcF9v10dkz0nUKtIMbkzLtrBeDToX8ba6Fp8shGh9Bihxib+Iu8yw2535INIiUfLHF2buOoREVkdqFZND7ssEnkiFKdTRj3/e/xxoEIwgmo9IscXS9iWW932fJz4VNCAKPyY spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: be09f2af-2f7d-42aa-e5d7-08d66935b6cb X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Dec 2018 00:21:13.9742 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR01MB0529 X-Rspamd-Queue-Id: C19409578F X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.40 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-2.03 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.940,0]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/17]; FROM_HAS_DN(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-0.97)[-0.972,0]; NEURAL_SPAM_SHORT(0.20)[0.196,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCVD_IN_DNSWL_NONE(0.00)[40.66.107.40.list.dnswl.org : 127.0.3.0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.66.107.40.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 00:21:26 -0000 Alexander Leidinger wrote: On Fri, 21 Dec 2018 23:49:58 +0000 Rick Macklem wrote: [stuff snipped] >There was an effort in 2012 to improve on that. >https://sourceforge.net/projects/nfse/ >I can't find code for this. > >In GNATS this was as kern/136865, but I can't find it in bugzilla. > >It was not picked up by us, no idea why (maybe nobody bothered back >then). The technical problem was that it did not support the "administrative contr= ols" exports (the ones that define which directories in a file system are mounta= ble via the NFSv3 Mount protocol). As such, replacing mountd with it would have been a major POLA violation. (Personally, I'd be happy to see these "administrative control exports" go away, but others disagree.) The author was not willing to make nfse compatible with mountd w.r.t. the a= bove. Then sometime later, the author posted that he didn't want anyone to use th= e code and was removing it from open source distribution. rick From owner-freebsd-fs@freebsd.org Mon Dec 24 23:13:59 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B76531351ECE for ; Mon, 24 Dec 2018 23:13:59 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55579701D4 for ; Mon, 24 Dec 2018 23:13:58 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-io1-xd2f.google.com with SMTP id k7so9837985iob.6 for ; Mon, 24 Dec 2018 15:13:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=T39k9a+kJCNuVzMUMCrAIyH97mMg8qkcXnvFjpuX6JU=; b=GjS94Dv9sSCj9DIGLJSF3NWdD7DvRkd9aPy1LRqeGeH4FVm9QAakfpxzDz51G6vt97 u2tosbD/sNqKcj90tSW0TmM+y+LSMFT5VMS+L8U0F5KJMgC9+XilU4gVCQ/OUC6CkXov DzU3JCrchoIgqUUGLyeZRdpcklnXKd2SlOMIDW+bZOA9u35T7kI0mbUL5nRU9VnGW0qk n8O/g5MjWVTqMW11dCfLn3MckWe+I+7mPYgxFBQn8OGKQ6NixNdcIe3gFOYK9VyYh+Tr U8BRxHHPNLKZv/Mc/IYxslGCLbEw/5ZE+DgzbedGkBowoykb+BS+TmsAOz3qDzP4M1Np N3Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=T39k9a+kJCNuVzMUMCrAIyH97mMg8qkcXnvFjpuX6JU=; b=sb3h5WYwJmJpc3tzQYZAo6yMRYkr9jyJB97c7eONqWjGvheyi2urp+YwJyyu6FVQY7 8TwD6PpgAJEOtbiHMHF5P7mv9H8qr8/FD8pTKqJzJBuqf+ST6bBCTooTRPSIiQb4T+IS CI/xsHJyAebiE1UoS+XuQm8hMac8Ws1I64hb5mFbgVt9td6VJGDHPaXc7GOXkttQWmfz 10S2H3QadCkhepXJqqiIDrB2FSbUOvAey2lYicMc5YO9uhLP6afdiXqJQdEnh+tQeE/d Ipfc8AQiPEcG31DfAnGnLC5CjskRdJka9vrnj7Xq375Ns3w3Vjk4fr+sbiUuMCA+Jgdj QC/Q== X-Gm-Message-State: AJcUukcvOboYdkctF/AZuZNmaUhZSLqyfwtUKdc2tU3dW0q+tVqGD/cD EsO7sSFWuhFCfYoCYGpOxhB9mw16hD/Ejzxt6wV/uHo= X-Google-Smtp-Source: ALg8bN7Y3HWEYGrUuLSxrMNem7GMlFYhasvs9wf1HI6f4BAYJ8XuybO3/DDF0V9fEsW+3/pbrys30v+0VKRNGon1g5k= X-Received: by 2002:a6b:3806:: with SMTP id f6mr10536369ioa.194.1545693236812; Mon, 24 Dec 2018 15:13:56 -0800 (PST) MIME-Version: 1.0 References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <20181223113031.00005150@Leidinger.net> In-Reply-To: From: Zaphod Beeblebrox Date: Mon, 24 Dec 2018 18:13:47 -0500 Message-ID: Subject: Re: Suggestion for hardware for ZFS fileserver To: freebsd-fs X-Rspamd-Queue-Id: 55579701D4 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=GjS94Dv9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2607:f8b0:4864:20::d2f as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-6.41 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[f.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-2.41)[ip: (-8.64), ipnet: 2607:f8b0::/32(-1.85), asn: 15169(-1.48), country: US(-0.08)]; NEURAL_HAM_SHORT(-0.99)[-0.985,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Dec 2018 23:14:00 -0000 [ regarding ZFS hardware thread ] There's another type of server --- the "ghetto" or home storage serer. For this server, I like to optimize for not loosing data, not for uptime. Going back a few years, there were consumer motherboards with 10 or 12 SATA onboard. Mostly, this was at the change of technologies ... so you had some of one kind of port and some of another. Used SAS HBAs are another option ... but they have a caviat: many SATA drives will eventually reject them under load. Good SATA drives won't (but again, we're talking a ghetto system). If you're taking WD reds (and not, say, seagate barracudas) ... these work well. On the seagates, however, I've had drives repeatedly fail ... only to go on working fine in a workstation with a SATA controller. Another source of ghetto ports are 4 port SATA controllers (~$50). Cheaper than SAS, but taking more PCIe ports. I've had a 16 drive system running on 10 motherboard ports and 2x 4 port SATA PCIe. Watch out for the 8 port ones --- the only 8 port SATA I've come accross are using a SATA expander for the last 5 ports !?!. Not a good value. The motherboards with 10 or so ports tend to be high end boards, so a ghetto system will want to get those on eBay used. ASUS ROG and ASUS TUF boards of the 4-ish generation (Intel) or the bulldozer (AMD) are configured this way. Having 10 + 8 gives you enough to have 2 SSD drives for cache/log or 2 boot drives. Get as much RAM as you can stuff in. 32G is often that number. You'll get a good GigE on the motherboard, but 10GE will need to be an addin card. You'll likely be getting short on slots. The next iteration of this hardware is the only one that will have a chance at ECC RAM. The cheapest threadripper motherboards with the basic threadripper CPU will be in this range used soon ... giving you upto 128G of ECC ram and many, many more PCIe lanes. Might make me reverse my use of SAS controllers and push me to all NAS drives. To be clear, with SATA controllers, cheap SATA drives seem to last fine. But the SAS controllers seem to spar with the cheap drive electronics (the drives smart test fine, but the electronics disconnect --- and if the SAS controller is swapped for a SATA one, this goes away). Last point. RAID-Z2 at a minimum. I could even see the argument for Z3. My current array is 16x 4T drvies in to 8 disk Z2 plexes. Of that, one plex is all WD Red on a SAS controller ... and the other (older) plex is still largely cheap drives on SATA. Right now, drives below 4T are artificially expensive. Drives right up to 10T are about the same price per G (at least here in Canada). This server (obviously) has many single points of failure. Watch Netflix when this happens :). It doesn't fail often (I haven't had more than a few hours downtime in 10-or-so-years) and the basic system can be had very cheaply. You don't need a recent CPU to fill GigE. I'm about to upgrade mine to 10GE ... but I realize that I may only get something less than 10GE from it ... some fraction... but more than GigE nonetheless... From owner-freebsd-fs@freebsd.org Tue Dec 25 00:02:18 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A7591353A72 for ; Tue, 25 Dec 2018 00:02:18 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C28572471 for ; Tue, 25 Dec 2018 00:02:17 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 0EC9E211339 for ; Mon, 24 Dec 2018 19:02:10 -0500 (EST) Received: from [192.168.10.30] (unknown [192.168.10.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 44C781212F8 for ; Mon, 24 Dec 2018 18:02:10 -0600 (CST) Subject: Re: Suggestion for hardware for ZFS fileserver To: freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <20181223113031.00005150@Leidinger.net> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <1d76f92c-6665-81ef-1b94-dc1b4b8925d1@denninger.net> Date: Mon, 24 Dec 2018 18:02:09 -0600 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms030505090906090109090306" X-Rspamd-Queue-Id: 9C28572471 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.16 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[px.denninger.net]; NEURAL_HAM_SHORT(-0.95)[-0.954,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.00)[ip: (-9.84), ipnet: 104.236.64.0/18(-3.28), asn: 14061(3.20), country: US(-0.08)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Dec 2018 00:02:18 -0000 This is a cryptographically signed message in MIME format. --------------ms030505090906090109090306 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 12/24/2018 17:13, Zaphod Beeblebrox wrote: > [ regarding ZFS hardware thread ] > > There's another type of server --- the "ghetto" or home storage serer. = For > this server, I like to optimize for not loosing data, not for uptime. > > Going back a few years, there were consumer motherboards with 10 or 12 = SATA > onboard. Mostly, this was at the change of technologies ... so you had= > some of one kind of port and some of another. Used SAS HBAs are anothe= r > option ... but they have a caviat: many SATA drives will eventually rej= ect > them under load. Good SATA drives won't (but again, we're talking a gh= etto > system). If you're taking WD reds (and not, say, seagate barracudas) .= =2E. > these work well. On the seagates, however, I've had drives repeatedly = fail > ... only to go on working fine in a workstation with a SATA controller.= I've run "ghetto mode" fileservers with the LSI adapters in IT mode (that always just seem to work) with one of their SFP ports connected to a SAS expander, and then fanned THAT out to SATA drives.=C2=A0 The only constraint is that you can run into problems booting from an expander, so don't -- use the ports on the HBA (or even the motherboard) for the boot drives. Never had a problem doing this with HGST drives, Intel SSDs and most others.=C2=A0 The Seagates I've had fail actually physically failed; they= didn't throw a protocol hissy fit on the bus.=C2=A0 I don't buy Seagates = any more as I've had too many die out-of-warranty for my taste.=C2=A0 They wo= rk fine with WD drives too.=C2=A0 Never had one of the drives that failed ca= use a cascade detach event either.=C2=A0 The last few years (five or so) for spinning rust HGST seems to sell the most-reliable stuff in my experience but YMMV on that. Those adapters and expanders are cheap these days.=C2=A0 The expanders us= ed to be expensive, but not any more -- there's a ton of them around on the secondary market for very little money (not much more than the LSI cards.)=C2=A0 Their only downside is they run hot so you need good fan coverage in the case. Older SuperMicro boards (X8DTL- series) that will take the 5600-series Westmere Xeon processors can be had for almost nothing (note you have to have the latest BIOS in them, which can be flashed, to run the Westmere processors), and the CPUs are a literal $25.=C2=A0 The only "gotcha" is y= ou need ECC memory, but if you can find it at a decent price used you're golden.=C2=A0 I would NOT run a ZFS filesystem without ECC memory; the ri= sk of undetected data corruption that you don't catch for months or even years is material and if it happens you WILL cry since your backup coverage may have expired at that point. > Last point. RAID-Z2 at a minimum. I could even see the argument for Z= 3. > My current array is 16x 4T drvies in to 8 disk Z2 plexes. Of that, one= > plex is all WD Red on a SAS controller ... and the other (older) plex i= s > still largely cheap drives on SATA. Right now, drives below 4T are > artificially expensive. Drives right up to 10T are about the same pric= e > per G (at least here in Canada). Yes, RaidZ2 is a (very) good idea. Oh, and keep a spare power supply in the building, plus a spare disk adapter and expander.=C2=A0 Hell, for $25 you can keep a spare CPU in the= building! :)=C2=A0 A box like this can trivially saturate more than one G= igE port (the SuperMicros typically have two on board plus the dedicated management port) all day long.=C2=A0 While there are failure points in su= ch a ghetto system config they're coverable (with a reboot after swapping the dead item) at very low cost in backstock in the building. I've had a couple of these in 24x7 field use for coming up on ten years now -- they're still running and have been dead-balls reliable -- duplicating them today is disgustingly inexpensive.=C2=A0 If there's one downside its that modern CPUs are FAR more power-efficient per-cycle than the older Xeons (they're much faster too, but that's not the limiting factor for a fileserver.) --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030505090906090109090306 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTgxMjI1MDAwMjA5 WjBPBgkqhkiG9w0BCQQxQgRAncWgPmWTxU0h09+qfelrnvtjiGpVMduaj0Dzmw80kgrHW6Os pOSVUmWTj/zKxiaRbYHHypEUA2HRpzN4gRMFyzBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgBWLb8QWo40fY5nEdWVjEKEQxAjZ5LDBfu9ZMXWEoOXEAn7SiSC5EQKPaR0G+bFSAN/ wLmq1misefVRKClBJmO/j5+PbbKYqnXAGoOZoZBPIofGbWZErd+7E9dHAtk9BlN1R7/iNHhk KHaqrXJkUqlCGb0GaAcV/r7cBx4BSjRTTE9Is8GLiEzi+dimoVOTL1I+575jFmfZj0aQGaJs h1jebChlZwS7O4G3xX+dSO8D9o9NGuGBQYzeJMWWF/ZXq67vRCqKeA9tjGvT3HUqI5MRueRB J/K9Xf0qULoAcz0KMvawoW95oxz/oR/oZtIy4AZAqHNZX0K8Y/VN7bMjot+LdG9+88FB9Wc3 ssmILj3YXJ22gK0MWiD9IS0LTk3WMVIt3MNjjO69fwcCQaeMZ5uDKYUjl5CHjQ/lYocQeXon mNfEFSVHYF5Z1xRWamR3QjQenrXbKWjDoUnlr/WsTx00l30ydM0IIQwO4dD9cuPS5ctB+bQk no06R4sHc7zQFiflxgABF+JbuBYZ/X7EZrTs0unig9I1Xl0UcQ6/NoYU78WWL1SthmD3afVn qrx5fkv5hU2XPYRafg8M+MYnaUVYUpeCW5r9Ja8vEE/oplr46no8IKdfXkJow8yyslbJ7n+E G7Vrk8zKijfbgis13OJHB4m3gU/DxX4q4Qcb81LP6AAAAAAAAA== --------------ms030505090906090109090306-- From owner-freebsd-fs@freebsd.org Thu Dec 27 10:24:23 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9AAD13578B0 for ; Thu, 27 Dec 2018 10:24:23 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 550186CA28 for ; Thu, 27 Dec 2018 10:24:23 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 9BD54B8EC9; Thu, 27 Dec 2018 11:24:19 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Vwf7iwmbjymr; Thu, 27 Dec 2018 11:24:18 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id C8EA4B8EC3; Thu, 27 Dec 2018 11:24:18 +0100 (CET) Subject: Re: Suggestion for hardware for ZFS fileserver To: Peter Eriksson , freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> From: Willem Jan Withagen Message-ID: <362ced1c-f39e-cc6d-cf50-aab878c6cd49@digiware.nl> Date: Thu, 27 Dec 2018 11:24:20 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 10:24:24 -0000 On 21/12/2018 19:53, Peter Eriksson wrote: Thanx, for the introduction to a serious platform running ZFS. Always interesting to read what other do with their sstuff. :-D But this can be "easily" fixed for the moment. > First thing we noticed was that booting would take forever… Mounting the 20-100k filesystems _and_ enabling them to be shared via NFS is not done efficient at all (for each filesystem it re-reads /etc/zfs/exports (a couple of times) befor appending one line to the end. Repeat 20-100,000 times… Not to mention the big kernel lock for NFS “hold all NFS activity while we flush and reinstalls all sharing information per filesystem” being done by mountd… > > Wish list item #1: A BerkeleyDB-based ’sharetab’ that replaces the horribly slow /etc/zfs/exports text file. > Wish list item #2: A reimplementation of mountd and the kernel interface to allow a “diff” between the contents of the DB-based sharetab above be input into the kernel instead of the brute-force way it’s done now.. > > (I’ve written some code that implements item #1 above and it helps quite a bit. Nothing near production quality yet though. I have looked at item #2 a bit too but not done anything about it.) It would be possible to not user /etc/zfs/exports. The syntax that can be inserted in that file through zfs itself is not quite flexible enough. I remember patching it, and then not getting this accepted, because it was too FreeBSD specific. So I started using /etc/exports. But cram all in /etc/exports, and only load and export files once zfs has loaded all its filesystems. Only speeds up so much, but keeps a few bottlenecks out of the equation. --WjW From owner-freebsd-fs@freebsd.org Thu Dec 27 10:54:26 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C32301358DB4 for ; Thu, 27 Dec 2018 10:54:25 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46F026DD88 for ; Thu, 27 Dec 2018 10:54:25 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id C6D22BAD43; Thu, 27 Dec 2018 11:54:15 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WR8iBsSt9sfI; Thu, 27 Dec 2018 11:54:14 +0100 (CET) Received: from [192.168.101.70] (unknown [192.168.101.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id DB7ECBAD34; Thu, 27 Dec 2018 11:54:14 +0100 (CET) Subject: Re: Suggestion for hardware for ZFS fileserver To: Sami Halabi Cc: freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> From: Willem Jan Withagen Message-ID: Date: Thu, 27 Dec 2018 11:54:16 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 10:54:26 -0000 On 22/12/2018 15:49, Sami Halabi wrote: > Hi, > > What sas hba card do you recommend for 16/24 internal ports and 2 external > that are recognized and work well with freebsd ZFS. There is no real advise here, but what I saw is that it is relatively easy to overload a lot of the busses involved int his. I got this when building Ceph clusters on FreeBSD, where each disk has its own daemon to hammer away on the platters. The first bottleneck is the disk "backplane". It you do not need to wire every disk with a dedicated HBA-disk cable, then you are sharing the bandwidth on the backplane between all the disks. and dependant on the architecture on the backplane serveral disk share one expander. And the feed into that will be share by the disks attached to that expander. Some expanders will have multiple inputs from the HBA, but I seen cases where 4 sas lanes go in and only 2 get used. The second bottleneck is that once you have all these nice disks connected to your HBA, but that is only on a PCIe 4x slot..... You will need PCIe x8 or x16 for that, and PCIe 3.0 stuff. Total Bandwidth: (x16 link): PCIe 3.0 = 32GB/s, PCIe 2.0 = 16GB/s, PCIe 1.1 = 8GB/s. So lets say that your 24 port HBA has 24 disks connected, each doing 100Mbytes/sec = 19,2 Gbit/s Which will very likely saturate that PCI bus. Note that I'm 0nly talking 100Mbyte/sec. Since that is what I seed spinning rust do under Ceph. I'm not even talking about the SSDs used for journals and cache. For ZFS the bus challenge is a bit more of a problem, because you cannot scale out. But I've seen designs where an extra disk cabinet with 96 disks is attached over something like 4*12Gbit/s into a controller in a PCIe 16x slot, wondering why it doesn't do what they thought it was going to do. For Ceph there is a "nice" way out, because it is able to scale out by more smaller servers with less disks per chassis. So we tend to use 16 drive chassis with 2 8 ports HBAs that have dedicated connections per disk. It is a bit more expensive but it seems to work much better. Note that you then will then run into network problems, which are more of the same. Only just a bit further up the scale. With Ceph that only plays a role during recovery of lost nodes, which hopefully is not too often. But a died/replaced disk will be able to restore at the max speed a disk can take. A lost/replaced node will recover at the speed limited by the disk infrastructure of the recovering node, since the data will come from a lot of other disks on other servers. The local busses will saturate when the HW design was poor. --WjW From owner-freebsd-fs@freebsd.org Thu Dec 27 13:59:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3CAD135FD79 for ; Thu, 27 Dec 2018 13:59:25 +0000 (UTC) (envelope-from ath@heybey.org) Received: from resqmta-ch2-10v.sys.comcast.net (resqmta-ch2-10v.sys.comcast.net [IPv6:2001:558:fe21:29:69:252:207:42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "resqmta-po-01v.sys.comcast.net", Issuer "COMODO RSA Organization Validation Secure Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2DB8174BAC for ; Thu, 27 Dec 2018 13:59:24 +0000 (UTC) (envelope-from ath@heybey.org) Received: from resomta-ch2-11v.sys.comcast.net ([69.252.207.107]) by resqmta-ch2-10v.sys.comcast.net with ESMTP id cV7mga4cqzx9VcWC3ghyHP; Thu, 27 Dec 2018 13:59:23 +0000 Received: from spaten.heybey.org ([98.237.106.248]) by resomta-ch2-11v.sys.comcast.net with ESMTPA id cWC1gjUF3V20XcWC2gVo2d; Thu, 27 Dec 2018 13:59:23 +0000 X-Xfinity-VAAS: gggruggvucftvghtrhhoucdtuddrgedtledrtddvgdehlecutefuodetggdotefrodftvfcurfhrohhfihhlvgemucevohhmtggrshhtqdftvghsihdpqfgfvfenuceurghilhhouhhtmecufedttdenucenucfjughrpefuvfhfhffkffgfgggjtgfgsehtkeertddtfeejnecuhfhrohhmpeetnhgurhgvficujfgvhigsvgihuceorghthheshhgvhigsvgihrdhorhhgqeenucfkphepleekrddvfeejrddutdeirddvgeeknecurfgrrhgrmhephhgvlhhopehsphgrthgvnhdrhhgvhigsvgihrdhorhhgpdhinhgvthepleekrddvfeejrddutdeirddvgeekpdhmrghilhhfrhhomheprghthheshhgvhigsvgihrdhorhhgpdhrtghpthhtohepfhhrvggvsghsugdqfhhssehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-Xfinity-VMeta: sc=0;st=legit Received: from yuengling.local (murphys [10.1.0.49]) by spaten.heybey.org (Postfix) with ESMTPSA id AF6ABB977 for ; Thu, 27 Dec 2018 08:59:20 -0500 (EST) Subject: Re: Suggestion for hardware for ZFS fileserver To: freebsd-fs@freebsd.org References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> <20181223113031.00005150@Leidinger.net> <1d76f92c-6665-81ef-1b94-dc1b4b8925d1@denninger.net> From: Andrew Heybey Message-ID: <814be2c2-4c09-9f1c-2a99-ed5a4c9bf7e7@heybey.org> Date: Thu, 27 Dec 2018 08:57:44 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.3.3 MIME-Version: 1.0 In-Reply-To: <1d76f92c-6665-81ef-1b94-dc1b4b8925d1@denninger.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 13:59:25 -0000 On 12/24/18 7:02 PM, Karl Denninger wrote: > On 12/24/2018 17:13, Zaphod Beeblebrox wrote: >> [ regarding ZFS hardware thread ] >> >> There's another type of server --- the "ghetto" or home storage serer. For >> this server, I like to optimize for not loosing data, not for uptime. >> >> Going back a few years, there were consumer motherboards with 10 or 12 SATA >> onboard. Mostly, this was at the change of technologies ... so you had >> some of one kind of port and some of another. Used SAS HBAs are another >> option ... but they have a caviat: many SATA drives will eventually reject >> them under load. Good SATA drives won't (but again, we're talking a ghetto >> system). If you're taking WD reds (and not, say, seagate barracudas) ... >> these work well. On the seagates, however, I've had drives repeatedly fail >> ... only to go on working fine in a workstation with a SATA controller. > > I've run "ghetto mode" fileservers with the LSI adapters in IT mode > (that always just seem to work) with one of their SFP ports connected to > a SAS expander, and then fanned THAT out to SATA drives.  The only > constraint is that you can run into problems booting from an expander, > so don't -- use the ports on the HBA (or even the motherboard) for the > boot drives. > > Never had a problem doing this with HGST drives, Intel SSDs and most > others.  The Seagates I've had fail actually physically failed; they > didn't throw a protocol hissy fit on the bus.  I don't buy Seagates any > more as I've had too many die out-of-warranty for my taste.  They work > fine with WD drives too.  Never had one of the drives that failed cause > a cascade detach event either.  The last few years (five or so) for > spinning rust HGST seems to sell the most-reliable stuff in my > experience but YMMV on that. > > Those adapters and expanders are cheap these days.  The expanders used > to be expensive, but not any more -- there's a ton of them around on the > secondary market for very little money (not much more than the LSI > cards.)  Their only downside is they run hot so you need good fan > coverage in the case. > > Older SuperMicro boards (X8DTL- series) that will take the 5600-series > Westmere Xeon processors can be had for almost nothing (note you have to > have the latest BIOS in them, which can be flashed, to run the Westmere > processors), and the CPUs are a literal $25.  The only "gotcha" is you > need ECC memory, but if you can find it at a decent price used you're > golden.  I would NOT run a ZFS filesystem without ECC memory; the risk > of undetected data corruption that you don't catch for months or even > years is material and if it happens you WILL cry since your backup > coverage may have expired at that point. My current "ghetto" server in the basement is an AMD Phenom 8-core, using SATA drives plugged into the motherboard, and 32GB ECC RAM. It is getting long in the tooth and I have been contemplating replacements. The existing one replaced an old Intel server chassis ("Jarrell" IIRC) when I realized that I could pay for the new server in a year with the power savings. The basement is cooler now too. I am considering going back to a "real" server and getting a generation or two old Dell server like the R710, R520 or R720 and putting a couple 8-core low-power Xeons like the E5-2648L in it. The Dells on Ebay seem to often come with a RAID controller that can be flashed with the "IT" firmware for JBOD for ZFS, and DDR3 ECC server RAM is relatively cheap. The low-power Xeons are 65W TDP (compared to my current AMD at 125W) so it shouldn't use too much more power (maybe?). For $500-$750 I can get lots of (slow) cores, 64-128GB RAM and 8 or 12 disk slots. My idea for keeping my ZFS2 disk array from crapping out on me is ECC RAM, scrubbing every week, and replacing the oldest drive every year or so (or whenever SMART complains about reallocated sectors). I started out with 1.5, 2 and 3TB drives, and am up to 3,4 and 6TB drives. (I use partitions for the ZFS array so that I can use the extra space on the larger drives for other stuff.) As the older smaller drives get replaced with larger ones, the array grows in size to (hopefully) keep up with my needs. I also then have a variety of drives (by both manufacturer and date of manufacture) so hopefully I won't stumble across the same bug on multiple drives at the same time. andrew From owner-freebsd-fs@freebsd.org Thu Dec 27 18:37:30 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DFD81141F415 for ; Thu, 27 Dec 2018 18:37:29 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4669486B26 for ; Thu, 27 Dec 2018 18:37:29 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-lj1-x22c.google.com with SMTP id q2-v6so16921042lji.10 for ; Thu, 27 Dec 2018 10:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Gwac9yOv9LGxARYKTBYIJjyab+q3thvr71nXBPJwBvk=; b=k7n/Pzq29EPPGRpSvf2TxHuInkb8Dp2sZveVv4GHc14g1BR0E2CfWnkLP3nNAIIS+S yfqbOVnGBPi4TeSAy2+xGntM28F+ciAF7HgpzSR3OjPHF8XRk7oRoo1a4BYJJluB3zIL A8FLw8p0Rl1mjEPtRDrid0HCkDGHbpRSEwB3LTYDCKKPvZtwUzKzjek7/YTjCyiAvztY 5/1ZfubYcK3GERkAUv9mdQ2yAR3x0a39WEdUYMxNz+FeK0YMmcrii1B1R28BNarg7jZc K6/XhINMtv9DlrLbC9lBNma3eX9gY9wLa3Vl8EqJLbaBsFgu96O4Sxe8U2AOWFhZvKsw Ep0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Gwac9yOv9LGxARYKTBYIJjyab+q3thvr71nXBPJwBvk=; b=G8RY+zmcZb76KCsdN6zPFG18ELfMyaerI4LJrLh+2LKaDrFNh7tplobkm6SVF8lyOT eEiyVoQtQpbgekpLwcZ13iHc02Pr0764ifHas3oVTanGocFjkEXGPDko4R1jMRtWpFH+ DoGHiTE8DchlTsnW0nWCiw6hNnmOUyn7nBlffkDE/GxkrG/+gRHtHkKEDnXYc4Xg4x6e 97IeGCIj1FoGS74bNohf3dSZAsrpcB5d5l8dK+GWyoJsGKfGCP+uX3AJOFXc0ikijSzD 5N+5WIsCNR9ZF3JgDmgYpZH7GhnAcMy+3BgVKn/LH5cvB89iIwcu1OWkv1IbjXOfqo3L hoPA== X-Gm-Message-State: AJcUukfU1vX1Of3Pskt1FDaG5AdbvMkxcZgeBk60SItpAK9iiLF1qjXO NrWhCfJWJ9tqJLkD86sAa+GrGSMBpstIhzw3aDnBlw== X-Google-Smtp-Source: ALg8bN5KCy7crFPbEr/xsO3p+phsA4Q8jeLFJVoHa9YHpL5k9T7VdPAXJV1icl77HFLQjWOwD4mavpZv0+Dch3ELzXU= X-Received: by 2002:a2e:81a:: with SMTP id 26-v6mr15605374lji.14.1545935847684; Thu, 27 Dec 2018 10:37:27 -0800 (PST) MIME-Version: 1.0 References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net> <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se> In-Reply-To: From: Freddie Cash Date: Thu, 27 Dec 2018 10:37:15 -0800 Message-ID: Subject: Re: Suggestion for hardware for ZFS fileserver To: Willem Jan Withagen Cc: Sami Halabi , FreeBSD Filesystems X-Rspamd-Queue-Id: 4669486B26 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2018 18:37:30 -0000 On Thu, Dec 27, 2018, 2:55 AM Willem Jan Withagen On 22/12/2018 15:49, Sami Halabi wrote: > > Hi, > > > > What sas hba card do you recommend for 16/24 internal ports and 2 > external > > that are recognized and work well with freebsd ZFS. > > There is no real advise here, but what I saw is that it is relatively > easy to overload a lot of the busses involved int his. > > I got this when building Ceph clusters on FreeBSD, where each disk has > its own daemon to hammer away on the platters. > > The first bottleneck is the disk "backplane". It you do not need to wire > every disk with a dedicated HBA-disk cable, then you are sharing the > bandwidth on the backplane between all the disks. and dependant on the > architecture on the backplane serveral disk share one expander. And the > feed into that will be share by the disks attached to that expander. > Some expanders will have multiple inputs from the HBA, but I seen cases > where 4 sas lanes go in and only 2 get used. > You can get backplanes that use multi-lane SFF-8087 connectors and cables between the HBA and backplane, but provide individual connections to each drive bay. You get the best of both worlds (individual connections to each drive, but only 1 cable for every 4 drives. :) No expanders or port multipliers involved. Supermicro 836A backplane is an example of that. It's what we use for all our ZFS and iSCSI boxes. AMD Epyc motherboards provide lots of PCIe slots and lanes to stuff with HBAs, without worrying about bottlenecking. :) -- Cheers, Freddie Typos due to phone keyboard. From owner-freebsd-fs@freebsd.org Fri Dec 28 00:20:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B51741427A80 for ; Fri, 28 Dec 2018 00:20:11 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660051.outbound.protection.outlook.com [40.107.66.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 64EE26B70F for ; Fri, 28 Dec 2018 00:20:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM (10.169.142.146) by YQBPR01MB0179.CANPRD01.PROD.OUTLOOK.COM (10.169.141.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1471.20; Fri, 28 Dec 2018 00:20:09 +0000 Received: from YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c]) by YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM ([fe80::9d84:f9d8:b5bb:3b7c%8]) with mapi id 15.20.1471.019; Fri, 28 Dec 2018 00:20:08 +0000 From: Rick Macklem To: Peter Eriksson , "freebsd-fs@freebsd.org" Subject: Re: Suggestion for hardware for ZFS fileserver Thread-Topic: Suggestion for hardware for ZFS fileserver Thread-Index: AQHUmU1yCLTPDjfxZEms4Ul8AJXDZ6WJb0QAgAAbMACAAE3yOYAJeSyb Date: Fri, 28 Dec 2018 00:20:08 +0000 Message-ID: References: <4f816be7-79e0-cacb-9502-5fbbe343cfc9@denninger.net>, <3160F105-85C1-4CB4-AAD5-D16CF5D6143D@ifm.liu.se>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; YQBPR01MB0179; 6:jlvDfA+nXCX8RsjjZWlefV9mtc7lI/pYlhT+V59pK2zgppJ3NfLWll/cJVP+h2YpSy75qI54Sx1NBqsikLSlbb480f0dHj/W4NgKEuvU1DUAU0axgPa/OR8HYVwhPnzEF+z6j5mE01c2jPITeEsCSB/fojEO+5oW8kSCNCxQ1G+eWxSWk9nci9u3MtK98Q9PoNt6SeY1U6gZZe1ABPvfpGJcHXEOTIzHzxvFxKIEIg29jaNWYSHl9L0E9YKGBJ99DTYsZvflHgsMqI2Mspjb0oYqVpfP3pqaDWsr18NroqKnp4Z7Lod+z8ZL7kqjx/UXTwshd04mUkyN5h2l9SH6IZWr+vOms7UQfLfPgpJY7Kv/0CjMmdyc6cZrJYavyC8tJ45silgv3oRpyoYm1e3H5cox7rGRlWfYsWkCvKGbTFBvXTc64BCA2GAt2cRHA4GLIjKuV7f0X904L755RsIed5kiag57BmQT7Ii3jolgG3Y=; 5:KaxTDzFzjtuIQeziDj9WD8D+kX9Dd/80/faGKoFi8TccP8jfUd73ieRjgYB9kSXK2sqUvcqJ8t9loy3BY5KghjguQaggpFxhfB19JjB8k88z+qRmf7UrAKdWJ948aAHpX/8xKDuaNrkrt5wtuNAzqT9ZDc72cYTh2zR6O6eSOAU=; 7:LkDQayKxCgPWVF0BUlKwSoqzYBeVVFNZPz74FE7MNFmsvM+jqqPb2UihPXNpLbnAOcYSe72k6r0++/Xt6lciwzihqSGCym4mRCNGdDKSJfXZt5jNNuQH4AX1+apVit7sDE7wXSJcSSRWS7ocoHvnWA== x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 1e8f13dd-5eae-45a9-62df-08d66c5a3996 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600107)(711020)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:YQBPR01MB0179; x-ms-traffictypediagnostic: YQBPR01MB0179: x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(3230021)(908002)(999002)(5005026)(6040522)(2401047)(8121501046)(93006095)(93001095)(3231475)(944501520)(52105112)(3002001)(10201501046)(6041310)(20161123564045)(201703131423095)(201703031522075)(201702281528075)(201702281529075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123560045)(20161123562045)(201708071742011)(7699051)(76991095); SRVR:YQBPR01MB0179; BCL:0; PCL:0; RULEID:; SRVR:YQBPR01MB0179; x-forefront-prvs: 09007040D4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(39860400002)(136003)(366004)(396003)(346002)(174864002)(199004)(189003)(5660300001)(7696005)(99286004)(53936002)(110136005)(74316002)(102836004)(6246003)(316002)(46003)(76176011)(11346002)(446003)(296002)(6506007)(33656002)(105586002)(305945005)(786003)(106356001)(186003)(476003)(229853002)(9686003)(55016002)(93886005)(86362001)(81166006)(81156014)(6436002)(68736007)(97736004)(25786009)(8676002)(14444005)(8936002)(74482002)(478600001)(14454004)(71200400001)(2501003)(71190400001)(256004)(2906002)(486006); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR01MB0179; H:YQBPR01MB0388.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Bbfk3iU+xt5gwyTlTfP75mSKeX6IpIIheUv4kq1FbBrwerSHXhikQ+tRwlNTsy4sgGpiBex7p+wBalNUm4/UipAd2cltr4dKZKRXuPE5t5sYVjUxS85LKedyfG9/lnHJkTHMnoAu0cfqcwl6SAnub5LhPiSp+RaPaLShY4w9c7AWTGLmjNoGNe+zhIi1ftBkeU8X7H2TmHMABhkS2Yzku235+XS/kRnhQoCQ3wknis/kxo+kvpP/00Z8Ik3vueFCfopJbCLUyyG+SEkNE+5IfGyQ4HKWOlPKK1CLTJNUD+cp+X/GZJVDvtYt4cgWEwH7 spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 1e8f13dd-5eae-45a9-62df-08d66c5a3996 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Dec 2018 00:20:08.7966 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR01MB0179 X-Rspamd-Queue-Id: 64EE26B70F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.51 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-3.91 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com,mx2.hc184-76.ca.iphmx.com,mx1.hc184-76.ca.iphmx.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[51.66.107.40.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_SHORT(-0.71)[-0.707,0]; IP_SCORE(-0.89)[ipnet: 40.64.0.0/10(-2.27), asn: 8075(-2.11), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 00:20:11 -0000 I wrote: >Peter Eriksson wrote: >[good stuff snipped] >>This has caused some interesting problems=85 >> >>First thing we noticed was that booting would take forever=85 Mounting th= e 20-100k >>filesystems _and_ enabling them to be shared via NFS is not don= e efficient at all (for each filesystem it re-reads /etc/zfs/exports (a cou= ple of times) befor appending one line to the end. Repeat 20-100,000 times= =85 Not to mention the big kernel lock for NFS =93hold all NFS activity whi= le we flush and reinstalls all sharing information per filesystem=94 being = done by mountd=85 >Yes, /etc/exports and mountd were implemented in the 1980s, when a dozen >file systems would have been a large server. Scaling to 10,000 or more fil= e systems wasn't even conceivable back then. >Wish list item #1: A BerkeleyDB-based =92sharetab=92 that replaces the hor= ribly >slow /etc/zfs/exports text file. >Wish list item #2: A reimplementation of mountd and the kernel interface t= o allow >a =93diff=94 between the contents of the DB-based sharetab above b= e input into the >kernel instead of the brute-force way it=92s done now.. >The parser in mountd for /etc/exports is already an ugly beast and I think >implementing a "diff" version will be difficult, especially figuring out w= hat needs >to be deleted. > >I do have a couple of questions related to this: >1 - Would your case work if there was an "add these lines to /etc/exports"= ? > (Basically adding entries for file systems, but not trying to delete = anything > previously exported. I am not a ZFS guy, but I think ZFS just genera= tes another > exports file and then gets mountd to export everything again.) >2 - Are all (or maybe most) of these ZFS file systems exported with the sa= me > arguments? > - Here I am thinking that a "default-for-all-ZFS-filesystems" line c= ould be > put in /etc/exports that would apply to all ZFS file systems not = exported > by explicit lines in the exports file(s). > This would be fairly easy to implement and would avoid trying to han= dle > 1000s of entries. > >In particular, #2 above could be easily implemented on top of what is alre= ady >there, using a new type of line in /etc/exports and handling that as a spe= cial >case by the NFS server code, when no specific export for the file system t= o the >client is found. Unfortunately, it doesn't sound like #2 above would be useful for Peter. Al= though it is easy to implement a single default export for all ZFS file systems not alre= ady exported, it would not be easy to say "export all file systems below /foo/bar this wa= y", since the kernel code basically doesn't know the directory structure. It has vnod= es for file objects and mount points to work with. (The kernel exports hang off of= the mount points.) >>(I=92ve written some code that implements item #1 above and it helps quit= e a bit. >>Nothing near production quality yet though. I have looked at ite= m #2 a bit too but >>not done anything about it.) Btw, this "item #2" is not what I am referring to. [more good stuff snipped] rick From owner-freebsd-fs@freebsd.org Fri Dec 28 01:49:25 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 087AE142AAC2 for ; Fri, 28 Dec 2018 01:49:25 +0000 (UTC) (envelope-from 01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com) Received: from a8-60.smtp-out.amazonses.com (a8-60.smtp-out.amazonses.com [54.240.8.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 57BA56E4FB for ; Fri, 28 Dec 2018 01:49:23 +0000 (UTC) (envelope-from 01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=dqtolf56kk3wpt62c3jnwboqvr7iedax; d=tarsnap.com; t=1545961426; h=Subject:From:To:References:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=gpLc5OzctMdVEaQxdSC0cOBjOg9kY79haQx5mevJTUA=; b=QSYvdfIJ7H5eNDSsKb0svxqgyh5ZNnw/K8hThXF3yHXqCbnxP0pR8rqBk/S+uQbs HERrJUx7k5Y4tM6icHJP/XODTwJ/6cQyajqPsPvQy1t6g7nq0meLHLc10QRfgemlEX/ QSv1nLR396THIizD3EmXMJNqJuhCONvYZTfMDf4U= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545961426; h=Subject:From:To:References:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=gpLc5OzctMdVEaQxdSC0cOBjOg9kY79haQx5mevJTUA=; b=XnYWdHmKbojxs9OPmFiQ8Q9Yl060F57iViO31YFb+1HHCy+fUkWrfca0RPblzomi jeU1AwLfl6SwybVlsNjyPOvpjFrgynxB15zLRac38elHiHyxx4Eeyz6/AbBCQBZccwR L+drOKbr8XmSPdAyufyAtGIEtm4Rt1gK+ET+aq8U= Subject: Re: Any plans for EC2 AMIs with RootOnZFS? From: Colin Percival To: Ben Woods , freebsd-cloud@freebsd.org, "freebsd-fs@freebsd.org" References: <01000167dd7b16cd-58ee733f-c662-42d5-9c75-2301fbc8377a-000000@email.amazonses.com> Openpgp: preference=signencrypt Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= mQGhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgStCVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+iGEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEuQINBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFiEkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@email.amazonses.com> Date: Fri, 28 Dec 2018 01:43:45 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <01000167dd7b16cd-58ee733f-c662-42d5-9c75-2301fbc8377a-000000@email.amazonses.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2018.12.28-54.240.8.60 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Rspamd-Queue-Id: 57BA56E4FB X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tarsnap.com header.s=dqtolf56kk3wpt62c3jnwboqvr7iedax header.b=QSYvdfIJ; dkim=pass header.d=amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=XnYWdHmK; spf=pass (mx1.freebsd.org: domain of 01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com designates 54.240.8.60 as permitted sender) smtp.mailfrom=01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com X-Spamd-Result: default: False [-1.23 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:54.240.0.0/18]; DKIM_TRACE(0.00)[tarsnap.com:+,amazonses.com:+]; MX_GOOD(-0.01)[cached: feedback-smtp.us-east-1.amazonses.com]; NEURAL_HAM_SHORT(-0.94)[-0.940,0]; FORGED_SENDER(0.30)[cperciva@tarsnap.com,01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_ZERO(0.00)[0]; IP_SCORE(-1.58)[ipnet: 54.240.8.0/21(-4.57), asn: 14618(-3.26), country: US(-0.08)]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14618, ipnet:54.240.8.0/21, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; FROM_NEQ_ENVFROM(0.00)[cperciva@tarsnap.com,01000167f27b4bf2-d7bb71e2-70f4-4b20-9848-4aa7fdafccc9-000000@amazonses.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[tarsnap.com:s=dqtolf56kk3wpt62c3jnwboqvr7iedax,amazonses.com:s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tarsnap.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[60.8.240.54.list.dnswl.org : 127.0.15.0]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 01:49:25 -0000 [Adding freebsd-fs in the hope of finding more ZFS people.] On 12/23/18 3:51 PM, Colin Percival wrote: > On 12/23/18 7:12 AM, Ben Woods wrote: >> Is there any plans to provide official FreeBSD AMIs in the AWS Marketplace >> which use RootOnZFS out of the box? > > Nothing concrete yet, but it's on my radar. Now that FreeBSD AMIs are being > published by a separate (release engineering) account, I'm more comfortable > with working on more "experimental" AMIs. (And if this turns out to be widely > useful I'll talk to the release engineering team about adding it to the set > they publish.) I've created an experimental FreeBSD 12.0 ZFS AMI in the us-east-1 region: ami-0786f5b55d5aa573f Since I'm basically a ZFS newbie (I run it on my laptop, but only with the default setup from the installer, and I've never used any of the fancy ZFS features) I don't know if I've set everything up properly. So far I have one piece of feedback, which is that I should have marked canmount=off for zroot. Can people please try out this AMI and let me know if there's anything else (ZFS-related) which I should fix? Some time next week I'll fix whatever people notice and build AMIs for all the regions. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-fs@freebsd.org Fri Dec 28 10:18:49 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 481E41433E3B for ; Fri, 28 Dec 2018 10:18:49 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32E3083E7F for ; Fri, 28 Dec 2018 10:18:48 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f195.google.com with SMTP id v1-v6so18363751ljd.0 for ; Fri, 28 Dec 2018 02:18:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=bpmOZszWdl/QYM8cV1avAWlM4aIcrupf54L2ZcU6F+Y=; b=KzcIxxCBFo0a9ft50egZBzZhbxo4UlVD3gvHChXCw8W9Kzd7zcPoK7i8qxQm6EAbLp Z85tsrGZH+EpE+2Pt6RT23fmzmu1hdngEJsGX6ueU3ZwnK/11W+c5iTyMQP1OOnT5CK9 DauXQTfGcBswU2+Zol9GG9AA2KsREJcSOcXMM8thAWdB27+rxv8g8C/x4hD9luaiNQ9h GhN8mJxkSvVMr1CmEcgrDPwaSVIrNBgolitUWgSbPyxBN6kFdBDGpmybTUTnUh8Ch/UC j5Z1HR8shcBRglPjHT/BblzYBl05gviMt3Zing7xjWjcbQK+qXYgjtxipjHnyRhyFyuI YvUA== X-Gm-Message-State: AJcUukc0DHy5HYu9fY79rKc0ugte9PRiQNgtY9lkY7zNvpl+r496VDzG FK2nb7k1648zJgMSixc6k1+tIRtI X-Google-Smtp-Source: ALg8bN67SIb8ShN6p4QxPkV6AQEREw23mL00kbySeB9skHzfIon2wU7TsH5VEBg29re6Ac9dgQA7iQ== X-Received: by 2002:a2e:2b11:: with SMTP id q17-v6mr15421179lje.25.1545992325826; Fri, 28 Dec 2018 02:18:45 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id k21-v6sm8339599ljc.15.2018.12.28.02.18.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 28 Dec 2018 02:18:45 -0800 (PST) Subject: Re: Upgrade to 12-Release system not booting kernel panic To: Kaya Saman , freebsd-fs@freebsd.org References: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <9c5e215a-6cb0-2f97-4652-0bca8683e555@FreeBSD.org> Date: Fri, 28 Dec 2018 12:18:44 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 32E3083E7F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.195 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.86 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.79)[-0.787,0]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; IP_SCORE(-1.06)[ipnet: 209.85.128.0/17(-3.70), asn: 15169(-1.51), country: US(-0.08)]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[195.208.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[195.208.85.209.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 10:18:49 -0000 On 16/12/2018 09:27, Kaya Saman wrote: > Hi, > > > I've been digging around a little on this one and it seems that there is some > ZFS wierdness going on. > > > My original install was an 11.2-RELEASE fresh install and upon an upgrade to > 12-RELEASE the system is having issues booting. > > > I have a ZFS root pool called zroot; by default this mounts at > zroot/ROOT/default and a few other non-root / bootable ZFS pools. > > > The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. > > > I have the boot disks ada0 and ada1 plugged into the rear of the chassis and > directly into the systemboard which is also a SuperMicro. These drives are both > Samsung SSD's. > > The 22 drive slots at the front of the chassis are occupied by the other various > data pools. > > > So here is some strangeness... if I remove all 22 drives from the front, the > system boots fine but straight after boot goes into kernel panic mode and > reboots before I can even look at the error or get to the login prompt. > > > With the non-root pools installed at the BTX loader after scanning through all > the bios drives I get a bunch of: > > read 264 from ... to 0x...., error 0x10 errors > > then: > > ZFS: i/o error - all block copies unavailable > > ZFS: can't read MOS of pool I suspect that this is because of which disks the BIOS presents to the loader (and in what order). > after this the system simply hangs? > > > I have tried looking around but everything mentioning the MOS error is talking > about the root pool, a particularly good reference is here: > http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html > > > In fact I did try to boot with a USB stick and go into Live mode then import all > the pools on the system. This works without any issue! The pools are fine the > data is there everything looks normal. > > - I also rebuilt the zpool.cache according to the link just incase there was > some kind of corruption there, however upon reboot I still get the same issue?? > > > Looking at a bug report with a kernel panic: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 > > > I have attempted to add: > > > kern.cam.scsi_delay="50000" > > kern.cam.boot_delay="50000" > > > into the /boot/loader.conf file but unfortunately the issue still continues :-( > > > I wonder if there is a way to tell to tell the system to only look at certain > drives for booting?? The boot code looks at all disks presented by the BIOS. It should look at the disk configured as a boot disk first. > There is this line in my loader.conf: > > vfs.root.mountfrom="zfs:zroot" > > > It maybe the wrong hunch I have but it seems like the system is looking for > "zroot" on all pools instead of the actual root pool hence the above errors?? You can't look for a pool on all pool, that sounds nonsensical. The correct way to say this is that the system looks for zroot on all disks. > Would anyone be able to suggest anything or have any ideas about how to get the > system back online and booting?? I think that you need to look at the BIOS / disk controller configuration. -- Andriy Gapon From owner-freebsd-fs@freebsd.org Fri Dec 28 10:25:12 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78A9B14341A3 for ; Fri, 28 Dec 2018 10:25:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 11E52842B8 for ; Fri, 28 Dec 2018 10:25:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C9E3114341A2; Fri, 28 Dec 2018 10:25:11 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B87FB14341A1 for ; Fri, 28 Dec 2018 10:25:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 52E1B842B2 for ; Fri, 28 Dec 2018 10:25:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 7954EF7BB for ; Fri, 28 Dec 2018 10:25:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBSAPA73059287 for ; Fri, 28 Dec 2018 10:25:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBSAPA6b059284 for fs@FreeBSD.org; Fri, 28 Dec 2018 10:25:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 210409] zfs: panic during boot, spa_refcount < spa_minref Date: Fri, 28 Dec 2018 10:25:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 10:25:12 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210409 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mshirk@daemon-security.com --- Comment #13 from Andriy Gapon --- *** Bug 223612 has been marked as a duplicate of this bug. *** --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-fs@freebsd.org Fri Dec 28 10:25:11 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A2BF143419E for ; Fri, 28 Dec 2018 10:25:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C8B20842AF for ; Fri, 28 Dec 2018 10:25:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8C900143419D; Fri, 28 Dec 2018 10:25:10 +0000 (UTC) Delivered-To: fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B0C1143419C for ; Fri, 28 Dec 2018 10:25:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19AD9842AE for ; Fri, 28 Dec 2018 10:25:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5F856F7B8 for ; Fri, 28 Dec 2018 10:25:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id wBSAP9sQ059246 for ; Fri, 28 Dec 2018 10:25:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id wBSAP9LY059243 for fs@FreeBSD.org; Fri, 28 Dec 2018 10:25:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: fs@FreeBSD.org Subject: [Bug 223612] zfs panic when importing zpool from USB key while system under heavy load Date: Fri, 28 Dec 2018 10:25:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: avg@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: DUPLICATE X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: fs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Dec 2018 10:25:11 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223612 Andriy Gapon changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |DUPLICATE Status|New |Closed --- Comment #1 from Andriy Gapon --- I believe that this problem has the same root cause as bug 210409, even tho= ugh the crash happens at a different time. *** This bug has been marked as a duplicate of bug 210409 *** --=20 You are receiving this mail because: You are the assignee for the bug.=