From owner-freebsd-fs@freebsd.org Mon Oct 21 05:50:37 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 835D71745EA for ; Mon, 21 Oct 2019 05:50:37 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate2.uni-hannover.de (mailgate2.uni-hannover.de [130.75.2.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46xQj84WGyz3J5w for ; Mon, 21 Oct 2019 05:50:36 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mailgate2.uni-hannover.de (Postfix) with ESMTPS id 1D6E6C63 for ; Mon, 21 Oct 2019 07:50:34 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP8) with ESMTP id 2019102107503384-8605 ; Mon, 21 Oct 2019 07:50:33 +0200 Date: Mon, 21 Oct 2019 07:50:33 +0200 From: Gerrit =?UTF-8?B?S8O8aG4=?= To: freebsd-fs@freebsd.org Subject: Re: zfs capacity Message-ID: <20191021075033.6c9748c7@arc.aei.uni-hannover.de> In-Reply-To: <20191018113554.4f5822e3@arc.aei.uni-hannover.de> References: <20191018113554.4f5822e3@arc.aei.uni-hannover.de> Organization: Albert-Einstein-Institut X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 21/10/2019 07:50:33, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP8|February 23, 2017) at 21/10/2019 07:50:34, Serialize complete at 21/10/2019 07:50:34 X-TNEFEvaluated: 1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 X-Rspamd-Queue-Id: 46xQj84WGyz3J5w X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of gerrit.kuehn@aei.mpg.de has no SPF policy when checking 130.75.2.114) smtp.mailfrom=gerrit.kuehn@aei.mpg.de X-Spamd-Result: default: False [-1.10 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.91)[-0.909,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.91)[-0.906,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[mpg.de]; RCVD_IN_DNSWL_MED(-0.20)[114.2.75.130.list.dnswl.org : 127.0.11.2]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:130.75.0.0/16, country:DE]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(0.02)[asn: 680(0.11), country: DE(-0.01)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Oct 2019 05:50:37 -0000 Am Fri, 18 Oct 2019 11:35:54 +0200 schrieb Gerrit K=C3=BChn : Hi, > I just tried these two raidz2 setups with 36 disks: 3x 10+2, 4x 7+2. > The result is a bit puzzling for me, because I get the same capacity, > although one setup is using 6 drives for redundancy, the other is using > 8: For the record: I found this to be caused by allocation padding (and I was most irritated by the fact that in the case I tested it turned out to yield /exactly/ the same number of usable space regardless of having two more data disks). Anyway, I found this table very helpful, so I'd like to share the link (even if this may be well-known to others already, it took me some time to find it): cu Gerrit From owner-freebsd-fs@freebsd.org Wed Oct 23 05:41:00 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17D50170058 for ; Wed, 23 Oct 2019 05:41:00 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 46yfP64pQ4z3CKT for ; Wed, 23 Oct 2019 05:40:58 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PZT00JJCCI44B10@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 22 Oct 2019 22:56:29 -0700 (PDT) To: "freebsd-fs@freebsd.org" From: Michelle Sullivan Subject: Performance problems on LSI cards.. SOLVED. Message-id: <170599f2-330f-18b1-b7b7-23f70ba8cae7@sorbs.net> Date: Wed, 23 Oct 2019 16:40:55 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 X-Rspamd-Queue-Id: 46yfP64pQ4z3CKT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-1.98 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sorbs.net]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-0.28)[ip: (-0.69), ipnet: 72.12.192.0/19(-0.36), asn: 11114(-0.29), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; CTE_CASE(0.50)[]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2019 05:41:00 -0000 A long time ago I posted about my ZFS woes (which I might add are fixed thanks to the 'zfs-recovery' tool) ...but throughout since 2011 I have had performance issues with my LSI-9260-16i cards now I finally have solved the performance problems resulting in maximum scrub speed of 67M/s... (its a 15disk raidz2 setup with Seagate SATAIII x 3T drives)... 2 problems, one hid the other... first turn off bios for presenting the drives to the OS ... exactly why this had an impact I'm not sure but I think I can guess based on what actually fixed it... My systems have 6 PCIe slots. 3 off PCIe, 1 off PCIe2x16 and 2 off PCIe3/2x16 (or 2x8) ... specifically the manual says: 2 x PCI Express 3.0/2.0 x16 slots (single at x16 or dual at x8/x8 mode) 1 x PCI Express 2.0 x16 slot* (max. at x4 mode, compatible with PCIe x1 and x4 devices) 3 x PCI Express 2.0 x1 slots** * The PCIe 2.0 x16 slot shares bandwidth with PCIe 2.0 x1_1 slot, PCIe 2.0 x1_2 slot, and PCIe 2.0 x1_3 slot. The PCIe 2.0 x16 slot default setting is x1 mode. ** The PCIe 2.0 x1_1 slot shares bandwidth with eSATA 6.0 Gb/s ports. The eSATA 6.0 Gb/s ports default settings are enabled. Being there is a video card in appropriate PCIe3 slot and the card is PCIe2 I put them in non shared slots... If anyone else is looking at doing similar... don't .. the cards are 8 channel and they (not sure why - probably the firmware - forcing to x4 didn't work either - all other slots are empty so no sharing for that slot) seem to work at 8 channel or 1 channel. Putting the video card and the LSI in the PCI3/2 x16 slots and switching x8/x8 mode and the scrubs are now up at 421M/s (average.) Anyhow I'll go bck to my corner now. Regards, -- Michelle Sullivan http://www.mhix.org/