From owner-freebsd-fs@freebsd.org Wed Oct 23 05:41:00 2019 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17D50170058 for ; Wed, 23 Oct 2019 05:41:00 +0000 (UTC) (envelope-from michelle@sorbs.net) Received: from hades.sorbs.net (hades.sorbs.net [72.12.213.40]) by mx1.freebsd.org (Postfix) with ESMTP id 46yfP64pQ4z3CKT for ; Wed, 23 Oct 2019 05:40:58 +0000 (UTC) (envelope-from michelle@sorbs.net) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from isux.com (gate.mhix.org [203.206.128.220]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0PZT00JJCCI44B10@hades.sorbs.net> for freebsd-fs@freebsd.org; Tue, 22 Oct 2019 22:56:29 -0700 (PDT) To: "freebsd-fs@freebsd.org" From: Michelle Sullivan Subject: Performance problems on LSI cards.. SOLVED. Message-id: <170599f2-330f-18b1-b7b7-23f70ba8cae7@sorbs.net> Date: Wed, 23 Oct 2019 16:40:55 +1100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:51.0) Gecko/20100101 Firefox/51.0 SeaMonkey/2.48 X-Rspamd-Queue-Id: 46yfP64pQ4z3CKT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of michelle@sorbs.net designates 72.12.213.40 as permitted sender) smtp.mailfrom=michelle@sorbs.net X-Spamd-Result: default: False [-1.98 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:hades.sorbs.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sorbs.net]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-0.28)[ip: (-0.69), ipnet: 72.12.192.0/19(-0.36), asn: 11114(-0.29), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[40.213.12.72.list.dnswl.org : 127.0.10.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; CTE_CASE(0.50)[]; ASN(0.00)[asn:11114, ipnet:72.12.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Oct 2019 05:41:00 -0000 A long time ago I posted about my ZFS woes (which I might add are fixed thanks to the 'zfs-recovery' tool) ...but throughout since 2011 I have had performance issues with my LSI-9260-16i cards now I finally have solved the performance problems resulting in maximum scrub speed of 67M/s... (its a 15disk raidz2 setup with Seagate SATAIII x 3T drives)... 2 problems, one hid the other... first turn off bios for presenting the drives to the OS ... exactly why this had an impact I'm not sure but I think I can guess based on what actually fixed it... My systems have 6 PCIe slots. 3 off PCIe, 1 off PCIe2x16 and 2 off PCIe3/2x16 (or 2x8) ... specifically the manual says: 2 x PCI Express 3.0/2.0 x16 slots (single at x16 or dual at x8/x8 mode) 1 x PCI Express 2.0 x16 slot* (max. at x4 mode, compatible with PCIe x1 and x4 devices) 3 x PCI Express 2.0 x1 slots** * The PCIe 2.0 x16 slot shares bandwidth with PCIe 2.0 x1_1 slot, PCIe 2.0 x1_2 slot, and PCIe 2.0 x1_3 slot. The PCIe 2.0 x16 slot default setting is x1 mode. ** The PCIe 2.0 x1_1 slot shares bandwidth with eSATA 6.0 Gb/s ports. The eSATA 6.0 Gb/s ports default settings are enabled. Being there is a video card in appropriate PCIe3 slot and the card is PCIe2 I put them in non shared slots... If anyone else is looking at doing similar... don't .. the cards are 8 channel and they (not sure why - probably the firmware - forcing to x4 didn't work either - all other slots are empty so no sharing for that slot) seem to work at 8 channel or 1 channel. Putting the video card and the LSI in the PCI3/2 x16 slots and switching x8/x8 mode and the scrubs are now up at 421M/s (average.) Anyhow I'll go bck to my corner now. Regards, -- Michelle Sullivan http://www.mhix.org/