From owner-freebsd-stable@freebsd.org Tue Sep 24 16:05:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CD1C7129D25 for ; Tue, 24 Sep 2019 16:05:38 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [174.136.98.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.nomadlogic.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46d5dF5bTdz4cH3 for ; Tue, 24 Sep 2019 16:05:37 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.206] (cpe-23-243-162-239.socal.res.rr.com [23.243.162.239]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id 1acdf6d5 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Tue, 24 Sep 2019 16:05:35 +0000 (UTC) Subject: Re: Question about bottle neck in storage To: John Fleming , freebsd-stable@freebsd.org References: From: Pete Wright Message-ID: <9415ff89-a36a-86ee-3f3f-47e9b807059e@nomadlogic.org> Date: Tue, 24 Sep 2019 09:05:34 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 46d5dF5bTdz4cH3 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pete@nomadlogic.org designates 174.136.98.114 as permitted sender) smtp.mailfrom=pete@nomadlogic.org X-Spamd-Result: default: False [-4.88 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[239.162.243.23.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[nomadlogic.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-2.58)[ip: (-9.21), ipnet: 174.136.96.0/20(-3.33), asn: 25795(-0.31), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:25795, ipnet:174.136.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Sep 2019 16:05:38 -0000 On 9/24/19 8:45 AM, John Fleming wrote: > Is there anyway to see how busy a SAS/Sata controller is vs disks? I > have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid > 10 ZFS. > > When firing off a dd I (bs=1G count=10) seems like the disks never go > above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW > or if its something else (like terrible dd options). > > my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a > dedicated 8 drive board inside the front of the R720. Basically i'm > just saying its not a single SAS cable to 14 drives. > > Don't have cpu info hand.. zeon something. DDR3-1600 (128GB) > > Both controllers are in 8x slots running PCIe gen 3. might want to take a look at sysutils/intel-pcm (https://github.com/opcm/pcm).  I *think* this should give you metrics on PCIe bus utilization among other useful status. Also, lookup the bandwidth for the PCIe bus and see if your aggregate disk throughput on one of the PCIe lanes is saturating the bus (pcm should also help here).  You can also run "zpool iostat -v 2" to see per disk i/o metrics to help determine if this is an issue. > BTW i'm sure this has been asked a million times but what would be > some decent benchmark tests while i'm at it? I generally run several tests and then compare results, for example bonnie++, iozone, iperf (writing over the wire and to disk) as well as some more realistic scripts based on the use-case i'm building a solution for.  hope that helps. -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA