From owner-freebsd-stable@freebsd.org Sun Sep 29 15:35:57 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 901BB12360C for ; Sun, 29 Sep 2019 15:35:57 +0000 (UTC) (envelope-from john@spikefishsolutions.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46h8kh0sCSz4dvL for ; Sun, 29 Sep 2019 15:35:55 +0000 (UTC) (envelope-from john@spikefishsolutions.com) Received: by mail-lf1-x132.google.com with SMTP id w67so5184831lff.4 for ; Sun, 29 Sep 2019 08:35:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spikefishsolutions.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=zcBNVsOm7iNWAOs7xK2TWQZBtHuJqkBOY9GaWzTSktU=; b=D2XC9hvo3CtVTGcuFRy0DcEXvdxyYnT9fJxxFePEx7rP25vowCVjGovJKS1jaLMPjU 6QSWEB+B0EXFgxvq0aDcZXRJZ5ZWNmOu6qvBKurIJi2fVdd/pM99vYNPFByyoZs9ZoPD XtIyNc+wYR2bI9ETuuaqLY/xmS9z+wDQGbZ7c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=zcBNVsOm7iNWAOs7xK2TWQZBtHuJqkBOY9GaWzTSktU=; b=P/FyODntgHhS6IOfMfilUO1gfrbeyXinm5Z0Mf+UCHiz//PtLOIuKnuvtyu+rWsAUj niMVnCSl+KlMaZy7gYE1/Me832PTxcRVRQDEQDbaKR7lhSYZ2c5s/fC3nYupfrvh8+6J EHZITOK+52TSeJvkO9g9LMbCQ7KB/P7PH5b3NbTm7iqUPlGnoHPgzlnsNEbU/7/2/0Vk NM2UyuXgCtb9lmnjmCtDAecaYFrBXHXlBtJPSuNB9ZyjbywZYEzJBAbhRZEpDYi4n4G+ JCfpWtKyk3iGUDGetAvIcsNo4Kpf9qpSReZSU/MgsJnhfCfi0bD2I9ZM3OB1Tf+isNe1 lS5Q== X-Gm-Message-State: APjAAAUZLbGPzGcZB1H9uthl9buSzs5kyXm/TU5fKm/f67yncxv+QRnQ n+DXSZBvJx5dlGxlin/sVAX1qgwsR8eb5K6zOD7gyYH1vT8= X-Google-Smtp-Source: APXvYqyrW0vlGEMdrC+JcTL5urcjlpUaqh/v3A/ueyzCuFmFwgyqqVIi3qt2LGPvPUVuAjxZ3LYlyiGbSBRfGE0YWO0= X-Received: by 2002:ac2:46f8:: with SMTP id q24mr8616165lfo.51.1569771353095; Sun, 29 Sep 2019 08:35:53 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: John Fleming Date: Sun, 29 Sep 2019 11:35:41 -0400 Message-ID: Subject: Re: Question about bottle neck in storage To: Warner Losh Cc: FreeBSD-STABLE Mailing List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46h8kh0sCSz4dvL X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=spikefishsolutions.com header.s=google header.b=D2XC9hvo; dmarc=none; spf=pass (mx1.freebsd.org: domain of john@spikefishsolutions.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=john@spikefishsolutions.com X-Spamd-Result: default: False [-5.42 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[spikefishsolutions.com:s=google]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[spikefishsolutions.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[spikefishsolutions.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.92)[ip: (-9.48), ipnet: 2a00:1450::/32(-2.90), asn: 15169(-2.17), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Sep 2019 15:35:57 -0000 On Tue, Sep 24, 2019 at 1:09 PM Warner Losh wrote: > > > > On Tue, Sep 24, 2019 at 5:46 PM John Fleming wrote: >> >> Is there anyway to see how busy a SAS/Sata controller is vs disks? I >> have a R720 with 14 Samsung 860 EVOs in it (its a lab server) in raid >> 10 ZFS. >> >> When firing off a dd I (bs=3D1G count=3D10) seems like the disks never g= o >> above %50 busy. I'm trying to figure out if i'm maxing out SATA 3 BW >> or if its something else (like terrible dd options). > > > Two points to consider here. First, NVMe has lots of queues and needs lot= s of concurrent transactions to saturate, so the 50% busy means you are no = where close to saturating the drives. Schedule more I/O too fix that. It's = better to do lots and lots of concurrent DD to different parts of the drive= , or to use fio with the aio kernel option and posixaio I/O scheduling meth= od. > > I use the following script, but often need to increase the number of thre= ads / jobs to saturate. > > ; SSD testing: 128k I/O 64 jobs 32 deep queue > > [global] > direct=3D1 > rw=3Drandread > refill_buffers > norandommap > randrepeat=3D0 > bs=3D128k > ioengine=3Dposixaio > iodepth=3D32 > numjobs=3D64 > runtime=3D60 > group_reporting > thread > > [ssd128k] > I didn't catch what utilty was using that. I started poking around an iozone and bonnie++ BTW these are SATA not nvme. > Second, the system's % busy statistics are misleading. They are the %of t= he time that a command is outstanding on the drive. 100% busy can be a tiny= percentage of the total bandwidth you can get from the drive. > >> >> my setup is Dell R720 with 2 x LSI 9361 cards. Each card is going to a >> dedicated 8 drive board inside the front of the R720. Basically i'm >> just saying its not a single SAS cable to 14 drives. >> >> Don't have cpu info hand.. zeon something. DDR3-1600 (128GB) >> >> Both controllers are in 8x slots running PCIe gen 3. >> >> BTW i'm sure this has been asked a million times but what would be >> some decent benchmark tests while i'm at it? > > > See above... :) > > Warner So my UPS got angry and shut everything down. I figured this was a good change to look at iostats again. This is while the array is being scrubbed. I'm very happy with these numbers! BTW da0 and 8 are OS drives and not raid 10 members. extended device statistics device r/s w/s kr/s kw/s ms/r ms/w ms/o ms/t qlen %b da0 0 0 0.0 0.0 0 0 0 0 0 0 da1 4003 7 505202.5 207.6 0 0 1 0 2 100 da2 3980 10 508980.2 265.5 0 0 0 0 2 100 da3 3904 8 499675.8 183.1 0 0 0 0 2 99 da4 3850 8 488870.5 263.9 0 0 0 0 2 100 da5 4013 11 513640.6 178.8 0 0 1 0 2 100 da6 3851 13 489035.8 286.4 0 0 1 0 2 100 da7 3931 12 503197.6 271.6 0 0 0 0 2 100 da8 0 0 0.0 0.0 0 0 0 0 0 0 da9 4002 8 505164.1 207.6 0 0 1 0 2 100 da10 3981 10 509133.8 265.5 0 0 0 0 2 100 da11 3905 8 499791.0 183.1 0 0 0 0 2 100 da12 3851 9 488985.6 263.9 0 0 0 0 2 100 da13 4012 11 513576.6 178.8 0 0 1 0 2 100 da14 3850 14 488971.8 286.4 0 0 0 0 2 100 da15 3930 12 503108.0 271.6 0 0 0 0 2 100