From owner-svn-src-all@freebsd.org Thu Apr 14 22:31:02 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91143ADAC90 for ; Thu, 14 Apr 2016 22:31:02 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qg0-x22c.google.com (mail-qg0-x22c.google.com [IPv6:2607:f8b0:400d:c04::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43F0E10AE for ; Thu, 14 Apr 2016 22:31:02 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-qg0-x22c.google.com with SMTP id c6so72113116qga.1 for ; Thu, 14 Apr 2016 15:31:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=bJSbC73NWehKOyWahkAzYitkKpsysPLA0qZyNWewtwo=; b=0auouxlkgT1s16Jfr9mfAY1ip7cpKWr/FoSO5abhzZpZxKCz1/Ibqhh9GAT40Mod8X h6wJ94gKLMyZwEaDB0qc8GrWRbSw7VsJ0DAt+5ZoWAzDBa00RTZJv67gS8bEOC5UXxuA m3pCCHC4PunKJnnAZjenwIItSJ+f006dhm4Sn7BqNvOfipXy6uvEynzWAGdeyUe/Nf4s d2FoLTTFG/OFU5da2HYhG1R4ASpNiOPfDV+8nH+1/TEeR34eEh6oK7zUySZIbwLg5azJ I9e8n9bi0JLrVVZg8RYtQD8iYWfArM6v4I6Yg+uZtej+M1+Uz/lSaomqePFpOqyJ1k9E avnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=bJSbC73NWehKOyWahkAzYitkKpsysPLA0qZyNWewtwo=; b=dSHUFML4ls5+OB6kMioGdARRAsQQjeffsCtiPO92iodSQCm2WqXeEcwmPNhhR5G0ce fg6nwVlE/i0zzqcQ+mpYkjA/f5th6iDDnRf9EXnupL4dv67U5wE87jkaCHsKhQ1xuazM imakIFCfeOp5Up8WQf6gB+72Hkj8D8aSpuxoXBxSxVMh1fps+q2YqPtI7NUpLYoFBMdt 0KElylfPmr9RqlbHxXKHpp+eY03FWXWUIHR8QPfAXM/v9diRikvyGgh1CHQwXWDPFlCy 8hus3Rm3k7ZaybyQzEFku/3XA0LtgYf1Evoc8fuUUrKQIvA9hRJZOaR7L9H0yQVQZnBy cFGw== X-Gm-Message-State: AOPr4FU4CZKTDDUlVRHJySiKIgqxOUFGyTInen/z8aNZPaJgduW7Am9SGVL00N+k5oM1IKgw X-Received: by 10.140.86.101 with SMTP id o92mr21342760qgd.49.1460673061213; Thu, 14 Apr 2016 15:31:01 -0700 (PDT) Received: from mutt-hardenedbsd (c-73-135-80-144.hsd1.md.comcast.net. [73.135.80.144]) by smtp.gmail.com with ESMTPSA id c90sm19141111qkj.47.2016.04.14.15.30.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 14 Apr 2016 15:30:59 -0700 (PDT) Date: Thu, 14 Apr 2016 18:30:57 -0400 From: Shawn Webb To: Warner Losh Cc: Dmitry Morozovsky , "svn-src-head@freebsd.org" , "svn-src-all@freebsd.org" , src-committers , Warner Losh Subject: Re: svn commit: r298002 - in head/sys: cam cam/ata cam/scsi conf dev/ahci Message-ID: <20160414223057.GB66711@mutt-hardenedbsd> References: <201604142147.u3ELlwYo052010@repo.freebsd.org> <20160414221517.GA66711@mutt-hardenedbsd> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="yEPQxsgoJgBvi8ip" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hardenedbsd 11.0-CURRENT-HBSD FreeBSD 11.0-CURRENT-HBSD X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Apr 2016 22:31:02 -0000 --yEPQxsgoJgBvi8ip Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 14, 2016 at 04:24:45PM -0600, Warner Losh wrote: > On Thu, Apr 14, 2016 at 4:15 PM, Shawn Webb > wrote: >=20 > > On Thu, Apr 14, 2016 at 04:04:27PM -0600, Warner Losh wrote: > > > On Thu, Apr 14, 2016 at 3:54 PM, Dmitry Morozovsky > > wrote: > > > > > > > Warner, > > > > > > > > On Thu, 14 Apr 2016, Warner Losh wrote: > > > > > > > > > Author: imp > > > > > Date: Thu Apr 14 21:47:58 2016 > > > > > New Revision: 298002 > > > > > URL: https://svnweb.freebsd.org/changeset/base/298002 > > > > > > > > > > Log: > > > > > New CAM I/O scheduler for FreeBSD. The default I/O scheduler is= the > > > > same > > > > > > > > [snip] > > > > > > > > First, thanks so much for this quite a non-trivial work! > > > > What are the ways to enable this instead of deafult, and what ar the > > > > benefits > > > > and drawbacks? > > > > > > > > > You add CAM_NETFLIX_IOSCHED to your kernel config to enable it. Hmmm, > > > looking at the diff, perhaps I should add that to LINT. > > > > > > In production, we use it for three things. First, our scheduler keeps= a > > lot > > > more > > > statistics than the default one. These statistics are useful for us > > knowing > > > when > > > a system is saturated and needs to shed load. Second, we favor reads = over > > > writes because our workload, as you might imagine, is a read mostly w= ork > > > load. > > > Finally, in some systems, we throttle the write throughput to the SSD= s. > > The > > > SSDs > > > we buy can do 300MB/s write while serving 400MB/s read, but only for > > short > > > periods of time (long enough to do 10-20GB of traffic). After that, w= rite > > > performance > > > drops, and read performance goes out the window. Experiments have sho= wn > > that > > > if we limit the write speed to no more than 30MB/s or so, then the > > garbage > > > collection the drive is doing won't adversely affect the read latency= / > > > performance. > > > > Going on a tangent here, but related: > > > > As someone who is just barely stepping into the world of benchmarks and > > performance metrics, can you shed some light as to how you gained those > > metrics? I'd be extremely interested to learn. > > >=20 > These numbers were derived through an iterative process. All our systems > report > a large number of statistics while they are running. The disk performance > numbers > come from gstat(8) which ultimately derives them from devstat(9). When we > enabled > serving customer traffic while refreshing content, we noticed a large > number of > reports from our playback clients indicating problems with the server > during this > time period. I looked at the graphs to see what was going on. Once I found > the problem, > I was able to see that as the write load varied, the latency numbers for > the reads > would vary substantially as well. I added code to the I/O scheduler so I > could rate > limit the write speed to the SSDs. After running through a number of > different > machines over a number of nights of filling and serving, I was able to fi= nd > the right > number. If I set it to 30MB, the 20 machines I tested didn't have any > reports above > background level of problems. When I set it to 35MB/s there was a couple = of > those > machines that had problems. when I set it to 40MB/s there were a couple > more. When > I set it to 80MB/s, almost all had problems. Being conservative, I set it > to the highest > number that showed no ill effect on the clients. I was able to see large > jumps in read > latency as low as 25MB/s though. Sadly, this is with Netflix internal > tools, but one > could do the same research with gstat and scripting. One could also use > dtrace > to study the latency patterns to a much finer degree of fidelity than gst= at > offers. Awesome. Thank you very much for the informative response. I didn't even know tools like gstat and devstat existed. I think Michael W. Lucas' new book covers some of the DTrace side of things. I'll be taking a look at that, too. "Learn something new everyday" certainly applies here, and thanks for the input. :) Thanks again, --=20 Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --yEPQxsgoJgBvi8ip Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXEBofAAoJEGqEZY9SRW7uGHQQAK4tqKArldtfDYZED9mK803c DgoaTf6LFDEJeZvkiXPZ83vVx+sabs9zb3up+y87PFWmVJYtRRGHxYt98jvjCK7e xgYq9YEyOZxlv9ugjNNYZqHQ0S6P8UmN9HyoD3hIAAoY3biPgfIvYM0lENpAFCBS GOKWUgfVXImrJxxIgXRG4b9j6LidXyBaoxJEU70RbG8Cn6A0FDxQxJhUHXDoSpNJ OqrWZxKPolBWFgrF7vQLwx0zS5D9zxLv/BT1zXzCCjf//CwWfzf2SgOeA1WeTDdR tYsaExtougt5M1pOFKO1nSjATsZ/EfLBe4fQdlQQLn10n9CkGRJr8YhH3xL+ywaG 58TsSN3O2pECmjhY5YnV2en9L/TdnF7z1HdQBcxvTUWdauwCUsLXqxKQG4yPIF6w R+n911ZQaLqQD5d7kY1ycIdchfI/zlgTuZ1oHS3auS/U3Hq2nECJeWkJLFsVbxXD 5Q4P5/RNIqBepHKBGUjoiOQm/1oOFqk8rzUyNY+xDk86E8p61RCxIusJ47POkWRR hNEqbv5lXJYDIHuDIjKJYa5gUYi0qw6DpUDkne5Tp8SfOJoP1zvJQEbxzHTC+GrR 29gdUtB28j1DPbvpO2f2Kp9q4QI6jPo1v4ITmvx34/6oILczuOcjF7E4RUomR5t6 +cuHiMW7Tjtdypu3y29b =vzaT -----END PGP SIGNATURE----- --yEPQxsgoJgBvi8ip--