Date: Thu, 14 Apr 2016 18:30:57 -0400 From: Shawn Webb <shawn.webb@hardenedbsd.org> To: Warner Losh <imp@bsdimp.com> Cc: Dmitry Morozovsky <marck@rinet.ru>, "svn-src-head@freebsd.org" <svn-src-head@freebsd.org>, "svn-src-all@freebsd.org" <svn-src-all@freebsd.org>, src-committers <src-committers@freebsd.org>, Warner Losh <imp@freebsd.org> Subject: Re: svn commit: r298002 - in head/sys: cam cam/ata cam/scsi conf dev/ahci Message-ID: <20160414223057.GB66711@mutt-hardenedbsd> In-Reply-To: <CANCZdfqMDHbDacSuSZ-ALgZgzqYyXucj0NMisG=kVNo1d2M%2BXw@mail.gmail.com> References: <201604142147.u3ELlwYo052010@repo.freebsd.org> <alpine.BSF.2.00.1604150051440.76365@woozle.rinet.ru> <CANCZdfp8iJJoVZd-xQXqxpVsB5PiocTTcFVW-Vxkn6zX3OMUfw@mail.gmail.com> <20160414221517.GA66711@mutt-hardenedbsd> <CANCZdfqMDHbDacSuSZ-ALgZgzqYyXucj0NMisG=kVNo1d2M%2BXw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--yEPQxsgoJgBvi8ip Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 14, 2016 at 04:24:45PM -0600, Warner Losh wrote: > On Thu, Apr 14, 2016 at 4:15 PM, Shawn Webb <shawn.webb@hardenedbsd.org> > wrote: >=20 > > On Thu, Apr 14, 2016 at 04:04:27PM -0600, Warner Losh wrote: > > > On Thu, Apr 14, 2016 at 3:54 PM, Dmitry Morozovsky <marck@rinet.ru> > > wrote: > > > > > > > Warner, > > > > > > > > On Thu, 14 Apr 2016, Warner Losh wrote: > > > > > > > > > Author: imp > > > > > Date: Thu Apr 14 21:47:58 2016 > > > > > New Revision: 298002 > > > > > URL: https://svnweb.freebsd.org/changeset/base/298002 > > > > > > > > > > Log: > > > > > New CAM I/O scheduler for FreeBSD. The default I/O scheduler is= the > > > > same > > > > > > > > [snip] > > > > > > > > First, thanks so much for this quite a non-trivial work! > > > > What are the ways to enable this instead of deafult, and what ar the > > > > benefits > > > > and drawbacks? > > > > > > > > > You add CAM_NETFLIX_IOSCHED to your kernel config to enable it. Hmmm, > > > looking at the diff, perhaps I should add that to LINT. > > > > > > In production, we use it for three things. First, our scheduler keeps= a > > lot > > > more > > > statistics than the default one. These statistics are useful for us > > knowing > > > when > > > a system is saturated and needs to shed load. Second, we favor reads = over > > > writes because our workload, as you might imagine, is a read mostly w= ork > > > load. > > > Finally, in some systems, we throttle the write throughput to the SSD= s. > > The > > > SSDs > > > we buy can do 300MB/s write while serving 400MB/s read, but only for > > short > > > periods of time (long enough to do 10-20GB of traffic). After that, w= rite > > > performance > > > drops, and read performance goes out the window. Experiments have sho= wn > > that > > > if we limit the write speed to no more than 30MB/s or so, then the > > garbage > > > collection the drive is doing won't adversely affect the read latency= / > > > performance. > > > > Going on a tangent here, but related: > > > > As someone who is just barely stepping into the world of benchmarks and > > performance metrics, can you shed some light as to how you gained those > > metrics? I'd be extremely interested to learn. > > >=20 > These numbers were derived through an iterative process. All our systems > report > a large number of statistics while they are running. The disk performance > numbers > come from gstat(8) which ultimately derives them from devstat(9). When we > enabled > serving customer traffic while refreshing content, we noticed a large > number of > reports from our playback clients indicating problems with the server > during this > time period. I looked at the graphs to see what was going on. Once I found > the problem, > I was able to see that as the write load varied, the latency numbers for > the reads > would vary substantially as well. I added code to the I/O scheduler so I > could rate > limit the write speed to the SSDs. After running through a number of > different > machines over a number of nights of filling and serving, I was able to fi= nd > the right > number. If I set it to 30MB, the 20 machines I tested didn't have any > reports above > background level of problems. When I set it to 35MB/s there was a couple = of > those > machines that had problems. when I set it to 40MB/s there were a couple > more. When > I set it to 80MB/s, almost all had problems. Being conservative, I set it > to the highest > number that showed no ill effect on the clients. I was able to see large > jumps in read > latency as low as 25MB/s though. Sadly, this is with Netflix internal > tools, but one > could do the same research with gstat and scripting. One could also use > dtrace > to study the latency patterns to a much finer degree of fidelity than gst= at > offers. Awesome. Thank you very much for the informative response. I didn't even know tools like gstat and devstat existed. I think Michael W. Lucas' new book covers some of the DTrace side of things. I'll be taking a look at that, too. "Learn something new everyday" certainly applies here, and thanks for the input. :) Thanks again, --=20 Shawn Webb HardenedBSD GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --yEPQxsgoJgBvi8ip Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJXEBofAAoJEGqEZY9SRW7uGHQQAK4tqKArldtfDYZED9mK803c DgoaTf6LFDEJeZvkiXPZ83vVx+sabs9zb3up+y87PFWmVJYtRRGHxYt98jvjCK7e xgYq9YEyOZxlv9ugjNNYZqHQ0S6P8UmN9HyoD3hIAAoY3biPgfIvYM0lENpAFCBS GOKWUgfVXImrJxxIgXRG4b9j6LidXyBaoxJEU70RbG8Cn6A0FDxQxJhUHXDoSpNJ OqrWZxKPolBWFgrF7vQLwx0zS5D9zxLv/BT1zXzCCjf//CwWfzf2SgOeA1WeTDdR tYsaExtougt5M1pOFKO1nSjATsZ/EfLBe4fQdlQQLn10n9CkGRJr8YhH3xL+ywaG 58TsSN3O2pECmjhY5YnV2en9L/TdnF7z1HdQBcxvTUWdauwCUsLXqxKQG4yPIF6w R+n911ZQaLqQD5d7kY1ycIdchfI/zlgTuZ1oHS3auS/U3Hq2nECJeWkJLFsVbxXD 5Q4P5/RNIqBepHKBGUjoiOQm/1oOFqk8rzUyNY+xDk86E8p61RCxIusJ47POkWRR hNEqbv5lXJYDIHuDIjKJYa5gUYi0qw6DpUDkne5Tp8SfOJoP1zvJQEbxzHTC+GrR 29gdUtB28j1DPbvpO2f2Kp9q4QI6jPo1v4ITmvx34/6oILczuOcjF7E4RUomR5t6 +cuHiMW7Tjtdypu3y29b =vzaT -----END PGP SIGNATURE----- --yEPQxsgoJgBvi8ip--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160414223057.GB66711>