Date: Thu, 27 Mar 2014 06:52:38 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix Message-ID: <53341106.4060101@denninger.net> In-Reply-To: <8659e58b9fabd9f553c8be5da5dc61fd@mail.mikej.com> References: <201403261230.s2QCU3vI095105@freefall.freebsd.org> <8659e58b9fabd9f553c8be5da5dc61fd@mail.mikej.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 3/27/2014 4:11 AM, mikej wrote: > I've been running the latest patch now on r263711 and want to give it > a +1 > > No ZFS knobs set and I must go out of my way to have my system swap. > > I hope this patch gets a much wider review and can be put into the > tree permanently. > > Karl, thanks for the working on this. > > Regards, > > Michael Jung No problem; I was being driven insane by the stalls and related bad behavior... and there's that old saw about complaining about something without proposing a fix for it (I've done it!) being "less than optimum" so.... :-) Hopefully wider review (and, if the general consensus is similar to what I've seen here and what you're reporting as well, inclusion in the codebase) will come. On my sandbox system I have to get truly abusive before I can get the system to swap now, but that load is synthetic and we all know what sometimes happens when you try to extrapolate from synthetic loads to real production ones. What really has my attention is the impact on systems running live production loads. It has entirely changed the character of those machines, working equally-well for both pure ZFS machines and mixed UFS/ZFS systems. One of these systems that gets pounded on pretty good and has a moderately-large configuration (~10TB of storage, 2 Xeon quad-core processors and 24GB of RAM serving a combination of Samba users internally, a decently-large Postgres installation supporting an externally-facing web forum and blog application, email and similar things) has been completely transformed from being "frequently challenged" by its workload to literally loafing 90%+ of the day. DBMS response times have seen their standard deviation drop by an order of magnitude with best-response times down for one of the most-common query sequences (~30 separate ops) from ~180ms to ~140. This particular machine has a separate pool for the system itself (root, usr and var) which was formerly UFS because it had to be in order to avoid the worst of the "stall" bad behavior. It also has two other pools on it, one for read-nearly-only data sets that are comprised of very large files that are almost archival in character and a second that has the system's "working set" on it. The latter has a separate intent log; I had a cache SSD drive on it as well but have recently dropped that as with these changes it no longer produces a material improvement in performance. I'm frankly not sure the intent log is helping any more either but I've yet to drop it and instrument the results -- it used to be *necessary* to avoid nasty problems during busy periods. I now have that machine set up booting from ZFS with the system on a mirrored pool dedicated to system images, with lz4 *and* dedup on (for that filesystem's root), which allows me to clone it almost instantly, start a jail on the clone and then do a "buildworld buildkernel -j8" while only allocating storage to actual changes. Dedup ratio on that mirror set is 1.4x and lz4 is showing a net compression ratio of 2.01x. Even better I cannot provoke misbehavior by doing this sort of thing during the middle of the day where formerly that was just begging for trouble; the impact on user perceptible performance during it is zero although I can see the degradation in performance (a modest increase in system latency) in the stats. Oh, did I mention that everything except the boot/root/usr/var filesystems (including swap) are geli-encrypted on this machine as well and that the nightly PC backup jobs bury the GIG-E interface on which they're attached -- and sustain that performance against the ZFS disks for the duration? (The machine does have AESNI loaded....) Finally swap allocation remains at zero throughout all of this. At present, coming off the overnight that has an activity spike for routine in-house backup activity from connected PCs but is otherwise the "low point" of activity shows 1GB of free memory, an "auto-tuned" amount of 12.9GB of ARC cache (with a maximum size of 22.3) and inactive pages have remained stable. Wired memory is almost 19GB with Postgres using a sizable chunk of it. Cache efficiency is claimed to be 98.9% (!) That'll go down somewhat over the day but during the busiest part of the day it remains well into the 90s which I'm sure has a heck of a lot to do with the performance improvements.... Cross-posted over to -STABLE in the hope of expanding review and testing by others. -- -- Karl karl@denninger.net [-- Attachment #2 --] 0 *H 01 0 + 0 *H O0K030 *H 01 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 130824190344Z 180823190344Z0[1 0 UUS10UFlorida10UKarl Denninger1!0 *H karl@denninger.net0"0 *H 0 bi՞] MNԿawx?`)'ҴcWgR@BlWh+ u}ApdCF JVй~FOL }EW^bچYp3K& ׂ(R lxڝ.xz?6 &nsJ +1v9 v/( kqĪp[vjcK%fϻe?iq]z lyzFO'ppdX//Lw(3 JI A*S#՟H[f|CGqJKoo y.oEuOw$ /섀$삻J9b|A P~8]D1YI<"""Y^T2iQ2 b yH)] Ƶ0y$ _N 6XqMC 9 XgώjGTP"#nˋ"Bk1 00 U 0 0 `HB0 U 0, `HB OpenSSL Generated Certificate0 U |8 ˴d[20 U #0]Af4 U3x&^"408 `HB+)https://cudasystems.net:11443/revoked.crl0 *H gBwH] j\x`( &gW32"Uf^. ^Iϱ k!DQA g{(w /)\N'[ oRW @C HO>)XrTNɘ!u`xt5(=f\-l3<@C6mnhv##1ŃbH͍_Nq aʷ?rk$^9TIa!kh,D -ct1 0001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 + ;0 *H 1 *H 0 *H 1 140327115238Z0# *H 1-dXmN0l *H 1_0]0 `He*0 `He0 *H 0*H 0 *H @0+0 *H (0 +71001 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H 101 0 UUS10UFlorida10U Niceville10U Cuda Systems LLC1 0UCuda Systems LLC CA1/0- *H customer-service@cudasystems.net0 *H ohӨ[deZ-P* #Ec=lPeɽxUEI*+1N _5Sz(||4l#, mB6b\]J#MfK 1F_ Cz{ qML7J ^iWQ|}܌m _,[wy$UCqpF>~,jv- u_euC@Ć 4'#X 2puڑ#7xogȮ݆W֗iOzD]C`Z9_ȨBJEvQ1oƜ0 }pn.d~gڏ];:~: ;Fʈ& XDLjD}9LucRQR*;|; :BkIv`MX9 <vk{tD z) DFz*>qI7:z{22k;"[t7eC~fs7J)'Ue
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53341106.4060101>