Date: Thu, 12 Jul 2018 14:49:53 -0400 From: Mike Tancsa <mike@sentex.net> To: Jim Long <list@museum.rain.com> Cc: freebsd-questions@freebsd.org Subject: Re: Disk/ZFS activity crash on 11.2-STABLE Message-ID: <a069a076-df1c-80b2-1116-787e0a948ed9@sentex.net> In-Reply-To: <20180712183512.GA75020@g5.umpquanet.com> References: <20180711212959.GA81029@g5.umpquanet.com> <5ebd8573-1363-06c7-cbb2-8298b0894319@sentex.net> <20180712183512.GA75020@g5.umpquanet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 7/12/2018 2:35 PM, Jim Long wrote:
>
> Yes, ARC usage is high. top shows:
>
> last pid: 2215; load averages: 1.50, 1.69, 1.69 up 0+00:44:59 11:02:00
> 27 processes: 2 running, 25 sleeping
> CPU: 0.0% user, 0.0% nice, 9.0% system, 0.1% interrupt, 90.9% idle
> Mem: 1416K Active, 11M Inact, 2788K Laundry, 91G Wired, 1236M Free
> ARC: 84G Total, 68M MFU, 83G MRU, 15M Anon, 877M Header, 4078K Other
> 83G Compressed, 148G Uncompressed, 1.78:1 Ratio
> Swap: 16G Total, 15M Used, 16G Free
>
> I'm re-running the test now. I had shut off hyperthreading to go down to
> 24 cores, but it's back on now. This server will be a computational resource,
> not just storage, so I'd like to find a way to stablize the configuration with
> hyperthreading enabled, if possible.
I would leave HT on. Based on the zfs stats
CACHE HITS BY DATA TYPE:
Demand Data: 51.38% 457.85k
Prefetch Data: 0.00% 2
Demand Metadata: 44.71% 398.41k
Prefetch Metadata: 3.91% 34.86k
CACHE MISSES BY DATA TYPE:
Demand Data: 3.22% 17.71k
Prefetch Data: 81.77% 449.51k
Demand Metadata: 8.57% 47.14k
Prefetch Metadata: 6.43% 35.35k
If I read this right, you are not hurting too much for meta data misses.
I find things like
zfs list -t snapshots
REALLY slow down if there is not enough caching for meta data where
there are a lot of files and directories.
eg. on a box that has several million small files and many many
directories, I increased the amount of RAM to vfs.zfs.arc_meta_limit and
it helped a LOT. Even then I still see a lot of misses (it was worse before)
CACHE HITS BY DATA TYPE:
Demand Data: 10.82% 5.58b
Prefetch Data: 0.66% 339.34m
Demand Metadata: 50.83% 26.22b
Prefetch Metadata: 37.69% 19.44b
CACHE MISSES BY DATA TYPE:
Demand Data: 6.79% 680.20m
Prefetch Data: 8.43% 844.48m
Demand Metadata: 79.52% 7.97b
Prefetch Metadata: 5.26% 527.45m
But I dont think thats your case. I would try and set a ceiling. On
RELENG_11 you dont need to reboot
Try
sysctl -w vfs.zfs.arc_max=77946198016
which shaves off 20G from what ARC can gobble up. Not sure if thats your
issue, but it is an issue for some users.
If you are still hurting for caching, an SSD drive or NVME and make it a
caching device for your pool.
and what does
zpool status
show ?
---Mike
--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, mike@sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a069a076-df1c-80b2-1116-787e0a948ed9>