Date: Mon, 18 Jun 2018 15:44:30 +0930 From: Shane Ambler <FreeBSD@ShaneWare.Biz> To: tech-lists <tech-lists@zyxst.net> Cc: freebsd-questions@freebsd.org Subject: Re: lightly loaded system eats swap space Message-ID: <c2297794-1b9e-01b7-c568-ab1752c0a49f@ShaneWare.Biz> In-Reply-To: <c8277497-ffcf-0503-490b-96d1b4605af7@zyxst.net> References: <c8277497-ffcf-0503-490b-96d1b4605af7@zyxst.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18/06/2018 07:49, tech-lists wrote: > Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed Out of swap messages can be misleading, you can also get out of swap messages if too much ram gets wired - wired doesn't get swapped out. Too much wired can also cause everything else to be paged out. How do you start bhyve instances? A previous issue used one of the bhyve helper ports that used -S to wire all guest memory as a default setting. > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, You have 116G wired of 128G, this would be the cause behind everything else going to swap. 108G of that wired looks to be arc. > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse There is `sysctl vm.max_wired` which limits the kernel wired memory, the arc is then also wired but added on top of that setting, if your guests are then wiring more ram, you can get too much wired. There is a patch in review that attempts to improve arc releasing when free ram is low. See if this helps - https://reviews.freebsd.org/D7538 Other than that check that your guest ram + max_wired + arc_max adds up to less than your physical ram minus some host system ram. Note that wax_wired is a page count so multiply it by hw.pagesize -- FreeBSD - the place to B...Software Developing Shane Ambler
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c2297794-1b9e-01b7-c568-ab1752c0a49f>