Date: Wed, 15 Jul 2015 12:36:13 +1000 From: Jan Mikkelsen <janm@transactionware.com> To: Karl Denninger <karl@denninger.net> Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD 10.1 Memory Exhaustion Message-ID: <B3777FBA-EF15-49E3-8FCE-468955EF2ED6@transactionware.com> In-Reply-To: <55A3A800.5060904@denninger.net> References: <CAB2_NwCngPqFH4q-YZk00RO_aVF9JraeSsVX3xS0z5EV3YGa1Q@mail.gmail.com> <55A3A800.5060904@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, To make the 2015-02-05 patch work on 10.2-BETA1 I had to add this patch = as well: --- = //depot/vendor/freebsd/10-stable-local/src/sys/cddl/contrib/opensolaris/ut= s/common/fs/zfs/zio.c 2015-07-03 02:03:52.000000000 1000 +++ = /home/janm/p4/freebsd-std/FreeBSD/src/sys/cddl/contrib/opensolaris/uts/com= mon/fs/zfs/zio.c 2015-07-03 02:03:52.000000000 1000 @@ -44,9 +44,9 @@ SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO"); #if defined(__amd64__) -static int zio_use_uma =3D 1; +int zio_use_uma =3D 1; #else -static int zio_use_uma =3D 0; +int zio_use_uma =3D 0; #endif TUNABLE_INT("vfs.zfs.zio.use_uma", &zio_use_uma); SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, = &zio_use_uma, 0, Without changing the storage class of zio_use_uma this error appeared at = runtime "link_elf: symbol zio_use_uma undefined=94. This is unsurprising = because the linker can=92t resolve an extern symbol to a static in = another file. How are other people using this patch, or am I missing = something fundamental? Regards, Jan. > On 13 Jul 2015, at 21:58, Karl Denninger <karl@denninger.net> wrote: >=20 > Put this on your box and see if the problem goes away.... :-) >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D187594 >=20 > The 2015-02-10 refactor will apply against 10.1-STABLE and 10.2-PRE = (the > latter will give you a 10-line fuzz in one block but applies and = works.) >=20 > I've been unable to provoke misbehavior with this patch in and I run a > cron job that does auto-snapshotting. There are others that have run > this patch with similarly positive results. >=20 > On 7/13/2015 06:48, Christopher Forgeron wrote: >> TL;DR Summary: I can run FreeBSD out of memory quite consistently, = and it=92s >> not a TOS/mbuf exhaustion issue. It=92s quite possible that ZFS is = the >> culprit, but shouldn=92t the pager be able to handle aggressive = memory >> requests in a low memory situation gracefully, without needing custom >> tuning of ZFS / VM? >>=20 >>=20 >> Hello, >>=20 >> I=92ve been dealing with some instability in my 10.1-RELEASE and >> STABLEr282701M machines for the last few months. >>=20 >> These machines are NFS/iSCSI storage machines, running on Dell M610x = or >> similar hardware, 96 Gig Memory, 10Gig Network Cards, dual Xeon = Processors >> =96 Fairly beefy stuff. >>=20 >> Initially I thought it was more issues with TOS / jumbo mbufs, as I = had >> this problem last year. I had thought that this was properly = resolved, but >> setting my MTU to 1500, and turning off TOS did give me a bit more >> stability. Currently all my machines are set this way. >>=20 >> Crashes were usually represented by loss of network connectivity, and = the >> ctld daemon scrolling messages across the screen at full speed about = lost >> connections. >>=20 >> All of this did seem like more network stack problems, but with each = crash >> I=92d be able to learn a bit more. >>=20 >> Usually there was nothing of any use in the logfile, but every now = and then >> I=92d get this: >>=20 >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 3 13:02:04 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 3 13:02:04 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> --------- >> Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 4 03:03:09 san0 kernel: WARNING: icl_pdu_new: failed to allocate = 80 >> bytes >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): failed to allocate memory >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping >> connection >> Jun 4 03:03:09 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): connection error; dropping >> connection >> Jun 4 03:03:10 san0 kernel: WARNING: 172.16.0.97 >> (iqn.1998-01.com.vmware:esx5a-3387a188): waiting for CTL to terminate >> tasks, 1 remaining >> Jun 4 06:04:27 san0 syslogd: kernel boot file is /boot/kernel/kernel >>=20 >> So knowing that it seemed to be running out of memory, I started = leaving >> leaving =91vmstat 5=92 running on a console, to see what it was = displaying >> during the crash. >>=20 >> It was always the same thing: >>=20 >> 0 0 0 1520M 4408M 15 0 0 0 25 19 0 0 21962 1667 = 91390 >> 0 33 67 >> 0 0 0 1520M 4310M 9 0 0 0 2 15 3 0 21527 1385 = 95165 >> 0 31 69 >> 0 0 0 1520M 4254M 7 0 0 0 14 19 0 0 17664 1739 = 72873 >> 0 18 82 >> 0 0 0 1520M 4145M 2 0 0 0 0 19 0 0 23557 1447 = 96941 >> 0 36 64 >> 0 0 0 1520M 4013M 4 0 0 0 14 19 0 0 4288 490 = 34685 >> 0 72 28 >> 0 0 0 1520M 3885M 2 0 0 0 0 19 0 0 11141 1038 = 69242 >> 0 52 48 >> 0 0 0 1520M 3803M 10 0 0 0 14 19 0 0 24102 1834 = 91050 >> 0 33 67 >> 0 0 0 1520M 8192B 2 0 0 0 2 15 1 0 19037 1131 = 77470 >> 0 45 55 >> 0 0 0 1520M 8192B 0 22 0 0 2 0 6 0 146 82 = 578 0 >> 0 100 >> 0 0 0 1520M 8192B 1 0 0 0 0 0 0 0 130 40 = 510 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 143 40 = 501 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 201 62 = 660 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 101 28 = 404 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 97 27 = 398 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 93 28 = 377 0 >> 0 100 >> 0 0 0 1520M 8192B 0 0 0 0 0 0 0 0 92 27 = 373 0 >> 0 100 >>=20 >>=20 >> I=92d go from a decent amount of free memory to suddenly having none. = Vmstat >> would stop outputting, console commands would hang, etc. The whole = system >> would be useless. >>=20 >> Looking into this, I came across a similar issue; >>=20 >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D199189 >>=20 >> I started increasing v.v_free_min, and it helped =96 My crashes went = from >> being ~every 6 hours to every few days. >>=20 >> Currently I=92m running with vm.v_free_min=3D1254507 =96 That=92s = (1254507 * 4KiB) >> , or 4.78GiB of Reserve. The vmstat above is of a machine with that >> setting still running to 8B of memory. >>=20 >> I have two issues here: >>=20 >> 1) I don=92t think I should ever be able to run the system into the = ground on >> memory. Deny me new memory until the pager can free more. >> 2) Setting =91min=92 doesn=92t really mean =91min=92 as it can = obviously go below >> that threshold. >>=20 >>=20 >> I have plenty of local UFS swap (non-ZFS drives) >>=20 >> Adrian requested that I output a few more diagnostic items, and this = is >> what I=92m running on a console now, in a loop: >>=20 >> vmstat >> netstat -m >> vmstat -z >> sleep 1 >>=20 >> The output of four crashes are attached here, as they can be a bit = long. >> Let me know if that=92s not a good way to report them. They will each = start >> mid-way through a vmstat =96z output, as that=92s as far back as my = terminal >> buffer allows. >>=20 >>=20 >>=20 >> Now, I have a good idea of the conditions that are causing this: ZFS >> Snapshots, run by cron, during times of high ZFS writes. >>=20 >> The crashes are all nearly on the hour, as that=92s when crontab = triggers my >> python scripts to make new snapshots, and delete old ones. >>=20 >> My average FreeBSD machine has ~ 30 zfs datasets, with each pool = having ~20 >> TiB used. These all need to snapshot on the hour. >>=20 >> By staggering the snapshots by a few minutes, I have been able to = reduce >> crashing from every other day to perhaps once a week if I=92m lucky =96= But if >> I start moving a lot of data around, I can cause daily crashes again. >>=20 >> It=92s looking to be the memory demand of snapshotting lots of ZFS = datasets >> at the same time while accepting a lot of write traffic. >>=20 >> Now perhaps the answer is =91don=92t do that=92 but I feel that = FreeBSD should be >> robust enough to handle this. I don=92t mind tuning for now to >> reduce/eliminate this, but others shouldn=92t run into this pain just = because >> they heavily load their machines =96 There must be a way of avoiding = this >> condition. >>=20 >> Here are the contents of my /boot/loader.conf and sysctl.conf, so = show my >> minimal tuning to make this problem a little more bearable: >>=20 >> /boot/loader.conf >> vfs.zfs.arc_meta_limit=3D49656727553 >> vfs.zfs.arc_max =3D 91489280512 >>=20 >> /etc/sysctl.conf >> vm.v_free_min=3D1254507 >>=20 >>=20 >> Any suggestions/help is appreciated. >>=20 >> Thank you. >>=20 >>=20 >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >=20 > --=20 > Karl Denninger > karl@denninger.net <mailto:karl@denninger.net> > /The Market Ticker/ > /[S/MIME encrypted email preferred]/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B3777FBA-EF15-49E3-8FCE-468955EF2ED6>