Date: Thu, 24 Jul 2014 15:11:30 +0200 From: Mark Martinec <Mark.Martinec+freebsd@ijs.si> To: freebsd-net@freebsd.org Subject: Bumping up a default net.graph.maxdata to avoid "Write failed: Cannot allocate memory" Message-ID: <17db180e1fa763f41dd0051780741b9f@mailbox.ijs.si>
next in thread | raw e-mail | index | archive | help
Syncing zfs snapshots across the net using 'zfs send' over ssh started failing one day with ssh reporting "Write failed: Cannot allocate memory" after transferring about 15 to 25 GB of data (as it turned out this snapshot was larger than usual). Neither of the two hosts were particularly low on memory, the receiving end was running 10.0-STABLE amd64, network between the two was a gigabit ethernet, interface em. The problem was repeatable at will. Simplifying the experiment to a: ssh <remote-host> zfs send <snapshot> | wc -c also ended up with the same "Write failed: Cannot allocate memory" on the receiving side after transferring about 20 GB. Doing some other experiments ruled out a potential blame from zfs send. As it turned out (luckily for me, after banging my had over it) I'm not the only one with this problem - it was reported three years ago: http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html http://lists.freebsd.org/pipermail/freebsd-stable/2011-July/063322.html And yes, I too had a smallish virtual host running under VirtualBox on this receiving machine, which was mostly idling. That virtual host was not involved in any of these experiments. So the problem is that NetGraph was running out of space for data queue and net.graph.maxdata needed to be bumped up from a default of 512. My current setting is net.graph.maxdata=2048 and this suffices for reliable transfer of huge files (> 60 GB) over ssh. After one such transfer the vmstat -z reports: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP [...] NetGraph items: 72, 4123, 0, 527, 28, 0, 0 NetGraph data items: 72, 2077, 0, 1023,65650892, 0, 0 while previously the FAIL count was in the hundreds, just as in the problem report from July 2011. And the issue is not limited to ssh, others have reported the same over ftp. Btw, the 'ngctl list' shows: There are 5 total nodes: Name: ngctl78040 Type: socket ID: 00000010 Num hooks: 0 Name: em0 Type: ether ID: 00000001 Num hooks: 2 Name: em1 Type: ether ID: 00000002 Num hooks: 0 Name: vboxnet0 Type: ether ID: 00000003 Num hooks: 0 Name: vboxnetflt_em0 Type: vboxnetflt ID: 00000004 Num hooks: 2 Unfortunately the 2011 thread remained suspended in the air, with no action, neither in documentation nor bumping up the default queue limit. So to save more people from bumping into the same problem, puzzled over a mystery "Write failed: Cannot allocate memory" failure, I'd like to suggest bumping up the default value of net.graph.maxdata, or at least documenting the fact in the handbook. Mark
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?17db180e1fa763f41dd0051780741b9f>