Date: Thu, 5 Nov 2015 12:39:52 +0100 (CET) From: elof2@sentor.se To: Christian Peron <csjp@sqrt.ca> Cc: freebsd-net <freebsd-net@FreeBSD.org> Subject: Re: netstat -B "Recv" Message-ID: <alpine.BSF.2.00.1511051052120.49057@farmermaggot.shire.sentor.se> In-Reply-To: <04EF5880-800D-4A52-8CD1-DAF87EB6E37A@sqrt.ca> References: <alpine.BSF.2.00.1511041736240.49057@farmermaggot.shire.sentor.se> <04EF5880-800D-4A52-8CD1-DAF87EB6E37A@sqrt.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Christian. Sorry, I was in a bit of a hurry when I wrote the below. Here's more information: FreeBSD 10.1 amd64 (on both boxes). In rc.conf on the receiver I have: ifconfig_ix0="up -arp -lro" ifconfig_ix1="up -arp -lro" cloned_interfaces="bridge0" ifconfig_bridge0="up addm ix0 -discover ix0 -learn ix0 private ix0 \ addm ix1 -discover ix1 -learn ix1 private ix1 \ maxaddr 1 -arp monitor name mon0" So 'mon0' is a monitor interface consisting of ix0 and ix1. During my test, ix0 has no carrier, only ix1 is in use. On the sending box I run tcpreplay to send 2000000 packets for each run. Naturally I run the netstat commands on the receiving machine after the tcpreplay on the other machine has finished, not during. :-) I reboot to start over from scratch. I start a sniffer: # tcpdump -nli mon0 -w /dev/null # netstat -in Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 1500 <Link#1> 0c:c4:7a:58:e2:3c 0 0 0 0 0 0 ix1 1500 <Link#2> 0c:c4:7a:58:e2:3d 0 0 0 0 0 0 mon0 1500 <Link#8> 02:74:e9:56:9c:00 0 0 0 0 0 0 # netstat -B Pid Netif Flags Recv Drop Match Sblen Hblen Command 1422 mon0 p--s--- 0 0 0 0 0 tcpdump I send 2000000 packets. The receiver now show: # netstat -in Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 1500 <Link#1> 0c:c4:7a:58:e2:3c 0 0 0 0 0 0 ix1 1500 <Link#2> 0c:c4:7a:58:e2:3d 2000000 0 0 0 0 0 mon0 1500 <Link#8> 02:74:e9:56:9c:00 2000000 0 0 0 0 0 # netstat -B Pid Netif Flags Recv Drop Match Sblen Hblen Command 1422 mon0 p--s--- 1999851 0 2000000 0 0 tcpdump I press ctrl-c in tcpdump and it says: 2000000 packets captured 1999851 packets received by filter 0 packets dropped by kernel So the BPF stats from both netstat and tcpdump look quite strange. :-) I start a new tcpdump and send 2000000 new packets. I now run 'netstat -B' a few times during the transfer, and one can see that the diffing packets are evenly distributed over time: Pid Netif Flags Recv Drop Match Sblen Hblen Command 1557 mon0 p--s--- 0 0 0 0 0 tcpdump 1557 mon0 p--s--- 383905 0 383923 1993540 0 tcpdump 1557 mon0 p--s--- 651010 0 651038 1112814 0 tcpdump 1557 mon0 p--s--- 944090 0 944128 2883633 0 tcpdump 1557 mon0 p--s--- 1125245 0 1125292 2034033 0 tcpdump 1557 mon0 p--s--- 1319456 0 1319514 467992 0 tcpdump 1557 mon0 p--s--- 1516545 0 1516618 4183543 0 tcpdump 1557 mon0 p--s--- 1760527 0 1760608 1365257 0 tcpdump 1557 mon0 p--s--- 1999908 0 2000000 2870940 0 tcpdump 1557 mon0 p--s--- 1999908 0 2000000 0 0 tcpdump 1557 mon0 p--s--- 1999908 0 2000000 0 0 tcpdump ctrl-c on tcpdump says: 2000000 packets captured 1999908 packets received by filter 0 packets dropped by kernel I run a tcpdump directly on ix1, instead of the bridge, and try again. Same thing, so the bridge has nothing to do with it: Pid Netif Flags Recv Drop Match Sblen Hblen Command 1819 ix1 p--s--- 1999904 0 2000000 0 0 tcpdump ^C 2000000 packets captured 1999904 packets received by filter 0 packets dropped by kernel I rebooted it again, started tcpdump on ix1 and received 2000000 packets. By accident I forgot to kill other processes that might have an impact on my tests. Maybe that was a good thing, because 'netstat -B' now show: Pid Netif Flags Recv Drop Match Sblen Hblen Command 921 mon0 p--s--- 2000000 76390 2000000 0 0 suricata 1358 ix1 p--s--- 1999897 0 2000000 0 0 tcpdump So the BPF stats for suricata is showing 2000000 received and 2000000 matched (and a few drops) as it should. Just to see if there would be any difference, I tried running tcpdump linebuffered but throwing away the output: # tcpdump -nli ix1 > /dev/null # netstat -B Pid Netif Flags Recv Drop Match Sblen Hblen Command 921 mon0 p--s--- 3999997 214356 4000000 0 0 suricata 1521 ix1 p--s--- 1999875 46200 2000000 0 0 tcpdump ^C 1953800 packets captured 1999875 packets received by filter 46200 packets dropped by kernel Nope, same-same. (I don't know what I was expecting) Since tcpdump is the one showing odd values I test with tshark (dumpcap) instead: Pid Netif Flags Recv Drop Match Sblen Hblen Command 921 mon0 p--s--- 5999996 473528 6000000 0 0 suricata 1624 ix1 p--s--- 1999920 0 2000000 0 0 dumpcap Nope same problem there. Tshark received 2000000 packets but for some reason, the BPF stats show 1999920 received packets. Also, this run suricata also had the same problem, so maybe it was just a coincidence that it showed correct values that first run. I try again, this time with ngrep as the sniffer: # ngrep -d ix1 > /dev/null Pid Netif Flags Recv Drop Match Sblen Hblen Command 921 mon0 p--s--- 7999950 766788 8000000 0 0 suricata 1730 ix1 p--s--- 1999862 0 0 0 0 ngrep Ngrep also show too few recv packets (and zero matches for some reason). Suricata recv-stats had problems this run as well, so it must have been a fluke that it got correct values above. Here's some ix stats, in case they should reveal something interesting: # netstat -in Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ix0 1500 <Link#1> 0c:c4:7a:58:e2:3c 0 0 0 0 0 0 ix1 1500 <Link#2> 0c:c4:7a:58:e2:3d 12000000 0 0 0 0 0 mon0 1500 <Link#8> 02:74:e9:56:9c:00 11999998 0 0 0 0 0 # sysctl -a | egrep "ix\.|ixgbe" | grep -v "ix\.0" | more device ixgbe hw.ix.enable_aim: 1 hw.ix.max_interrupt_rate: 31250 hw.ix.rx_process_limit: 256 hw.ix.tx_process_limit: 256 hw.ix.enable_msix: 1 hw.ix.num_queues: 8 hw.ix.txd: 2048 hw.ix.rxd: 2048 dev.ix.%parent: dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15 dev.ix.1.%driver: ix dev.ix.1.%location: slot=0 function=1 dev.ix.1.%pnpinfo: vendor=0x8086 device=0x1528 subvendor=0x15d9 subdevice=0x0734 class=0x020000 dev.ix.1.%parent: pci1 dev.ix.1.fc: 3 dev.ix.1.enable_aim: 1 dev.ix.1.advertise_speed: 0 dev.ix.1.ts: 0 dev.ix.1.dropped: 0 dev.ix.1.mbuf_defrag_failed: 0 dev.ix.1.watchdog_events: 0 dev.ix.1.link_irq: 2 dev.ix.1.queue0.interrupt_rate: 500000 dev.ix.1.queue0.irqs: 1500836 dev.ix.1.queue0.txd_head: 0 dev.ix.1.queue0.txd_tail: 0 dev.ix.1.queue0.tso_tx: 0 dev.ix.1.queue0.no_tx_dma_setup: 0 dev.ix.1.queue0.no_desc_avail: 0 dev.ix.1.queue0.tx_packets: 0 dev.ix.1.queue0.rxd_head: 1704 dev.ix.1.queue0.rxd_tail: 1703 dev.ix.1.queue0.rx_packets: 1500840 dev.ix.1.queue0.rx_bytes: 676084080 dev.ix.1.queue0.rx_copies: 959400 dev.ix.1.queue0.lro_queued: 0 dev.ix.1.queue0.lro_flushed: 0 dev.ix.1.queue1.interrupt_rate: 500000 dev.ix.1.queue1.irqs: 971514 dev.ix.1.queue1.txd_head: 0 dev.ix.1.queue1.txd_tail: 0 dev.ix.1.queue1.tso_tx: 0 dev.ix.1.queue1.no_tx_dma_setup: 0 dev.ix.1.queue1.no_desc_avail: 0 dev.ix.1.queue1.tx_packets: 0 dev.ix.1.queue1.rxd_head: 768 dev.ix.1.queue1.rxd_tail: 767 dev.ix.1.queue1.rx_packets: 971520 dev.ix.1.queue1.rx_bytes: 347213640 dev.ix.1.queue1.rx_copies: 663480 dev.ix.1.queue1.lro_queued: 0 dev.ix.1.queue1.lro_flushed: 0 dev.ix.1.queue2.interrupt_rate: 500000 dev.ix.1.queue2.irqs: 2178838 dev.ix.1.queue2.txd_head: 0 dev.ix.1.queue2.txd_tail: 0 dev.ix.1.queue2.tso_tx: 0 dev.ix.1.queue2.no_tx_dma_setup: 0 dev.ix.1.queue2.no_desc_avail: 0 dev.ix.1.queue2.tx_packets: 0 dev.ix.1.queue2.rxd_head: 1816 dev.ix.1.queue2.rxd_tail: 1815 dev.ix.1.queue2.rx_packets: 2178840 dev.ix.1.queue2.rx_bytes: 1161300000 dev.ix.1.queue2.rx_copies: 1239240 dev.ix.1.queue2.lro_queued: 0 dev.ix.1.queue2.lro_flushed: 0 dev.ix.1.queue3.interrupt_rate: 500000 dev.ix.1.queue3.irqs: 2644559 dev.ix.1.queue3.txd_head: 0 dev.ix.1.queue3.txd_tail: 0 dev.ix.1.queue3.tso_tx: 0 dev.ix.1.queue3.no_tx_dma_setup: 0 dev.ix.1.queue3.no_desc_avail: 0 dev.ix.1.queue3.tx_packets: 0 dev.ix.1.queue3.rxd_head: 592 dev.ix.1.queue3.rxd_tail: 591 dev.ix.1.queue3.rx_packets: 2644560 dev.ix.1.queue3.rx_bytes: 1756521840 dev.ix.1.queue3.rx_copies: 1032240 dev.ix.1.queue3.lro_queued: 0 dev.ix.1.queue3.lro_flushed: 0 dev.ix.1.queue4.interrupt_rate: 500000 dev.ix.1.queue4.irqs: 1001156 dev.ix.1.queue4.txd_head: 0 dev.ix.1.queue4.txd_tail: 0 dev.ix.1.queue4.tso_tx: 0 dev.ix.1.queue4.no_tx_dma_setup: 0 dev.ix.1.queue4.no_desc_avail: 0 dev.ix.1.queue4.tx_packets: 0 dev.ix.1.queue4.rxd_head: 1736 dev.ix.1.queue4.rxd_tail: 1735 dev.ix.1.queue4.rx_packets: 1001160 dev.ix.1.queue4.rx_bytes: 566511600 dev.ix.1.queue4.rx_copies: 527520 dev.ix.1.queue4.lro_queued: 0 dev.ix.1.queue4.lro_flushed: 0 dev.ix.1.queue5.interrupt_rate: 500000 dev.ix.1.queue5.irqs: 1141199 dev.ix.1.queue5.txd_head: 0 dev.ix.1.queue5.txd_tail: 0 dev.ix.1.queue5.tso_tx: 0 dev.ix.1.queue5.no_tx_dma_setup: 0 dev.ix.1.queue5.no_desc_avail: 0 dev.ix.1.queue5.tx_packets: 0 dev.ix.1.queue5.rxd_head: 464 dev.ix.1.queue5.rxd_tail: 463 dev.ix.1.queue5.rx_packets: 1141200 dev.ix.1.queue5.rx_bytes: 759726240 dev.ix.1.queue5.rx_copies: 564600 dev.ix.1.queue5.lro_queued: 0 dev.ix.1.queue5.lro_flushed: 0 dev.ix.1.queue6.interrupt_rate: 500000 dev.ix.1.queue6.irqs: 1361040 dev.ix.1.queue6.txd_head: 0 dev.ix.1.queue6.txd_tail: 0 dev.ix.1.queue6.tso_tx: 0 dev.ix.1.queue6.no_tx_dma_setup: 0 dev.ix.1.queue6.no_desc_avail: 0 dev.ix.1.queue6.tx_packets: 0 dev.ix.1.queue6.rxd_head: 1168 dev.ix.1.queue6.rxd_tail: 1167 dev.ix.1.queue6.rx_packets: 1361040 dev.ix.1.queue6.rx_bytes: 772066560 dev.ix.1.queue6.rx_copies: 482640 dev.ix.1.queue6.lro_queued: 0 dev.ix.1.queue6.lro_flushed: 0 dev.ix.1.queue7.interrupt_rate: 500000 dev.ix.1.queue7.irqs: 1200826 dev.ix.1.queue7.txd_head: 0 dev.ix.1.queue7.txd_tail: 0 dev.ix.1.queue7.tso_tx: 0 dev.ix.1.queue7.no_tx_dma_setup: 0 dev.ix.1.queue7.no_desc_avail: 0 dev.ix.1.queue7.tx_packets: 0 dev.ix.1.queue7.rxd_head: 712 dev.ix.1.queue7.rxd_tail: 711 dev.ix.1.queue7.rx_packets: 1200840 dev.ix.1.queue7.rx_bytes: 744130800 dev.ix.1.queue7.rx_copies: 660720 dev.ix.1.queue7.lro_queued: 0 dev.ix.1.queue7.lro_flushed: 0 dev.ix.1.mac_stats.crc_errs: 0 dev.ix.1.mac_stats.ill_errs: 0 dev.ix.1.mac_stats.byte_errs: 0 dev.ix.1.mac_stats.short_discards: 0 dev.ix.1.mac_stats.local_faults: 7 dev.ix.1.mac_stats.remote_faults: 2 dev.ix.1.mac_stats.rec_len_errs: 0 dev.ix.1.mac_stats.xon_txd: 0 dev.ix.1.mac_stats.xon_recvd: 0 dev.ix.1.mac_stats.xoff_txd: 0 dev.ix.1.mac_stats.xoff_recvd: 0 dev.ix.1.mac_stats.total_octets_rcvd: 6831554760 dev.ix.1.mac_stats.good_octets_rcvd: 6831554760 dev.ix.1.mac_stats.total_pkts_rcvd: 12000000 dev.ix.1.mac_stats.good_pkts_rcvd: 12000000 dev.ix.1.mac_stats.mcast_pkts_rcvd: 12480 dev.ix.1.mac_stats.bcast_pkts_rcvd: 33720 dev.ix.1.mac_stats.rx_frames_64: 305160 dev.ix.1.mac_stats.rx_frames_65_127: 4237560 dev.ix.1.mac_stats.rx_frames_128_255: 2372520 dev.ix.1.mac_stats.rx_frames_256_511: 252720 dev.ix.1.mac_stats.rx_frames_512_1023: 1553160 dev.ix.1.mac_stats.rx_frames_1024_1522: 3278880 dev.ix.1.mac_stats.recv_undersized: 0 dev.ix.1.mac_stats.recv_fragmented: 0 dev.ix.1.mac_stats.recv_oversized: 0 dev.ix.1.mac_stats.recv_jabberd: 0 dev.ix.1.mac_stats.management_pkts_rcvd: 0 dev.ix.1.mac_stats.management_pkts_drpd: 0 dev.ix.1.mac_stats.checksum_errs: 0 dev.ix.1.mac_stats.good_octets_txd: 0 dev.ix.1.mac_stats.total_pkts_txd: 0 dev.ix.1.mac_stats.good_pkts_txd: 0 dev.ix.1.mac_stats.bcast_pkts_txd: 0 dev.ix.1.mac_stats.mcast_pkts_txd: 0 dev.ix.1.mac_stats.management_pkts_txd: 0 dev.ix.1.mac_stats.tx_frames_64: 0 dev.ix.1.mac_stats.tx_frames_65_127: 0 dev.ix.1.mac_stats.tx_frames_128_255: 0 dev.ix.1.mac_stats.tx_frames_256_511: 0 dev.ix.1.mac_stats.tx_frames_512_1023: 0 dev.ix.1.mac_stats.tx_frames_1024_1522: 0 PS: I'm not using zerocopy yet (that's the next step and part of the reason for my tests in the first place). # sysctl -a | grep bpf device bpf net.bpf.maxinsns: 512 net.bpf.zerocopy_enable: 0 net.bpf.optimize_writers: 0 net.bpf.bufsize: 4194304 net.bpf.maxbufsize: 2138815488 I hope this information help you understand what is going on. /Elof On Wed, 4 Nov 2015, Christian Peron wrote: > Your assumptions are correct, recv count is the number of packets which are received by the bpf peer, match count is the number of packets which matched the bpf filter that is active.. this shouldn’t be happening. Are you running netstat during the tcpreplay run or after it completed? Also can you give me more information on what specifically mon0 is? > > >> On Nov 4, 2015, at 10:44 AM, elof2@sentor.se wrote: >> >> >> Hi! >> >> Question: >> What do the Recv column in 'netstat -B' show? >> >> I thought it was tha amount of packets received, but appaently not so. >> >> I send 2000000 packets from a tcpreplay machine to a receiving machine. >> I do it a few times. >> >> On the receiver I see: >> netstat -in >> Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll >> ix0 1500 <Link#1> 0c:c4:7a:58:e2:3c 0 0 0 0 0 0 >> ix1 1500 <Link#2> 0c:c4:7a:58:e2:3d 6000000 0 0 0 0 0 >> >> and then >> netstat -in >> Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll >> ix0 1500 <Link#1> 0c:c4:7a:58:e2:3c 0 0 0 0 0 0 >> ix1 1500 <Link#2> 0c:c4:7a:58:e2:3d 8000000 0 0 0 0 0 >> >> So 6000000 has increased to 8000000. Good. >> >> >> However, 'netstat -B' show: >> Pid Netif Flags Recv Drop Match Sblen Hblen Command >> 25553 mon0 p--s--- 1996862 0 2000000 0 0 tcpdump >> >> How can the "Recv" be *lower* than "Match"? >> 1996862 < 2000000. >> >> >> For every new run (fast and slow) I get the same results, slightly less than 2000000 Recv. >> >> What am I missing? >> >> /Elof >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Thu Nov 5 12:09:48 2015 Return-Path: <owner-freebsd-net@freebsd.org> Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64425A25F6B for <freebsd-net@mailman.ysv.freebsd.org>; Thu, 5 Nov 2015 12:09:48 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) Received: from relay2.tomsk.ru (mail.sibptus.tomsk.ru [212.73.124.5]) by mx1.freebsd.org (Postfix) with ESMTP id 8C6B41AB6; Thu, 5 Nov 2015 12:09:46 +0000 (UTC) (envelope-from vas@mpeks.tomsk.su) X-Virus-Scanned: by clamd daemon 0.98.5_1 for FreeBSD at relay2.tomsk.ru Received: from admin.sibptus.TOMSK.ru ([212.73.125.240] verified) by relay2.tomsk.ru (CommuniGate Pro SMTP 5.1.16) with ESMTPS id 38946626; Thu, 05 Nov 2015 18:09:44 +0600 Received: from admin.sibptus.TOMSK.ru (sudakov@localhost [127.0.0.1]) by admin.sibptus.TOMSK.ru (8.14.9/8.14.7) with ESMTP id tA5C9gXf016395; Thu, 5 Nov 2015 18:09:44 +0600 (NOVT) (envelope-from vas@mpeks.tomsk.su) Received: (from sudakov@localhost) by admin.sibptus.TOMSK.ru (8.14.9/8.14.7/Submit) id tA5C9gLw016394; Thu, 5 Nov 2015 18:09:42 +0600 (NOVT) (envelope-from vas@mpeks.tomsk.su) X-Authentication-Warning: admin.sibptus.TOMSK.ru: sudakov set sender to vas@mpeks.tomsk.su using -f Date: Thu, 5 Nov 2015 18:09:42 +0600 From: Victor Sudakov <vas@mpeks.tomsk.su> To: Julian Elischer <julian@freebsd.org> Cc: Neel Natu <neelnatu@gmail.com>, freebsd-net <freebsd-net@freebsd.org> Subject: Re: tap(4) and host-only networking between host and guest Message-ID: <20151105120942.GC16353@admin.sibptus.tomsk.ru> References: <20151104075454.GA99850@admin.sibptus.tomsk.ru> <5639FF12.1020109@freebsd.org> <20151104131230.GA1117@admin.sibptus.tomsk.ru> <20151104184503.GC1117@admin.sibptus.tomsk.ru> <CAFgRE9EiXrqfMte7a36vkWtA7kisYTT=nBh5p8HNW9FaR69VSA@mail.gmail.com> <20151105023115.GA10139@admin.sibptus.tomsk.ru> <563B2DE4.2060307@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <563B2DE4.2060307@freebsd.org> Organization: OAO "Svyaztransneft", SibPTUS X-PGP-Key: http://www.dreamwidth.org/pubkey?user=victor_sudakov X-PGP-Fingerprint: 10E3 1171 1273 E007 C2E9 3532 0DA4 F259 9B5E C634 User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 05 Nov 2015 12:09:48 -0000 Julian Elischer wrote: [dd] > >>> For some reason, after a guest is shutdown or rebooted, the IP address > >>> on the host's tap0 interface is deleted. > >>> > >> Try using 'vmnet0' instead of 'tap0'. It will retain the IP address > >> even after a guest reboot/shutdown. > > Thanks, I see now vmnet is documented in tap(4). > I didn't know that... > Ok, so now you are an expert, can we count on you to answer all future > questions on this topic? ;-) > Oh, it's at http://victor-sudakov.dreamwidth.org/356635.html Sorry it's in Russian :-) -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:sudakov@sibptus.tomsk.ru
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?alpine.BSF.2.00.1511051052120.49057>