Date: Fri, 6 Apr 2018 00:19:16 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Ben RUBSON <ben.rubson@gmail.com>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Linux NFS client and FreeBSD server strangeness Message-ID: <YQBPR0101MB10422CD2250573C8908854D4DDBA0@YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <AF5B6AB4-8BAF-4BCF-BAD1-9D28A496DADD@gmail.com> References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net>, <AF5B6AB4-8BAF-4BCF-BAD1-9D28A496DADD@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Ben RUBSON wrote: >On 04 Apr 2018 20:27, Mike Tancsa wrote: >> Note, doing something like >> >> dd if=3D/dev/zero of=3D/backup/test.bin bs=3D4096 count=3D5000000 > >Note that this test may not be really relevant if you have ZFS compression >enabled. > >> I too am using 9000 for the MTU. > >Did you try to use smaller MTU ? >Some network adapters are known to have bugs requesting 9K mbufs for large >MTUs. >Especially Mellanox, not sure about Chelsio though. When the system uses a mix of mbuf cluster sizes (almost always happens whe= n you use jumbo packets), you can fragment the kernel memory pool they are allocated from to the point that jumbo clusters can't be allocated easi= ly. That breaks NFS performance badly. I had some patches that used jumbo mbuf clusters for the NFS read reply any write requests. They worked fine for a while, but would then get hammered by the fragmentat= ion problem. (As such, they never went into head, etc.) (The main advantage of using jumbo clusters was that the mbuf chain for an RPC had fewer mbufs in it and wouldn't get bit by bugs implementing TSO for interfaces that could only handle 32 buffers for a TSO segment. There is code that avoids this problem in tcp_output(), but it only works if the net device driver sets the parameters correctly.) Some have said that 9K jumbo clusters shouldn't exist in FreeBSD because of the fragmentation problem. Others proposed using separate pools for each mbuf cluster size, but nothing has happened as far as I know. rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQBPR0101MB10422CD2250573C8908854D4DDBA0>