Date: Thu, 1 Nov 2018 00:27:16 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Andrew Vylegzhanin <avv314@gmail.com> Cc: "Rodney W. Grimes" <freebsd-rwg@pdx.rh.CN85.dnsmgr.net>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "freebsd-infiniband@freebsd.org" <freebsd-infiniband@freebsd.org> Subject: Re: NFS + Infiniband problem Message-ID: <YTOPR0101MB11622A98B2358A0A83CD6927DDCE0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CA%2BBi_YgpzBYyho_ex4p9nx0HHvvRb5kncKapUBEJQk88Et3BCQ@mail.gmail.com> References: <CA%2BBi_YiHoxFc3wsEPnMeBHWgW-nh6sXQCEgBTb=-nD6-XcjZ%2Bg@mail.gmail.com> <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> <YTOPR0101MB11622A9797376128D2FA182ADDF30@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM> <CA%2BBi_YhHvpqkmP7WVd4wJj0u9rnbMZpum7FsKxZsu=%2B=8ze8-w@mail.gmail.com> <YTOPR0101MB116206BEB6259AA2836B16BCDDCD0@YTOPR0101MB1162.CANPRD01.PROD.OUTLOOK.COM>, <CA%2BBi_YgpzBYyho_ex4p9nx0HHvvRb5kncKapUBEJQk88Et3BCQ@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
Andrew Vylegzhanin wrote: [stuff snipped] >With this TCP settings same server serve NFS requests via 40G Ethernet on multiply >clients with speed via 1G Eth ~ 105/110 MB/sec write/read. >Of course I'll try to change congestion algorithm, but I don't think that will help. Yes, I doubt changing the congestion algorithm will make much difference. >Also need to test setup with infniband set from connected mode to datagram mode. Are you using IPv6 by any chance? Why I ask is that there was a problem with IPv6 fragmentation re-assembly. If InfiniBand is using a larger MTU than the ethernet, the switch would probably fragment the IP datagram. If any fragment is lost (or fragmentation re-assembly is broken), it isn't going to work well. "netstat -s | fgrep "fragments dropped after" will show you the count of fragments dropped after timeout. If this value is increasing, then fragmentation re-assembly is an issue. (Check on the receiving end. The server for writes.) You might want to post on freebsd-net@, since someone there might know more about InfiniBand (and someone over there will definitely know about the IPv6 fragmentation problem. rickhome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB11622A98B2358A0A83CD6927DDCE0>