From owner-freebsd-infiniband@freebsd.org Mon Oct 29 15:06:38 2018 Return-Path: Delivered-To: freebsd-infiniband@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1308D10DCBC0; Mon, 29 Oct 2018 15:06:38 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5E4F474B2F; Mon, 29 Oct 2018 15:06:37 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w9TF6Yve057203; Mon, 29 Oct 2018 08:06:34 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w9TF6YAP057202; Mon, 29 Oct 2018 08:06:34 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201810291506.w9TF6YAP057202@pdx.rh.CN85.dnsmgr.net> Subject: Re: NFS + Infiniband problem In-Reply-To: To: Andrew Vylegzhanin Date: Mon, 29 Oct 2018 08:06:34 -0700 (PDT) CC: freebsd-infiniband@freebsd.org, freebsd-fs@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 15:06:38 -0000 > Hello everyone, > > I have a several FreeBSD machines connected via Infiniband netwok ( FDR > switch Mellanox SW3036 + ConnectX-3 VPI cards ). > One of them is a NAS-server with multiply ZFS pools. > > All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on > server) are with infiniband connected mode (option IPOIB_CM, option SDM) > and world with OFED stack support. (WITH_OFED='yes'). > > File transfers via FTP or SSH between server and clients works almost > flawless ( ~ 12 Gbit/s ). > > But when I try to copy in/out some significant data via NFS share mounted > on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) > transfer speed after uncertain amount of copied data. For example, on the > one node I can copy 1GB file, and after NFS hang on file with size 30 kb. > > Some details: > [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt ^^^^^^^^^^^^ I am not sure what the interaction between page sizes, TSO needs, buffer needs and all that are but I always use a power of 2 wsize and rsize. You might try that. And as Rick suggested, turn of TSO, if you can. Is infiniband using RDMA to do this, if so then the page size stuff is probably very important, use multiples of 4096 only. > [root@node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024 > > Ctrl-T for "hang" dd > load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k > load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k > > for "slow" dd > load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k > > load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k > > I've tried mount with different wsize option with same result. > > Any help would be greatly appreciated. > > -- > Andrew > _______________________________________________ > freebsd-infiniband@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-infiniband > To unsubscribe, send any mail to "freebsd-infiniband-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org