From owner-freebsd-net@freebsd.org Thu Jun 25 23:56:13 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8256798C90D for ; Thu, 25 Jun 2015 23:56:13 +0000 (UTC) (envelope-from stl@wiredrive.com) Received: from mail-wi0-f199.google.com (mail-wi0-f199.google.com [209.85.212.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 105101BCD for ; Thu, 25 Jun 2015 23:56:12 +0000 (UTC) (envelope-from stl@wiredrive.com) Received: by wiea6 with SMTP id a6so1018867wie.1 for ; Thu, 25 Jun 2015 16:56:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=1uk5/rCw7HRK+0r9sJrYVKDCRoZMDg7htS1EwnW8pus=; b=PsObtB6QdCSPV4mjtUT84FxYsLD3S3VouIPnyHMyH9hZ2H2V62Eg4ihWu/h2E3xdYo 5BTvOfz3H2wxfWXvHoYncd/6232R3CC4W5eStOcC+2D5r70/AaNQG5EfkwK9CGcpdxDJ N/plFj4IrOfEk+QBpN0JYa3zfbCqNargHZ/Cyz0UuuUG0bJL8MUmHxDTRoEjBeTGL9Zu 1CKp/t6EFY/uDVv4p9F7ulmfWzyDPrbw4IUlzn8HIWP71Aba2ycq25qgHyRluwm5WDbl T/i5m7djGpBLi+5xKXdA0SFjvh+DSyrVX4AQ7uPPb5DgYVvY62gl4tdWJZi2+SRtBoNz pBUw== X-Gm-Message-State: ALoCoQnhfX7o7DtY6/P4DFMFkMskjEHHZz5Xwqxoe6Yy1ymmYWy9Jl1wQgFiOgbSipDmaqeGlOej MIME-Version: 1.0 X-Received: by 10.112.204.6 with SMTP id ku6mr1050669lbc.73.1435262196725; Thu, 25 Jun 2015 12:56:36 -0700 (PDT) Received: by 10.25.205.149 with HTTP; Thu, 25 Jun 2015 12:56:36 -0700 (PDT) In-Reply-To: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> References: <20150625145238.12cf9da3b368ef0b9a30f193@aei.mpg.de> Date: Thu, 25 Jun 2015 12:56:36 -0700 Message-ID: Subject: Re: NFS on 10G interface terribly slow From: Scott Larson To: =?UTF-8?B?R2Vycml0IEvDvGhu?= Cc: freebsd-net@freebsd.org, "carsten.aulbert@aei.mpg.de" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jun 2015 23:56:13 -0000 We've got 10.0 and 10.1 servers accessing Isilon and Nexenta via NFS with Intel 10G gear and bursting to near wire speed with the stock MTU/rsize/wsize works as expected. TSO definitely needs to be enabled for that performance. The fact iperf gives you the expected throughput but NFS does not would have me looking at tuning for the NFS platform. Other things to look at: Are all the servers involved negotiating the correct speed and duplex, with TSO? Does it need to have the network stack tuned with whatever it's equivalent of maxsockbuf and send/recvbuf are? Do the switch ports and NIC counters show any drops or errors? On the FBSD servers you could also run 'netstat -i -w 1' under load to see if drops are occurring locally, or 'systat -vmstat' for resource contention problems. But again, a similar setup here and no such issues have appeared. *[image: userimage]Scott Larson[image: los angeles] Lead Systems Administrator[image: wdlogo] [image: linkedin] [image: facebook] [image: twitter] [image: instagram] T 310 823 8238 x1106 <310%20823%208238%20x1106> | M 310 904 8818 <310%20904%208818>* On Thu, Jun 25, 2015 at 5:52 AM, Gerrit K=C3=BChn wrote: > Hi all, > > We have a recent FreeBSD 10.1 installation here that is supposed to act a= s > nfs (v3) client to an Oracle x4-2l server running Soalris 11.2. > We have Intel 10-Gigabit X540-AT2 NICs on both ends, iperf is showing > plenty of bandwidth (9.xGB/s) in both directions. > However, nfs appears to be terribly slow, especially for writing: > > root@crest:~ # dd if=3D/dev/zero of=3D/net/hellpool/Z bs=3D1024k count=3D= 1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 20.263190 secs (51747824 bytes/sec) > > > Reading appears to be faster, but still far away from full bandwidth: > > root@crest:~ # dd of=3D/dev/null if=3D/net/hellpool/Z bs=3D1024k > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 5.129869 secs (204406000 bytes/sec) > > > We have already tried to tune rsize/wsize parameters, but they appear to > have little (if any) impact on these results. Also, neither stripping dow= n > rxsum, txsum, tso etc. from the interface nor increasing MTU to 9000 for > jumbo frames did improve anything. > It is quite embarrassing to achieve way less than 1GBE performance with > 10GBE equipment. Are there any hints what else might be causing this (and > how to fix it)? > > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >