From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 20 18:58:10 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E1DECDC; Sat, 20 Oct 2012 18:58:10 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id C5CC28FC0A; Sat, 20 Oct 2012 18:58:09 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id 16so1196254wgi.31 for ; Sat, 20 Oct 2012 11:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=Hx/e2v4Q3S5kLxIH99nIJbATi8H1lEf/1RUNEZ222qQ=; b=NEjvDy2+6+IO0RzvPcpzx03w7NGsTuI9ph0+wtlA8tevitJ6aCDNy0fZRurIldtxHk y2TKZWFgAjxUjxk2LN7rTob0EB0+provIFCrtJkLijGjEm9gLnJqy0gwa3wVeoun3+g4 rDYWF4EPVXeGPbOgcEY9QO7CdtpPonp3hmFrCdaG+wXHU8djJ1XqUXxL6b5Uw4UOpdQZ C5J58f+Q0XY4sABI+8zqE1E7/mfK8ESvNzdjvk2jmw//7jmMhi4V2jpU95MZZU1DHwBt ZVh6AfOnKLOgYcVl2ycCaM2zJURWZvV7Q4tlcxe4qGyEY5IjKcLcHAMCbEOMQ2b56hCu 7uBw== Received: by 10.180.106.9 with SMTP id gq9mr10810119wib.12.1350759488552; Sat, 20 Oct 2012 11:58:08 -0700 (PDT) Received: from [10.0.0.86] ([93.152.184.10]) by mx.google.com with ESMTPS id w8sm41666214wif.4.2012.10.20.11.58.05 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 20 Oct 2012 11:58:07 -0700 (PDT) Subject: Re: NFS server bottlenecks Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: Date: Sat, 20 Oct 2012 21:58:03 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: References: <937460294.2185822.1350093954059.JavaMail.root@erie.cs.uoguelph.ca> <302BF685-4B9D-49C8-8000-8D0F6540C8F7@gmail.com> <0857D79A-6276-433F-9603-D52125CF190F@gmail.com> <6DAAB1E6-4AC7-4B08-8CAD-0D8584D039DE@gmail.com> <23D7CB3A-BD66-427E-A7F5-6C9D3890EE1B@gmail.com> To: "freebsd-hackers@freebsd.org Hackers" X-Mailer: Apple Mail (2.1498) Cc: Rick Macklem , Ivan Voras X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 18:58:10 -0000 On Oct 20, 2012, at 4:00 PM, Nikolay Denev wrote: >=20 > On Oct 20, 2012, at 3:11 PM, Ivan Voras wrote: >=20 >> On 20 October 2012 13:42, Nikolay Denev wrote: >>=20 >>> Here are the results from testing both patches : = http://home.totalterror.net/freebsd/nfstest/results.html >>> Both tests ran for about 14 hours ( a bit too much, but I wanted to = compare different zfs recordsize settings ), >>> and were done first after a fresh reboot. >>> The only noticeable difference seems to be much more context = switches with Ivan's patch. >>=20 >> Thank you very much for your extensive testing! >>=20 >> I don't know how to interpret the rise in context switches; as this = is >> kernel code, I'd expect no context switches. I hope someone else can >> explain. >>=20 >> But, you have also shown that my patch doesn't do any better than >> Rick's even on a fairly large configuration, so I don't think there's >> value in adding the extra complexity, and Rick knows NFS much better >> than I do. >>=20 >> But there are a few things other than that I'm interested in: like = why >> does your load average spike almost to 20-ties, and how come that = with >> 24 drives in RAID-10 you only push through 600 MBit/s through the 10 >> GBit/s Ethernet. Have you tested your drive setup locally (AESNI >> shouldn't be a bottleneck, you should be able to encrypt well into >> Gbyte/s range) and the network? >>=20 >> If you have the time, could you repeat the tests but with a recent >> Samba server and a CIFS mount on the client side? This is probably = not >> important, but I'm just curious of how would it perform on your >> machine. >=20 > The first iozone local run finished, I'll paste just the result here, = and also the same test over NFS for comparison: > (This is iozone doing 8k sized IO ops, on ZFS dataset with = recordsize=3D8k) >=20 > NFS: > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read =20 > 33554432 8 4973 5522 2930 2906 2908 = 3886 =20 >=20 > Local: > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read =20 > 33554432 8 34740 41390 135442 142534 24992 = 12493 =20 >=20 >=20 > P.S.: I forgot to mention that the network is with 9K mtu. Here are the full results of the test on the local fs : http://home.totalterror.net/freebsd/nfstest/local_fs/ I'm now running the same test on NFS mount over the loopback interface = on the NFS server machine.