From owner-freebsd-hackers@FreeBSD.ORG Sat Oct 20 12:11:57 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 906176FF for ; Sat, 20 Oct 2012 12:11:57 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3BB248FC12 for ; Sat, 20 Oct 2012 12:11:57 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id p27so648224qat.13 for ; Sat, 20 Oct 2012 05:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=xImakIC7JV2zly/Y0OKoJJDqcQE9Q+/z4zufdJBv3uw=; b=GESJwgsCwPgkl8atHyIDbENcpbKKqfayG2jHSiuYK95Wfy7+uyuejr3G6W8dfL1OkZ Vz7tiIF/fMcHqtZiLjXqAQgesFctaWEhSI9DhaNGz+Oubj7cZqB+AT4pXh/uyIt8vHvJ Kx+VdiHFQ40N6qnsuZgk17bQIxgA/HP52b6cI+tewv8o4MeI/ZORpLmV8u8vZxk7mS6n 79y/kEfxMRhK6jipij1IHkRzJlDc2HQw5XSy2NYJ8XVRlDVJC4K0J1q3qIqatL4ir7Cr 2D/EfOEhlZFnE6ykipFQ7fs1QhyYNNRFNTfxeAZOOzNFMe0L/Nf4rSshzb4RFhcQCoLs u7Dw== Received: by 10.224.178.4 with SMTP id bk4mr1928123qab.38.1350735110677; Sat, 20 Oct 2012 05:11:50 -0700 (PDT) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.49.82.231 with HTTP; Sat, 20 Oct 2012 05:11:10 -0700 (PDT) In-Reply-To: <23D7CB3A-BD66-427E-A7F5-6C9D3890EE1B@gmail.com> References: <937460294.2185822.1350093954059.JavaMail.root@erie.cs.uoguelph.ca> <302BF685-4B9D-49C8-8000-8D0F6540C8F7@gmail.com> <0857D79A-6276-433F-9603-D52125CF190F@gmail.com> <6DAAB1E6-4AC7-4B08-8CAD-0D8584D039DE@gmail.com> <23D7CB3A-BD66-427E-A7F5-6C9D3890EE1B@gmail.com> From: Ivan Voras Date: Sat, 20 Oct 2012 14:11:10 +0200 X-Google-Sender-Auth: PsZqbdk8UhFAMP1_lmPtiJzhIQk Message-ID: Subject: Re: NFS server bottlenecks To: Nikolay Denev Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org Hackers" , Rick Macklem X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 12:11:57 -0000 On 20 October 2012 13:42, Nikolay Denev wrote: > Here are the results from testing both patches : http://home.totalterror.net/freebsd/nfstest/results.html > Both tests ran for about 14 hours ( a bit too much, but I wanted to compare different zfs recordsize settings ), > and were done first after a fresh reboot. > The only noticeable difference seems to be much more context switches with Ivan's patch. Thank you very much for your extensive testing! I don't know how to interpret the rise in context switches; as this is kernel code, I'd expect no context switches. I hope someone else can explain. But, you have also shown that my patch doesn't do any better than Rick's even on a fairly large configuration, so I don't think there's value in adding the extra complexity, and Rick knows NFS much better than I do. But there are a few things other than that I'm interested in: like why does your load average spike almost to 20-ties, and how come that with 24 drives in RAID-10 you only push through 600 MBit/s through the 10 GBit/s Ethernet. Have you tested your drive setup locally (AESNI shouldn't be a bottleneck, you should be able to encrypt well into Gbyte/s range) and the network? If you have the time, could you repeat the tests but with a recent Samba server and a CIFS mount on the client side? This is probably not important, but I'm just curious of how would it perform on your machine.