From owner-freebsd-net@FreeBSD.ORG Thu Feb 26 09:21:55 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 97C62D35; Thu, 26 Feb 2015 09:21:55 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2078B63C; Thu, 26 Feb 2015 09:21:54 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t1Q9LmSe022578 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 26 Feb 2015 11:21:48 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t1Q9LmSe022578 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t1Q9LlRD022577; Thu, 26 Feb 2015 11:21:47 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 26 Feb 2015 11:21:47 +0200 From: Konstantin Belousov To: Rick Macklem Subject: Re: NFS: kernel modules (loading/unloading) and scheduling Message-ID: <20150226092147.GC2379@kib.kiev.ua> References: <201502250244.t1P2iu6N094346@hergotha.csail.mit.edu> <1930269117.485441.1424922935533.JavaMail.root@uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1930269117.485441.1424922935533.JavaMail.root@uoguelph.ca> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org, Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Feb 2015 09:21:55 -0000 On Wed, Feb 25, 2015 at 10:55:35PM -0500, Rick Macklem wrote: > Garrett Wollman wrote: > > In article > > <388835013.10159778.1424820357923.JavaMail.root@uoguelph.ca>, > > rmacklem@uoguelph.ca writes: > > > > >I tend to think that a bias towards doing Getattr/Lookup over > > >Read/Write > > >may help performance (the old "shortest job first" principal), I'm > > >not > > >sure you'll have a big enough queue of outstanding RPCs under normal > > >load > > >for this to make a real difference. > > > > I don't think this is a particularly relevant condition here. There > > are lots of ways RPCs can pile up where you really need to do better > > work-sharing than the current implementation does. One example is a > > client that issues lots of concurrent reads (e.g., a compute node > > running dozens of parallel jobs). Two such systems on gigabit NICs > > can easily issue large reads fast enough to cause 64 nfsd service > > threads to blocked while waiting for the socket send buffer to drain. > > Meanwhile, the file server is completely idle, but unable to respond > > to incoming requests, and the other users get angry. Rather than > > assigning new threads to requests from the slow clients, it would be > > better to let the requests sit until the send buffer drains, and > > process other clients' requests instead of letting the resources get > > monopolized by a single user. > > > > Lest you think this is purely hypothetical: we actually experienced > > this problem today, and I verified with "procstat -kk" that all of > > the > > nfsd threads were in fact blocked waiting for send buffer space to > > open up. I was able to restore service immediately by increasing the > > number of nfsd threads, but I'm unsure to what extent I can do this > > without breaking other things or hitting other bottlenecks.[1] So I > > have a user asking me why I haven't enable fair-share scheduling for > > NFS, and I'm going to have to tell him the answer is "no such thing". > > > > -GAWollman > > > > [1] What would the right number actually be? We could potentially > > have many thousands of threads in a compute cluster all operating > > simultaneously on the same filesystem, well within the I/O capacity > > of > > the server, and we'd really like to degrade gracefully rather than > > falling over when a single slow client soaks up all of the nfsd > > worker > > threads. > Well, each of these threads have two structures allocated to it. > 1 - The kthread info (sched_sizeof_thread() <-- struct thread + the scheduler info one) > 2 - A structure used by the krpc for each thread. > Since allocating two moderate sized structures isn't a lot of kernel > memory, I would think a server like yours would be fine with several > thousand nfsd threads. The biggest memory consumer for any thread, kernel or not, is the kernel thread stack. It consumes both physical memory and KVA, the later is not too sparce for amd64. > > What would be interesting would be the receive queue lengths for the > sockets for NFS client TCP connections when the server is running > normally. (This would be an indication of how many outstanding RPC > requests any scheduling effort would select between.) > I'll admit (given basic queuing theory) I would have expected these > receive queues to be small unless the server is overloaded. > > Oh, and I now realize my response related to your first idea > "Admission" was way off and didn't make much sense. Somehow, I > thought receive queue when you were talking about send queue. > (Basically, just ignore my response.) > However, given the different sizes of RPC replies, it might > be hard to come up with a reasonable high water mark for the > send queue. Also, the networking code would have to do some > sort of upcall to the krpc when the send queue shrinks. > (So, still not trivial to implement, I think?) > > I do agree with Alfred, in that I think you are experiencing > nfsd thread starvation and increasing the number of nfsd threads > a lot is the simple way to resolve this. This also increases indirect scheduler costs. Direct management of the runqueues costs proportionally to the number of runnable threads, but some rare operations have to account all threads.