Date: Wed, 25 Feb 2015 13:12:58 +0000 From: Tim Borgeaud <timothy.borgeaud@framestore.com> To: Garrett Wollman <wollman@hergotha.csail.mit.edu> Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org Subject: Re: NFS: kernel modules (loading/unloading) and scheduling Message-ID: <CADqOPxun6e0=bnYfH8cyHW89BPwT2xazOnOb7GSLdPvsciiNVg@mail.gmail.com> In-Reply-To: <201502250244.t1P2iu6N094346@hergotha.csail.mit.edu> References: <CADqOPxsJ4Sjt_u6%2Bh5B8sFWFzOHQA28E69H0LnxxZg1UPeup7g@mail.gmail.com> <388835013.10159778.1424820357923.JavaMail.root@uoguelph.ca> <201502250244.t1P2iu6N094346@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Rick, Garret & others, Thanks for the replies and useful info. I may take a look at enabling some kernel module reloading, but, if the usual approach is rebooting, I expect that this won't be an issue. Regarding the NFS functionality itself, I can give a bit more of an overview of our (Framestore) NFS use and what NFS server functionality we are considering. We have NFS storage systems accessed by both an HPC cluster and also multiple desktop users. Like many organizations, we want to have, and get the most from, high performance NFS storage, in terms of both total IO and low latency. One of the demands that we face is to allow users, as nodes of the cluster or more interactive desktops, to be provided with a reasonable level of service from heavily loaded systems. We do want to allow "users" to apply high load, but would like to avoid such requests from starving others. We are considering scheduling of NFS RPCs such that both client transports (xprt) and users themselves are given a fair share, including where a single user is using multiple transports. At the same time, we don't want hurt performance by loss of file handle affinity or similar (though it may be quite nice to remove responsibility from the RPC/NFS layer). We've already prototyped a 'fair' schedule when servicing the RPC requests, which appears to be an improvement. But, as Garrett pointed out, we may have moved onto bottlenecks in sending replies and, possibly, reading requests. It may be that, with such cases as slow clients, overall performance could also be improved. -- Tim Borgeaud Systems Developer On 25 February 2015 at 02:44, Garrett Wollman < wollman@hergotha.csail.mit.edu> wrote: > In article > <388835013.10159778.1424820357923.JavaMail.root@uoguelph.ca>, > rmacklem@uoguelph.ca writes: > > >I tend to think that a bias towards doing Getattr/Lookup over Read/Write > >may help performance (the old "shortest job first" principal), I'm not > >sure you'll have a big enough queue of outstanding RPCs under normal load > >for this to make a real difference. > > I don't think this is a particularly relevant condition here. There > are lots of ways RPCs can pile up where you really need to do better > work-sharing than the current implementation does. One example is a > client that issues lots of concurrent reads (e.g., a compute node > running dozens of parallel jobs). Two such systems on gigabit NICs > can easily issue large reads fast enough to cause 64 nfsd service > threads to blocked while waiting for the socket send buffer to drain. > Meanwhile, the file server is completely idle, but unable to respond > to incoming requests, and the other users get angry. Rather than > assigning new threads to requests from the slow clients, it would be > better to let the requests sit until the send buffer drains, and > process other clients' requests instead of letting the resources get > monopolized by a single user. > > Lest you think this is purely hypothetical: we actually experienced > this problem today, and I verified with "procstat -kk" that all of the > nfsd threads were in fact blocked waiting for send buffer space to > open up. I was able to restore service immediately by increasing the > number of nfsd threads, but I'm unsure to what extent I can do this > without breaking other things or hitting other bottlenecks.[1] So I > have a user asking me why I haven't enable fair-share scheduling for > NFS, and I'm going to have to tell him the answer is "no such thing". > > -GAWollman > > [1] What would the right number actually be? We could potentially > have many thousands of threads in a compute cluster all operating > simultaneously on the same filesystem, well within the I/O capacity of > the server, and we'd really like to degrade gracefully rather than > falling over when a single slow client soaks up all of the nfsd worker > threads. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADqOPxun6e0=bnYfH8cyHW89BPwT2xazOnOb7GSLdPvsciiNVg>