Date: Fri, 6 Apr 2018 01:38:22 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Mike Tancsa <mike@sentex.net>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Linux NFS client and FreeBSD server strangeness Message-ID: <YQBPR0101MB104246DC9F08DD13713F7F83DDBA0@YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <9040d0fa-f9c2-2cc3-efbd-f96408cff73b@sentex.net> References: <369fab06-6213-ba87-cc66-c9829e8a76a0@sentex.net> <YQBPR0101MB104297BE1296B72597086A3FDDBB0@YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM>, <9040d0fa-f9c2-2cc3-efbd-f96408cff73b@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Mike Tancsa wrote: >Thank you for all the feedback, pointers/insights. Coming directly from >'Mr. NFS', its particularly appreciated :) I could replace "Mr. NFS" with "Mr. stupid enough to do NFS without getting paid to do it";-) However, I should note that, although I am fairly familiar with the protoco= l and the FreeBSD code, I don't have a lot of experience w.r.t. performance, at l= east on newer hardware and fast networking. [good stuff snipped] >I think the root of the issue partially stems from the client having a >LOT of RAM. So according to this default behaviour There is a now rather ancient connectathon test suite for NFS, where one of the tests is writing/reading a large 10Mbyte file. The 10Mbyte size was selected because it was guaranteed to exceed the NFS client's buffer cache capacity. (Maybe no longer true;-) >---------------- > The NFS client treats the sync mount option differently than some >other file systems (refer to mount(8) for a description of the > generic sync and async mount options). If neither sync nor >async is specified (or if the async option is specified), the NFS > client delays sending application writes to the server until any >of these events occur: > > Memory pressure forces reclamation of system memory resource= s. > > An application flushes file data explicitly with sync(2), >msync(2), or fsync(3). > > An application closes a file with close(2). > > The file is locked/unlocked via fcntl(2). > > In other words, under normal circumstances, data written by an >application may not immediately appear on the server that hosts > the file. >----------------------------- Just fyi, the FreeBSD client starts a write when the buffer cache block is completely written with new data. (Called B_ASYNC in the code.) If only part of a block has been written with new data, the write is delaye= d until it is fully written with new data or one of the above cases applies. You might want to slap to-gether a test program that loops on write(2) for a while, does an fsync(2), then more writing... and see how that performs on both FreeBSD and Linux clients. I do find the fact that doing an "ls" concurrently with the writes makes th= ings work better interesting/weird. All the "ls" will do is inject a bunch of ot= her RPC messages into the TCP stream (small ones in the client->server dire= ction). The only thing I can think of is that the net interface is somehow "awakene= d" by the small RPC messages (each one almost always in one TCP segment). (Maybe something related to how the net interface device driver handles receive interrupts or ???) If you find the "magic bullet" that makes the Linux case work well without the concurrent "ls", please post and let us know what it is. Good luck with it, rick [lots of stuff snipped]=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQBPR0101MB104246DC9F08DD13713F7F83DDBA0>