From owner-freebsd-net@FreeBSD.ORG Thu Apr 7 19:51:19 2005 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 73AF816A4CE for ; Thu, 7 Apr 2005 19:51:19 +0000 (GMT) Received: from mta1.lbl.gov (mta1.lbl.gov [128.3.41.24]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4575B43D2F for ; Thu, 7 Apr 2005 19:51:19 +0000 (GMT) (envelope-from dart@nersc.gov) Received: from mta1.lbl.gov (localhost [127.0.0.1]) by mta1.lbl.gov (8.12.10/8.12.10) with ESMTP id j37JpGn3025588 for ; Thu, 7 Apr 2005 12:51:17 -0700 (PDT) Received: from [127.0.0.1] (dhcp163-8.nersc.gov [128.55.8.163]) by mta1.lbl.gov (8.12.10/8.12.10) with ESMTP id j37JpGP9025585; Thu, 7 Apr 2005 12:51:16 -0700 (PDT) Message-ID: <42558F33.5090109@nersc.gov> Date: Thu, 07 Apr 2005 12:51:15 -0700 From: Eli Dart Organization: NERSC Center, LBNL User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: freebsd-net@freebsd.org X-Enigmail-Version: 0.90.1.1 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Disable NFS client cache (or other traffic reduction methods) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: dart@nersc.gov List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Apr 2005 19:51:19 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi List, We've just deployed a new syslog server infrastructure consisting of two hosts -- one NFS server that has a large disk array attached via fibre channel and one NFS client that receives syslog messages, mounts the server's disk, and writes to the NFS exported filesystem. Statistics grinding is done on the server so as to avoid impacting collection of syslog messages. The NFS server and client are connected to each other via a private back-to-back GigE. The problem is that NFS is causing a 10x to 15x packet amplification on the link that carries the NFS traffic. By this I mean that for each syslog message received by the collector, there are somewhere between 10 and 15 (sometimes less, sometimes more) packets on the private NFS link. About half of these are full-MTU packets. When the client receives a syslog message, the sequence of operations on the NFS link appears to be: client -> server: write server -> client: ok client -> server: commit server -> client: ok client -> server: access server -> client: ok client -> server: read server -> client: ok + 4k-16k worth of data Looking at the packet contents, it appears to be fetching back the last few blocks of the log file. My guess is that this is the client keeping its NFS cache fresh. The client is never, ever, ever going to read that file (or any file on that filesystem) other than the bare minimum required to open it for writing and rotate the log files. Is there a way to disable client-side caching? I've looked, and can't seem to find one. Or, do I have this wrong and there is something else that is causing this? Any insight would be very much appreciated.... Thanks! --eli -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCVY8yLTFEeF+CsrMRApq9AJ4jK24kcKejo14/epibZX14IUeIngCfRMJQ HZ+pOKtoyLdpzWUGtrKG6RA= =t4Oo -----END PGP SIGNATURE-----