From owner-freebsd-fs@FreeBSD.ORG Tue Oct 11 07:22:25 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8941316A41F for ; Tue, 11 Oct 2005 07:22:25 +0000 (GMT) (envelope-from mark233@world-net.co.nz) Received: from smtp.world-net.co.nz (smtp.world-net.co.nz [203.96.119.35]) by mx1.FreeBSD.org (Postfix) with SMTP id 216E943D49 for ; Tue, 11 Oct 2005 07:22:24 +0000 (GMT) (envelope-from mark233@world-net.co.nz) Received: (qmail 62945 invoked from network); 11 Oct 2005 20:22:25 +1300 Received: from 202-37-167-46.dialup.world-net.co.nz (HELO ?202.37.167.46?) (202.37.167.46) by smtp.world-net.co.nz with SMTP; 11 Oct 2005 20:22:25 +1300 Message-ID: <434B683B.90502@world-net.co.nz> Date: Tue, 11 Oct 2005 20:22:35 +1300 From: mark233 User-Agent: Thunderbird 1.4 (Windows/20050908) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: OnlinePharmacy Cheap X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Oct 2005 07:22:25 -0000 hi, I suffer huge pain and am sick of traveling long distance to the doctor so would like to purchase nitrazapam and methadone tablets if posable. I can send my doctors notes and photos if need be. Even just the nitrazapam would be a big help. awaiting your reply, from Kieran. This is my brothers email so put attn. Kieran. cheers From owner-freebsd-fs@FreeBSD.ORG Wed Oct 12 06:09:05 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id 94B9716A41F for ; Wed, 12 Oct 2005 06:09:05 +0000 (GMT) (envelope-from owner-freebsd-net@freebsd.org) From: owner-freebsd-net@freebsd.org To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1705631082==" Message-ID: Date: Tue, 11 Oct 2005 23:09:04 -0700 Precedence: bulk X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 X-List-Administrivia: yes Sender: owner-freebsd-net@freebsd.org Errors-To: owner-freebsd-net@freebsd.org X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: The results of your email commands X-BeenThere: freebsd-fs@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2005 06:09:05 -0000 --===============1705631082== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit The results of your email command are provided below. Attached is your original message. - Results: Ignoring non-text/plain MIME parts freebsd-fs@freebsd.org is not a member of the freebsd-net mailing list - Unprocessed: hi - Done. --===============1705631082== Content-Type: message/rfc822 MIME-Version: 1.0 Return-Path: X-Original-To: freebsd-net-unsubscribe@freebsd.org Delivered-To: freebsd-net-unsubscribe@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAEC116A41F for ; Wed, 12 Oct 2005 06:09:03 +0000 (GMT) (envelope-from freebsd-fs@freebsd.org) Received: from dialin-213-170-163-019.ewetel.net (dialin-213-170-163-019.ewetel.net [213.170.163.19]) by mx1.FreeBSD.org (Postfix) with SMTP id DDD8B43D45 for ; Wed, 12 Oct 2005 06:09:02 +0000 (GMT) (envelope-from freebsd-fs@freebsd.org) Received: from freebsd.org (90.3.130.160) by ndm3-o2.freebsd.org with Microsoft SMTPSVC(28245.59634.4.4); Wed, 12 Oct 2005 08:09:12 +0100 Received: from freebsd.org (freebsd.org 199.79.216.21) by freebsd.org (8.12.10/8.12.9) with ESMTP id bxb30902DDVZRB7 for ; Wed, 12 Oct 2005 08:09:12 +0100 (envelope-from freebsd-fs@freebsd.org) Received: from O12 (modemcable2.5-23118.ib.freebsd.org 181.228.238.70) (authenticated bits=58151) by freebsd.org (8.12.10/8.12.9) with ESMTP id y3R55021bf2 for ; Wed, 12 Oct 2005 08:09:12 +0100 (envelope-from freebsd-fs@freebsd.org) Message-ID: <3f3pm3$ywg46077x3r2$3vs13731b36906@freebsd.org> From: "Terry" To: "Antonio" Subject: hi Content-Transfer-Encoding: 7bit Date: Wed, 12 Oct 2005 06:09:02 +0000 (GMT) MIME-Version: 1.0 Content-Type: text/plain; Over a million men have been helped with the potent ingredients in Pen is Growth Patch men have experienced bigger size, more action, and super-satisfying results for themselves and their partners. Don't be left behind! Take advantage of price specials going on now. [1]Click here! [2]Remove you e-mail References 1. http://www.qerupu.com/pt/?42 2. http://www.qerupu.com/un.php --===============1705631082==-- From owner-freebsd-fs@FreeBSD.ORG Wed Oct 12 09:00:27 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D070616A420 for ; Wed, 12 Oct 2005 09:00:27 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from itchy.rabson.org (mailgate.nlsystems.com [80.177.232.242]) by mx1.FreeBSD.org (Postfix) with ESMTP id 45E5743D5D for ; Wed, 12 Oct 2005 09:00:21 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from herring.rabson.org (herring [10.0.0.2]) by itchy.rabson.org (8.13.3/8.13.3) with ESMTP id j9C8xtkr037480; Wed, 12 Oct 2005 09:59:55 +0100 (BST) (envelope-from dfr@nlsystems.com) From: Doug Rabson To: freebsd-fs@freebsd.org, cel@citi.umich.edu Date: Wed, 12 Oct 2005 09:59:53 +0100 User-Agent: KMail/1.8.2 References: <4342F29D.1080302@citi.umich.edu> In-Reply-To: <4342F29D.1080302@citi.umich.edu> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-6" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200510120959.54281.dfr@nlsystems.com> X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (itchy.rabson.org [80.177.232.242]); Wed, 12 Oct 2005 09:59:56 +0100 (BST) X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on itchy.rabson.org X-Virus-Scanned: ClamAV 0.83/1128/Tue Oct 11 02:30:06 2005 on itchy.rabson.org X-Virus-Status: Clean Cc: Ivan Synyeokov Subject: Re: nfs4 on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2005 09:00:28 -0000 On Tuesday 04 October 2005 22:22, Chuck Lever wrote: > Ivan Synyeokov wrote: > > Hi, > > I'm playing with nfs4 under FreeBSD and I wonder is there any > > particular plans to improve its support and have fully functional > > nfs4 client and server in base or maybe ports? > > > > I've found that Rick's (ftp://ftp.cis.uoguelph.ca/pub/nfsv4/) work > > is quite stable (I mean server part) on FreeBSD 6.0 Beta5, but > > client is only available for OpenBSD. Rick, do you plan to port > > your client to FreeBSD also? > > On the other hand, lack of delegation and locking state features > > from native FreeBSD6 client, prevent me from putting nfs4 in > > production. So, I kindly regard if anybody clear the situation, and > > in any case I could provide some testing. > > hi ivan- > > i'm beginning to look at the FreeBSD NFS client to think about how to > finish the nascent NFSv4 and RPCGSS implementation. it's a slow > start though. I have a working userland RPCSEC_GSS implementation which follows the Solaris api for gss-specific extensions to the RPC api (e.g. rpc_gcc_get_mechanisms(), rpc_gss_set_svc_name(), rpc_gss_getcred() etc.). I also have a working GSSAPI redirector which allows the use of multiple GSSAPI mechanisms, such as SPKEY which is required by the spec. It even includes an almost complete set of manpages culled from the RFC. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 12 14:36:59 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8B3D716A42B for ; Wed, 12 Oct 2005 14:36:59 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4304943D45 for ; Wed, 12 Oct 2005 14:36:59 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id D000346B9F; Wed, 12 Oct 2005 10:36:58 -0400 (EDT) Date: Wed, 12 Oct 2005 15:36:58 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Doug Rabson In-Reply-To: <200510120959.54281.dfr@nlsystems.com> Message-ID: <20051012153618.P48006@fledge.watson.org> References: <4342F29D.1080302@citi.umich.edu> <200510120959.54281.dfr@nlsystems.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org, Ivan Synyeokov Subject: Re: nfs4 on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2005 14:36:59 -0000 On Wed, 12 Oct 2005, Doug Rabson wrote: >> i'm beginning to look at the FreeBSD NFS client to think about how to >> finish the nascent NFSv4 and RPCGSS implementation. it's a slow start >> though. > > I have a working userland RPCSEC_GSS implementation which follows the > Solaris api for gss-specific extensions to the RPC api (e.g. > rpc_gcc_get_mechanisms(), rpc_gss_set_svc_name(), rpc_gss_getcred() > etc.). > > I also have a working GSSAPI redirector which allows the use of multiple > GSSAPI mechanisms, such as SPKEY which is required by the spec. It even > includes an almost complete set of manpages culled from the RFC. Are any of you planning at being at EuroBSDCon? What with a FreeBSD developer summit happening about the same time, it would make sense to get NFSv4-interested people in one place. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Wed Oct 12 14:44:40 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9361016A41F; Wed, 12 Oct 2005 14:44:40 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from mail.qubesoft.com (gate.qubesoft.com [217.169.36.34]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0575C43D4C; Wed, 12 Oct 2005 14:44:39 +0000 (GMT) (envelope-from dfr@nlsystems.com) Received: from [192.168.1.254] (dhcp254.qubesoft.com [192.168.1.254]) by mail.qubesoft.com (8.13.3/8.13.3) with ESMTP id j9CEi9Nk097893; Wed, 12 Oct 2005 15:44:09 +0100 (BST) (envelope-from dfr@nlsystems.com) In-Reply-To: <20051012153618.P48006@fledge.watson.org> References: <4342F29D.1080302@citi.umich.edu> <200510120959.54281.dfr@nlsystems.com> <20051012153618.P48006@fledge.watson.org> Mime-Version: 1.0 (Apple Message framework v734) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <764B617E-AB5A-4260-8597-E8054E93757D@nlsystems.com> Content-Transfer-Encoding: 7bit From: Doug Rabson Date: Wed, 12 Oct 2005 15:44:07 +0100 To: Robert Watson X-Mailer: Apple Mail (2.734) X-Spam-Status: No, score=-2.8 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.0.4 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.qubesoft.com X-Virus-Scanned: ClamAV 0.86.2/1130/Wed Oct 12 13:34:00 2005 on mail.qubesoft.com X-Virus-Status: Clean Cc: freebsd-fs@FreeBSD.org, Ivan Synyeokov Subject: Re: nfs4 on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Oct 2005 14:44:40 -0000 On 12 Oct 2005, at 15:36, Robert Watson wrote: > > On Wed, 12 Oct 2005, Doug Rabson wrote: > > >>> i'm beginning to look at the FreeBSD NFS client to think about >>> how to finish the nascent NFSv4 and RPCGSS implementation. it's >>> a slow start though. >>> >> >> I have a working userland RPCSEC_GSS implementation which follows >> the Solaris api for gss-specific extensions to the RPC api (e.g. >> rpc_gcc_get_mechanisms(), rpc_gss_set_svc_name(), rpc_gss_getcred >> () etc.). >> >> I also have a working GSSAPI redirector which allows the use of >> multiple GSSAPI mechanisms, such as SPKEY which is required by the >> spec. It even includes an almost complete set of manpages culled >> from the RFC. >> > > Are any of you planning at being at EuroBSDCon? What with a > FreeBSD developer summit happening about the same time, it would > make sense to get NFSv4-interested people in one place. I'm afraid not - I'm limiting myself to one conference a year (since I'm paying out of my own pocket) and this year was Usenix. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 06:28:09 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 64B3D16A420 for ; Fri, 14 Oct 2005 06:28:09 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from antsrv1.ant.uni-bremen.de (antsrv1.ant.uni-bremen.de [134.102.176.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 04C1E43D46 for ; Fri, 14 Oct 2005 06:28:08 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from bremerhaven.ant.uni-bremen.de ([134.102.176.10]) by antsrv1.ant.uni-bremen.de with esmtp (Exim 4.54 (FreeBSD)) id 1EQJ32-000DwI-Cf; Fri, 14 Oct 2005 08:28:08 +0200 Message-ID: <434F4FF8.9050903@ant.uni-bremen.de> Date: Fri, 14 Oct 2005 08:28:08 +0200 From: Heinrich Rebehn User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050831 Debian/1.7.8-1sarge2 X-Accept-Language: en MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 06:28:09 -0000 Hi list, since i got no reply on the questions@ ML i try my luck here: I want to use ACLs to enable the group "wiss" to delete all files that a lab user has created in his home directory "/export/homes/lab/a1". I set up ACLs as follows: root@antsrv1 [/export/homes/lab] # getfacl a1 #file:a1 #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx mask::rwx other::--- root@antsrv1 [/export/homes/lab] # getfacl -d a1 #file:a1 #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx mask::rwx other::--- Now we create a directory in ~a1: root@antsrv1 [/export/homes/lab] # cd a1 root@antsrv1 [/export/homes/lab/a1] # mkdir d root@antsrv1 [/export/homes/lab/a1] # getfacl d #file:d #owner:0 #group:1022 user::rwx group::--- group:wiss:rwx # effective: r-x mask::r-x other::--- The mask has not been inherited from the upper level directory! The next directory has been created by the user extracting a tar ball: root@antsrv1 [/export/homes/lab/a1] # getfacl STonX-0.6.5/ #file:STonX-0.6.5/ #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx # effective: --x mask::--x other::--- In this case, the "wiss" group can not even read the directory. So, my idea to enable the wiss group to manage the lab user's files does not seem to work. Am i doing something wrong here? Why is the mask not propagated? Any hint would be greately appreciated. I am using 5.4-RELEASE-p7, the filesystem is UFS2. Update: I saw a post suggesting using different umasks, but that did not work either (besides being a bit clumsy solution). -- Heinrich Rebehn University of Bremen Physics / Electrical and Electronics Engineering - Department of Telecommunications - Phone : +49/421/218-4664 Fax : -3341 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 06:41:49 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6A97E16A41F for ; Fri, 14 Oct 2005 06:41:49 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) Received: from relay2.tomsk.ru (relay2.tomsk.ru [212.73.124.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 92F5843D45 for ; Fri, 14 Oct 2005 06:41:48 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) X-Virus-Scanned: by Dr.Web (R) daemon for FreeBSD, version 4.32.1 (2004-08-30) at relay2.tomsk.ru Received: from [172.16.138.125] (account sudakovva@sibptus.tomsk.ru HELO admin.sibptus.tomsk.ru) by relay2.tomsk.ru (CommuniGate Pro SMTP 4.3.2) with ESMTPSA id 1369522; Fri, 14 Oct 2005 13:41:46 +0700 Received: (from sudakov@localhost) by admin.sibptus.tomsk.ru (8.12.9p2/8.12.9/Submit) id j9E6fjqj040946; Fri, 14 Oct 2005 13:41:45 +0700 (OMSST) (envelope-from sudakov@sibptus.tomsk.ru) X-Authentication-Warning: admin.sibptus.tomsk.ru: sudakov set sender to sudakov@sibptus.tomsk.ru using -f Date: Fri, 14 Oct 2005 13:41:45 +0700 From: Victor Sudakov To: Heinrich Rebehn Message-ID: <20051014064145.GA40856@admin.sibptus.tomsk.ru> References: <434F4FF8.9050903@ant.uni-bremen.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <434F4FF8.9050903@ant.uni-bremen.de> User-Agent: Mutt/1.4.2.1i Organization: AO "Svyaztransneft", SibPTUS X-PGP-Key: http://vas.tomsk.ru/vas.asc Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 06:41:49 -0000 Heinrich Rebehn wrote: > [dd] > Am i doing something wrong here? Why is the mask not propagated? I am afraid the current umask prevents it. You must set it to something like "umask 002" before you create your files or directories (the group write bit matters here). > > Update: I saw a post suggesting using different umasks, but that did not > work either (besides being a bit clumsy solution). I agree it may be clumsy but it does work, I use it. Set the user's umask from login.conf -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:sudakov@sibptus.tomsk.ru From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 08:25:10 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0727116A41F; Fri, 14 Oct 2005 08:25:10 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60C5343D46; Fri, 14 Oct 2005 08:25:08 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9E8OwmQ008634 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO); Fri, 14 Oct 2005 10:24:58 +0200 (CEST) Received: from corbeau.imag.fr ([129.88.43.162]) by mail-veri.imag.fr with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQKs6-0002o8-2o; Fri, 14 Oct 2005 10:24:58 +0200 Date: Fri, 14 Oct 2005 10:24:58 +0200 From: Nicolas KOWALSKI To: freebsd-fs@freebsd.org, freebsd-net@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 10:24:58 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 08:25:10 -0000 Hello, This is a repost of a question I sent to freebsd-questions 10 days ago. I "crosspost" it to freebsd-fs and freebsd-net, because the question is about both... Our FreeBSD 4.10 NFS server has some problems serving files by NFS on TCP (no problem with UDP) when the Linux (2.6) or Solaris (5.9) clients shut down in an unclean manner (power failure). When the clients try to mount the shares from the server after an unclean shutdown, the mount process hang during several minutes (delay is varying), then succeeds. I used ethereal on a Linux client, just after an unclean shutdown, and it looks like the client is sending SYN packets to the 2049 port of the server, but the server does not respond to this ; the client retries a few seconds later, and the server still does not acknoledge it ; and it continues several times, until succeeding. Here is a trace ; the frames showing the problem are 33-36, when the client request a SYN to the 2049 port. No. Time Source Destination Protocol Info 16 0.001774 MOUNT V3 MNT Call (Reply In 17) 17 0.001983 MOUNT V3 MNT Reply (Call In 16) 18 0.002013 TCP 909 > 805 [ACK] Seq=105 Ack=73 Win=5840 Len=0 TSV=4294748848 TSER=1226883802 19 0.002233 TCP 911 > sunrpc [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=4294748848 TSER=0 WS=2 20 0.002418 TCP sunrpc > 911 [SYN, ACK] Seq=0 Ack=1 Win=57344 Len=0 MSS=1460 WS=0 TSV=1226883802 TSER=4294748848 21 0.002447 TCP 911 > sunrpc [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=4294748848 TSER=1226883802 22 0.002579 Portmap V2 GETPORT Call (Reply In 23) 23 0.002782 Portmap V2 GETPORT Reply (Call In 22) 24 0.002807 TCP 911 > sunrpc [ACK] Seq=61 Ack=33 Win=5840 Len=0 TSV=4294748849 TSER=1226883802 25 0.002941 TCP 911 > sunrpc [FIN, ACK] Seq=61 Ack=33 Win=5840 Len=0 TSV=4294748849 TSER=1226883802 26 0.003042 TCP sunrpc > 911 [ACK] Seq=33 Ack=62 Win=57920 Len=0 TSV=1226883803 TSER=4294748849 27 0.003057 TCP sunrpc > 911 [FIN, ACK] Seq=33 Ack=62 Win=57920 Len=0 TSV=1226883803 TSER=4294748849 28 0.003105 TCP 911 > sunrpc [ACK] Seq=62 Ack=34 Win=5840 Len=0 TSV=4294748849 TSER=1226883803 29 0.003264 TCP 909 > 805 [FIN, ACK] Seq=105 Ack=73 Win=5840 Len=0 TSV=4294748849 TSER=1226883802 30 0.003417 TCP 805 > 909 [ACK] Seq=73 Ack=106 Win=57920 Len=0 TSV=1226883803 TSER=4294748849 31 0.003433 TCP 805 > 909 [FIN, ACK] Seq=73 Ack=106 Win=57920 Len=0 TSV=1226883803 TSER=4294748849 32 0.003480 TCP 909 > 805 [ACK] Seq=106 Ack=74 Win=5840 Len=0 TSV=4294748849 TSER=1226883803 33 0.003881 TCP 798 > 2049 [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=4294748850 TSER=0 WS=2 34 3.004094 TCP 798 > 2049 [SYN] Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294751850 TSER=0 WS=2 35 9.003506 TCP 798 > 2049 [SYN] Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294757850 TSER=0 WS=2 36 21.001319 TCP 798 > 2049 [SYN] Seq=0 Ack=0 Win=23360 Len=0 MSS=1460 TSV=4294769850 TSER=0 WS=2 Is there something I can set on the server (sysctl variable perhaps), to make it accept the SYN packets from the clients ? As an counter-example, here is a trace when the same Linux client, after an unclean shutdown, tries to mount a previously (before the powerfailure) NFS mounted share from a SunOS 5.9 server. The SYN packet is acknowledged by the server (frames 36-37), so the client does not hang mounting the share. No. Time Source Destination Protocol Info 17 0.002257 MOUNT V3 MNT Call (Reply In 19) 18 0.002391 TCP 32785 > 911 [ACK] Seq=1 Ack=93 Win=49140 Len=0 TSV=613133117 TSER=4294753259 19 0.006256 MOUNT V3 MNT Reply (Call In 17) 20 0.006280 TCP 911 > 32785 [ACK] Seq=93 Ack=77 Win=5840 Len=0 TSV=4294753263 TSER=613133117 21 0.006514 TCP 913 > sunrpc [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=4294753264 TSER=0 WS=2 22 0.006670 TCP sunrpc > 913 [SYN, ACK] Seq=0 Ack=1 Win=49232 Len=0 TSV=613133117 TSER=4294753264 MSS=1460 WS=0 23 0.006700 TCP 913 > sunrpc [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=4294753264 TSER=613133117 24 0.006838 Portmap V2 GETPORT Call (Reply In 26) 25 0.007020 TCP sunrpc > 913 [ACK] Seq=1 Ack=61 Win=49172 Len=0 TSV=613133117 TSER=4294753264 26 0.007452 Portmap V2 GETPORT Reply (Call In 24) 27 0.007475 TCP 913 > sunrpc [ACK] Seq=61 Ack=33 Win=5840 Len=0 TSV=4294753265 TSER=613133117 28 0.007598 TCP 913 > sunrpc [FIN, ACK] Seq=61 Ack=33 Win=5840 Len=0 TSV=4294753265 TSER=613133117 29 0.007712 TCP 911 > 32785 [FIN, ACK] Seq=93 Ack=77 Win=5840 Len=0 TSV=4294753265 TSER=613133117 30 0.007766 TCP sunrpc > 913 [ACK] Seq=33 Ack=62 Win=49232 Len=0 TSV=613133118 TSER=4294753265 31 0.007816 TCP sunrpc > 913 [FIN, ACK] Seq=33 Ack=62 Win=49232 Len=0 TSV=613133118 TSER=4294753265 32 0.007839 TCP 913 > sunrpc [ACK] Seq=62 Ack=34 Win=5840 Len=0 TSV=4294753265 TSER=613133118 33 0.007893 TCP 32785 > 911 [ACK] Seq=77 Ack=94 Win=49232 Len=0 TSV=613133118 TSER=4294753265 34 0.008012 TCP 32785 > 911 [FIN, ACK] Seq=77 Ack=94 Win=49232 Len=0 TSV=613133118 TSER=4294753265 35 0.008145 TCP 911 > 32785 [ACK] Seq=94 Ack=78 Win=5840 Len=0 TSV=4294753265 TSER=613133118 36 0.008553 TCP 799 > 2049 [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=4294753266 TSER=0 WS=2 37 0.008790 TCP 2049 > 799 [SYN, ACK] Seq=0 Ack=1 Win=49232 Len=0 TSV=613133118 TSER=4294753266 MSS=1460 WS=0 38 0.008836 TCP 799 > 2049 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=4294753266 TSER=613133118 Thanks, -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 08:29:37 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A1B5A16A41F for ; Fri, 14 Oct 2005 08:29:37 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4F93643D46 for ; Fri, 14 Oct 2005 08:29:37 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id D9E3F46C05; Fri, 14 Oct 2005 04:29:36 -0400 (EDT) Date: Fri, 14 Oct 2005 09:29:36 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Victor Sudakov In-Reply-To: <20051014064145.GA40856@admin.sibptus.tomsk.ru> Message-ID: <20051014092250.D66245@fledge.watson.org> References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 08:29:37 -0000 On Fri, 14 Oct 2005, Victor Sudakov wrote: > Heinrich Rebehn wrote: >> > > [dd] >> Am i doing something wrong here? Why is the mask not propagated? > > I am afraid the current umask prevents it. You must set it to something > like "umask 002" before you create your files or directories (the group > write bit matters here). The problem, so to speak, is that we actually implement what is described in the POSIX.1e spec. When we did our initial implementation, the various OS's varied a bit in the semantics they implemented: - Solaris implemented umask override if the mask was specified in the default ACL. - IRIX implemented the spec. Since that time, Linux has turned up and implemented the Solaris model, and IRIX has switched to the Solaris model also as a result of peer pressure. I've previouly looked at switching us, but it tears up our kernel APIs some and will require significant testing. I had hoped to do this for FreeBSD 6.x but was derailed working on other problems that needed to be fixed. My hope is to change the default in FreeBSD 7.x. Doing this requires: (1) All file creation VOP's to accept different fields -- rather than accepting the completed creation mode, they need to accept the creation mask and requested creation mode. (2) The fairly dispersed current logic for combining the umask and requested creation mask needs to be discovered, normalized, and documented. You'll notice if you grep around that the umask + creation mode processing uses slightly different bit combination and masking operations depending on object type. Only code inspection combined with some portability testing will tell us if what's there now is a bug or a feature. (3) Addition of logic to kern_acl.c so that file systems implementing POSIX.1e can ask the revised question about initial ACL and file mode. (4) Much testing. Ideally, creastion of fairly extensive regression tests having to do with the modes of files on creation, ACLs, etc. There's also been a recent discussion on trustedbsd-discuss about implementing alternative semantics based on the NFSv4 ACL model. I've taken a walk through the spec and a bit of initial hacking, and need to send e-mail to the NFSv4 working group mailing list asking for clarification of some points. If we did do this, we would presumably add a new flag, nfsv4_acl, for UFS, to allow the administrator to select one of two models. A further complexity is that these models are require different, and so we'd have to look carefully at tools and application behavior. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 09:30:34 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4090116A420 for ; Fri, 14 Oct 2005 09:30:34 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) Received: from relay2.tomsk.ru (relay2.tomsk.ru [212.73.124.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id B6AC343D46 for ; Fri, 14 Oct 2005 09:30:32 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) X-Virus-Scanned: by Dr.Web (R) daemon for FreeBSD, version 4.32.1 (2004-08-30) at relay2.tomsk.ru Received: from [172.16.138.125] (account sudakovva@sibptus.tomsk.ru HELO admin.sibptus.tomsk.ru) by relay2.tomsk.ru (CommuniGate Pro SMTP 4.3.2) with ESMTPSA id 1371042 for freebsd-fs@freebsd.org; Fri, 14 Oct 2005 16:30:30 +0700 Received: (from sudakov@localhost) by admin.sibptus.tomsk.ru (8.12.9p2/8.12.9/Submit) id j9E9UUPN042407 for freebsd-fs@freebsd.org; Fri, 14 Oct 2005 16:30:30 +0700 (OMSST) (envelope-from sudakov@sibptus.tomsk.ru) X-Authentication-Warning: admin.sibptus.tomsk.ru: sudakov set sender to sudakov@sibptus.tomsk.ru using -f Date: Fri, 14 Oct 2005 16:30:30 +0700 From: Victor Sudakov To: freebsd-fs@freebsd.org Message-ID: <20051014093030.GA42382@admin.sibptus.tomsk.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Organization: AO "Svyaztransneft", SibPTUS X-PGP-Key: http://vas.tomsk.ru/vas.asc Subject: "dump -L " not working as expected? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 09:30:34 -0000 Colleagues, Perhaps my message was overlooked, so I risk to repeat it. I dump an active filesystem on a FreeBSD 5.4-RELEASE-p7 with the -L option. dump says: "Dumping snapshot of /dev/mirror/gm1s1h (/home) to ..." However when I later "restore -r" the filesystem, I keep getting messages like ./www/data/ASN/bay_3.log: (inode 805993) not found on tape expected next file 23553, got 6 expected next file 805964, got 805963 expected next file 806010, got 806009 Why is that? I am used to seeing such messages on FreeBSD 4.x and earlier systems, but I thought I would never see them again when dumping a snapshot. Thanks in advance for any input. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:sudakov@sibptus.tomsk.ru From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 09:41:29 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76CA316A420 for ; Fri, 14 Oct 2005 09:41:29 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9CBB843D48 for ; Fri, 14 Oct 2005 09:41:28 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9E9fPgS003964 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2005 11:41:25 +0200 (CEST) Received: from corbeau.imag.fr ([129.88.43.162]) by mail-veri.imag.fr with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQM45-000417-8Y for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 11:41:25 +0200 Received: from kowalski by corbeau.imag.fr with local (Exim 4.50) id 1EQM45-0005t4-5e for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 11:41:25 +0200 To: freebsd-fs@FreeBSD.org References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> From: Nicolas KOWALSKI Date: Fri, 14 Oct 2005 11:41:25 +0200 In-Reply-To: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Nicolas Kowalski X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 11:41:25 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 09:41:29 -0000 on@cs.ait.ac.th writes: > Nicolas KOWALSKI wrote: >> Our FreeBSD 4.10 NFS server has some problems serving files by NFS on >> TCP (no problem with UDP) when the Linux (2.6) or Solaris (5.9) >> clients shut down in an unclean manner (power failure). When the >> clients try to mount the shares from the server after an >> unclean shutdown, the mount process hang during several minutes (delay >> is varying), then succeeds. > > That is just a wild guess, but NFS mounting would happen always at > the same stage of the boot, so maybe with the same source port > number and you could be facing the problem that the connection is > waiting for termination on the server (close_wait or fin_wait or > something)... Se source port in working example is 798 and source > port in failing example is 799 certainly not random. This is certainly right. So, what can I do to make the server not wait so long for the connection termination ? Is there a TTL parameter I can adjust somewhere ? Thanks, -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 11:59:43 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AF2B416A41F for ; Fri, 14 Oct 2005 11:59:43 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from antsrv1.ant.uni-bremen.de (antsrv1.ant.uni-bremen.de [134.102.176.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4DAEA43D48 for ; Fri, 14 Oct 2005 11:59:42 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from bremerhaven.ant.uni-bremen.de ([134.102.176.10]) by antsrv1.ant.uni-bremen.de with esmtp (Exim 4.54 (FreeBSD)) id 1EQODu-000Ff8-7a; Fri, 14 Oct 2005 13:59:42 +0200 Message-ID: <434F9DAE.6070607@ant.uni-bremen.de> Date: Fri, 14 Oct 2005 13:59:42 +0200 From: Heinrich Rebehn User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050831 Debian/1.7.8-1sarge2 X-Accept-Language: en MIME-Version: 1.0 To: Victor Sudakov References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> In-Reply-To: <20051014064145.GA40856@admin.sibptus.tomsk.ru> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 11:59:43 -0000 Victor Sudakov wrote: > Heinrich Rebehn wrote: > > > [dd] > >>Am i doing something wrong here? Why is the mask not propagated? > > > I am afraid the current umask prevents it. > You must set it to something like "umask 002" before you create your > files or directories (the group write bit matters here). > This does not always work: # # Show ACLs on current directory # -bash-2.05b$ getfacl . #file:. #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx mask::rwx other::--- -bash-2.05b$ getfacl -d . #file:. #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx mask::rwx other::--- # # create a dir with umask 022 and umask 000, then extract a tar ball # -bash-2.05b$ umask 0022 -bash-2.05b$ mkdir D1 -bash-2.05b$ umask 0 -bash-2.05b$ mkdir D2 -bash-2.05b$ !tar tar xzf /export/linux/root/debian/usr/local/src/TARS/STonX-0.6.5.tar.gz -bash-2.05b$ getfacl * #file:D1 #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx # effective: r-x mask::r-x other::--- #file:D2 #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx mask::rwx other::--- #file:STonX-0.6.5 #owner:624 #group:1022 user::rwx group::--- group:wiss:rwx # effective: --x mask::--x other::--x -bash-2.05b$ As you can see, it works for the dirs created by hand, but not for the dir created by tar. > >>Update: I saw a post suggesting using different umasks, but that did not >>work either (besides being a bit clumsy solution). > > > I agree it may be clumsy but it does work, I use it. > Set the user's umask from login.conf > It's not only clumsy, it doesn't even work reliably :-( I want to have members of the group "wiss" to have full control, no matter what tools are used to create the files (unless the user deliberately resets the ACLs, of course). Regards, Heinrich From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 12:51:53 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4196916A41F; Fri, 14 Oct 2005 12:51:53 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from antsrv1.ant.uni-bremen.de (antsrv1.ant.uni-bremen.de [134.102.176.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id A60CD43D45; Fri, 14 Oct 2005 12:51:52 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from bremerhaven.ant.uni-bremen.de ([134.102.176.10]) by antsrv1.ant.uni-bremen.de with esmtp (Exim 4.54 (FreeBSD)) id 1EQP2N-000Fyk-3c; Fri, 14 Oct 2005 14:51:51 +0200 Message-ID: <434FA9E6.9070009@ant.uni-bremen.de> Date: Fri, 14 Oct 2005 14:51:50 +0200 From: Heinrich Rebehn User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050831 Debian/1.7.8-1sarge2 X-Accept-Language: en MIME-Version: 1.0 To: Robert Watson References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> <20051014092250.D66245@fledge.watson.org> In-Reply-To: <20051014092250.D66245@fledge.watson.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 12:51:53 -0000 Robert Watson wrote: > > On Fri, 14 Oct 2005, Victor Sudakov wrote: > >> Heinrich Rebehn wrote: >> >>> >> >> [dd] >> >>> Am i doing something wrong here? Why is the mask not propagated? >> >> >> I am afraid the current umask prevents it. You must set it to >> something like "umask 002" before you create your files or directories >> (the group write bit matters here). > > > The problem, so to speak, is that we actually implement what is > described in the POSIX.1e spec. When we did our initial implementation, > the various OS's varied a bit in the semantics they implemented: > > - Solaris implemented umask override if the mask was specified in the > default ACL. does umask override or is umask overriden? :-) I suppose the former. > - IRIX implemented the spec. > > Since that time, Linux has turned up and implemented the Solaris model, > and IRIX has switched to the Solaris model also as a result of peer > pressure. I've previouly looked at switching us, but it tears up our > kernel APIs some and will require significant testing. I had hoped to > do this for FreeBSD 6.x but was derailed working on other problems that > needed to be fixed. My hope is to change the default in FreeBSD 7.x. > Doing this requires: > > (1) All file creation VOP's to accept different fields -- rather than > accepting the completed creation mode, they need to accept the > creation mask and requested creation mode. > > (2) The fairly dispersed current logic for combining the umask and > requested creation mask needs to be discovered, normalized, and > documented. You'll notice if you grep around that the umask + > creation mode processing uses slightly different bit combination and > masking operations depending on object type. Only code inspection > combined with some portability testing will tell us if what's there > now is a bug or a feature. > > (3) Addition of logic to kern_acl.c so that file systems implementing > POSIX.1e can ask the revised question about initial ACL and file mode. > > (4) Much testing. Ideally, creastion of fairly extensive regression tests > having to do with the modes of files on creation, ACLs, etc. > > There's also been a recent discussion on trustedbsd-discuss about > implementing alternative semantics based on the NFSv4 ACL model. I've > taken a walk through the spec and a bit of initial hacking, and need to > send e-mail to the NFSv4 working group mailing list asking for > clarification of some points. If we did do this, we would presumably > add a new flag, nfsv4_acl, for UFS, to allow the administrator to select > one of two models. A further complexity is that these models are > require different, and so we'd have to look carefully at tools and > application behavior. > > Robert N M Watson Thanks for this in-depth explanation. This sounds like we cannot expect a solution any time soon. I will think about another method of managing our lab users (or use adjust umask - better than nothing). I would really appreciate alternative models for NFS4. --Heinrich From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 12:56:51 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 916D216A41F for ; Fri, 14 Oct 2005 12:56:51 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id CA2AF43D45 for ; Fri, 14 Oct 2005 12:56:48 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9ECuhJ9003296 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2005 14:56:43 +0200 (CEST) Received: from obiou.imag.fr ([129.88.43.2]) by mail-veri.imag.fr with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQP75-0008Ir-Es for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 14:56:43 +0200 Received: from kowalski by obiou.imag.fr with local (Exim 4.50) id 1EQP75-0001gj-9q for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 14:56:43 +0200 To: freebsd-fs@FreeBSD.org References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> From: Nicolas KOWALSKI Date: Fri, 14 Oct 2005 14:56:43 +0200 In-Reply-To: <20051014045824.V5343@odysseus.silby.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Nicolas Kowalski X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 14:56:43 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 12:56:51 -0000 Mike Silbersack writes: > On Fri, 14 Oct 2005, on@cs.ait.ac.th wrote: > >> Nicolas KOWALSKI wrote: >>> Our FreeBSD 4.10 NFS server has some problems serving files by NFS >>> on TCP (no problem with UDP) when the Linux (2.6) or Solaris (5.9) >>> clients shut down in an unclean manner (power failure). When the >>> clients try to mount the shares from the server after an unclean >>> shutdown, the mount process hang during several minutes (delay is >>> varying), then succeeds. >> >> That is just a wild guess, but NFS mounting would happen always at >> the same stage of the boot, so maybe with the same source port >> number and you could be facing the problem that the connection is >> waiting for termination on the server (close_wait or fin_wait or >> something)... Se source port in working example is 798 and source >> port in failing example is 799 certainly not random. > > The socket on the server would still be in the ESTABLISHED state, > which is even worse than the close_wait or fin_wait states in this > case. The SYN will be accepted if it's greater than the previous > sequence number, so that's a 50% chance it'll work. Thanks for this explanation. > Assuming that port reuse is the problem, there is no quick fix for > this, just resetting connections when a SYN comes in would be a > really big security problem. Really? Are Linux and Solaris that insecure because of this behaviour? > Actually, there may be a quick fix for this specific machine. If you > set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz is), > that'll cause keepalive packets to be sent every minute to an idle > connection, rather than every 2 hours. That would kill the stuck > connections much quicker. Unfortunately, this does not work as expected. I just tested with my workstation (Linux 2.6), with NFS filesystems mounted with TCP; when the station rebooted abruptely, mounting the same NFS filesystems hung more than 1 minute (15 minutes just now). During this hang, I saw on the server, using netstat, the nfsd process related to my workstation in ESTABLISHED state. Any other tip? Many Thanks in advance, -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 13:20:56 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4225316A41F for ; Fri, 14 Oct 2005 13:20:56 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id E647F43D46 for ; Fri, 14 Oct 2005 13:20:55 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 0F52346BA6; Fri, 14 Oct 2005 09:20:55 -0400 (EDT) Date: Fri, 14 Oct 2005 14:20:54 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Heinrich Rebehn In-Reply-To: <434FA9E6.9070009@ant.uni-bremen.de> Message-ID: <20051014141732.J22507@fledge.watson.org> References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> <20051014092250.D66245@fledge.watson.org> <434FA9E6.9070009@ant.uni-bremen.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 13:20:56 -0000 On Fri, 14 Oct 2005, Heinrich Rebehn wrote: >> The problem, so to speak, is that we actually implement what is >> described in the POSIX.1e spec. When we did our initial >> implementation, the various OS's varied a bit in the semantics they >> implemented: >> >> - Solaris implemented umask override if the mask was specified in the >> default ACL. > > does umask override or is umask overriden? :-) I suppose the former. Sorry -- to be more specific, in the Solaris ACL model, the umask will be ignored if a mask exists in the default ACL of the parent. In POSIX.1e, the umask and parent mask are combined to generate a conservative result, avoiding applications leaking data in the event they understand permissions but not ACLs. Of course, many people find it desirable to be able to override the umaks by directory, hence interest in the less conservative model. >> - IRIX implemented the spec. And to clarify this: IRIX and FreeBSD both implemented POSIX.1eD17 as written. We implemented it because it was the spec, and SGI implemented it because the primary editor of that draft of the spec was running their trusted systems team. :-) > Thanks for this in-depth explanation. This sounds like we cannot expect > a solution any time soon. I will think about another method of managing > our lab users (or use adjust umask - better than nothing). I would > really appreciate alternative models for NFS4. I think a solution for 7.0 is quite likely, but a solution for 6.x is less likely because I'm not sure I want to change something like the semantics of ACLs and file system interfaces during a -STABLE branch. I'll have to think about it a bit -- we may be able to offer it as a non-default option that will be configured by default in 7.x, if it's OK to change the internal kernel file system interfaces during the RELENG_6 life span. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 13:48:24 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F2E5616A41F for ; Fri, 14 Oct 2005 13:48:23 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) Received: from relay2.tomsk.ru (relay2.tomsk.ru [212.73.124.8]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2712E43D45 for ; Fri, 14 Oct 2005 13:48:22 +0000 (GMT) (envelope-from sudakov@sibptus.tomsk.ru) X-Virus-Scanned: by Dr.Web (R) daemon for FreeBSD, version 4.32.1 (2004-08-30) at relay2.tomsk.ru Received: from [172.16.138.125] (account sudakovva@sibptus.tomsk.ru HELO admin.sibptus.tomsk.ru) by relay2.tomsk.ru (CommuniGate Pro SMTP 4.3.2) with ESMTPSA id 1372967; Fri, 14 Oct 2005 20:48:21 +0700 Received: (from sudakov@localhost) by admin.sibptus.tomsk.ru (8.12.9p2/8.12.9/Submit) id j9EDmKYu043907; Fri, 14 Oct 2005 20:48:20 +0700 (OMSST) (envelope-from sudakov@sibptus.tomsk.ru) X-Authentication-Warning: admin.sibptus.tomsk.ru: sudakov set sender to sudakov@sibptus.tomsk.ru using -f Date: Fri, 14 Oct 2005 20:48:20 +0700 From: Victor Sudakov To: Heinrich Rebehn Message-ID: <20051014134820.GA43849@admin.sibptus.tomsk.ru> References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> <434F9DAE.6070607@ant.uni-bremen.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <434F9DAE.6070607@ant.uni-bremen.de> User-Agent: Mutt/1.4.2.1i Organization: AO "Svyaztransneft", SibPTUS X-PGP-Key: http://vas.tomsk.ru/vas.asc Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 13:48:24 -0000 Heinrich Rebehn wrote: > > As you can see, it works for the dirs created by hand, but not for the > dir created by tar. I think tar does a chmod on extracted files because it stores and extracts permission information. I really see no way of working around this. However, I think those people who designed POSIX ACLs might have had a solution for this problem, it is too common. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:sudakov@sibptus.tomsk.ru From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:05:11 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AAB9716A41F for ; Fri, 14 Oct 2005 16:05:11 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from mailhub.cs.uoguelph.ca (mailhub.cs.uoguelph.ca [131.104.96.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4466B43D45 for ; Fri, 14 Oct 2005 16:05:11 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by mailhub.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EG59Q3030133 for ; Fri, 14 Oct 2005 12:05:09 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id MAA21731 for fs@freebsd.org; Fri, 14 Oct 2005 12:06:33 -0400 (EDT) Date: Fri, 14 Oct 2005 12:06:33 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141606.MAA21731@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.75 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:05:11 -0000 From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:06:01 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7063016A41F for ; Fri, 14 Oct 2005 16:06:01 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from moe.cs.uoguelph.ca (moe.cs.uoguelph.ca [131.104.96.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 184D243D45 for ; Fri, 14 Oct 2005 16:06:00 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by moe.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EG60sN019966 for ; Fri, 14 Oct 2005 12:06:00 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id MAA21757 for fs@freebsd.org; Fri, 14 Oct 2005 12:07:23 -0400 (EDT) Date: Fri, 14 Oct 2005 12:07:23 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141607.MAA21757@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.55 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:06:01 -0000 As others have noted, the problem is that the connection is still established on the server and the client re-uses the same port#. Due to the NDA, I can't tell you the details, but there was an amusing case at the last NFSv4 Bakeathon which I'll call the "Psychic Server Theory". Basically, my server had a bug (that was fixed) where it would generate a Readdir reply larger than requested for a certain case. A client would crash when it saw this reply. The interesting part was that it would crash again as soon as it was rebooted, before it even attempted a remount. The theory was that my "Evil Psychic Server" KNEW that the client might try to mount it and would crash it "somehow" before the mount took place. (What was really happening was that the server had the TCP connection still established and would resend the reply to the same port#, which this client already had configured for its first mount.) >From an NFS point of view, I can't see a security risk w.r.t. the server breaking the TCP connection more quickly. (I don't know what implications this change might have w.r.t. TCP level denial of service attacks, etc.) What it will do is increase the risk of data corruption on the server, since a TCP reconnect on the client implies it must retry all outstanding requests, including non-idempotent ones. In the generic FreeBSD NFS server, the recent request cache is normally disabled for TCP, so a retry of a non-idempotent RPC can/will corrupt the server file system (from the point of view of what the client expects to have on the file system). My new server does have a redesigned recent request cache that would minimize this risk, although there will always be a worst case scenario where problems could still occur. In summary, unless there is an increased risk of a "denial of service" attack at the TCP level, it would be nice if the nfsd threads could tell TCP that connections can be broken down fairly quickly for unresponsive clients. This assumes a recent request cache that works well for TCP. (Related to this, having "keep alives" done more frequently should detect the rebooted client, since the connection doesn't exist at the other end?) rick ps: It would be nice if someone with the right expertise could explore other things in TCP specifically for NFS. For example, I don't see why a retransmit timeout should go above about 100msec, since net delays are well below that level, even half way around the world these days. Having said that, I don't know enough about TCP retransmit to say that one second retry intervals aren't correct? From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:06:17 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C87E16A420 for ; Fri, 14 Oct 2005 16:06:17 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id C630C43D46 for ; Fri, 14 Oct 2005 16:06:16 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j9EG6EaE009488; Fri, 14 Oct 2005 11:06:15 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FD761.3050506@centtech.com> Date: Fri, 14 Oct 2005 11:05:53 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nicolas KOWALSKI References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:06:17 -0000 Nicolas KOWALSKI wrote: > Mike Silbersack writes: > > >>On Fri, 14 Oct 2005, on@cs.ait.ac.th wrote: >> >> >>>Nicolas KOWALSKI wrote: >>> >>>>Our FreeBSD 4.10 NFS server has some problems serving files by NFS >>>>on TCP (no problem with UDP) when the Linux (2.6) or Solaris (5.9) >>>>clients shut down in an unclean manner (power failure). When the >>>>clients try to mount the shares from the server after an unclean >>>>shutdown, the mount process hang during several minutes (delay is >>>>varying), then succeeds. >>> >>>That is just a wild guess, but NFS mounting would happen always at >>>the same stage of the boot, so maybe with the same source port >>>number and you could be facing the problem that the connection is >>>waiting for termination on the server (close_wait or fin_wait or >>>something)... Se source port in working example is 798 and source >>>port in failing example is 799 certainly not random. >> >>The socket on the server would still be in the ESTABLISHED state, >>which is even worse than the close_wait or fin_wait states in this >>case. The SYN will be accepted if it's greater than the previous >>sequence number, so that's a 50% chance it'll work. > > > Thanks for this explanation. > > >>Assuming that port reuse is the problem, there is no quick fix for >>this, just resetting connections when a SYN comes in would be a >>really big security problem. > > > Really? Are Linux and Solaris that insecure because of this behaviour? > > >>Actually, there may be a quick fix for this specific machine. If you >>set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz is), >>that'll cause keepalive packets to be sent every minute to an idle >>connection, rather than every 2 hours. That would kill the stuck >>connections much quicker. > > > Unfortunately, this does not work as expected. I just tested with my > workstation (Linux 2.6), with NFS filesystems mounted with TCP; when > the station rebooted abruptely, mounting the same NFS filesystems hung > more than 1 minute (15 minutes just now). During this hang, I saw on > the server, using netstat, the nfsd process related to my workstation > in ESTABLISHED state. > > Any other tip? > > Many Thanks in advance, Man fixmount? -A Issues a command to the remote mountd declaring that all of its file systems have been unmounted. This should be used with cau- tion, as it removes all remote mount entries pertaining to the local system, whether or not any file systems are still mounted locally. -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:18:15 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D91916A41F for ; Fri, 14 Oct 2005 16:18:15 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id CC85C43D45 for ; Fri, 14 Oct 2005 16:18:14 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j9EGIDYH009732; Fri, 14 Oct 2005 11:18:14 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FDA30.1040204@centtech.com> Date: Fri, 14 Oct 2005 11:17:52 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: rick@snowhite.cis.uoguelph.ca References: <200510141607.MAA21757@snowhite.cis.uoguelph.ca> In-Reply-To: <200510141607.MAA21757@snowhite.cis.uoguelph.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:18:15 -0000 rick@snowhite.cis.uoguelph.ca wrote: [..snip..] > rick > ps: It would be nice if someone with the right expertise could explore > other things in TCP specifically for NFS. For example, I don't see > why a retransmit timeout should go above about 100msec, since net > delays are well below that level, even half way around the world > these days. Having said that, I don't know enough about TCP retransmit > to say that one second retry intervals aren't correct? Wouldn't this be a problem for a server under high disk load? If the disks are very very busy, and clients are requesting stat's on files, etc, then the server would be waiting on disk, and the time could be way more than 100ms, even more than 1s. Of course, this would be a slow server because of the load, however it does occur, and so lowering it to 100msec might be too aggresive. If you have many many clients, all attempting lots of NFS activity, during times of load you could make the server even more overloaded with all the retransmits, right? Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:21:24 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2525A16A41F for ; Fri, 14 Oct 2005 16:21:24 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0D6C943D5E for ; Fri, 14 Oct 2005 16:21:22 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dumaguete.citi.umich.edu [141.211.133.51]) by citi.umich.edu (Postfix) with ESMTP id 8D1571BBDE; Fri, 14 Oct 2005 12:21:22 -0400 (EDT) To: rick@snowhite.cis.uoguelph.ca From: Jim Rees In-Reply-To: rick@snowhite.cis.uoguelph.ca, Fri, 14 Oct 2005 12:07:23 EDT Date: Fri, 14 Oct 2005 12:21:22 -0400 Sender: rees@citi.umich.edu Message-Id: <20051014162122.8D1571BBDE@citi.umich.edu> Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:21:24 -0000 ps: It would be nice if someone with the right expertise could explore other things in TCP specifically for NFS. For example, I don't see why a retransmit timeout should go above about 100msec nfs/rpc shouldn't retransmit at all over tcp except when there has been a reconnect. Tcp might retransmit, but modern implementations will always choose the right timeout dynamically, unless packet loss is excessive. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:33:32 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A041D16A41F for ; Fri, 14 Oct 2005 16:33:32 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from mailhub.cs.uoguelph.ca (mailhub.cs.uoguelph.ca [131.104.96.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D7AE43D45 for ; Fri, 14 Oct 2005 16:33:32 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by mailhub.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EGXViE009821 for ; Fri, 14 Oct 2005 12:33:31 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id MAA22435 for fs@freebsd.org; Fri, 14 Oct 2005 12:34:55 -0400 (EDT) Date: Fri, 14 Oct 2005 12:34:55 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141634.MAA22435@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.75 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:33:32 -0000 > ps: It would be nice if someone with the right expertise could explore > other things in TCP specifically for NFS. For example, I don't see > why a retransmit timeout should go above about 100msec > > nfs/rpc shouldn't retransmit at all over tcp except when there has been a > reconnect. Tcp might retransmit, but modern implementations will always > choose the right timeout dynamically, unless packet loss is excessive. > Yes, I was referring to TCP retransmits and I was suggesting that TCP might not be "choosing the right timeout dynamically". A lot of what is in TCP timers now dates back to work done w.r.t. congestion avoidance (very good work, I might add) in the late 1980s. I don't know if anyone has revisited this, now that almost everything is gigibit fibre (in the late 1980s, there were still 56Kbps internet links out there). I don't know enough about TCP (and haven't looked at recent literature), but I do know that I see retransmit timeouts back off to 1-2sec quite rapidly, as soon as I introduce a 20msec transit delay between client<->server and throw away the odd packet. The result is abismal NFS/TCP preformance. rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 16:54:27 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1568516A41F for ; Fri, 14 Oct 2005 16:54:27 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from dargo.cs.uoguelph.ca (dargo.cs.uoguelph.ca [131.104.96.159]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8A96043D46 for ; Fri, 14 Oct 2005 16:54:26 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by dargo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EGsPPr027724 for ; Fri, 14 Oct 2005 12:54:25 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id MAA22902 for fs@freebsd.org; Fri, 14 Oct 2005 12:55:49 -0400 (EDT) Date: Fri, 14 Oct 2005 12:55:49 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141655.MAA22902@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.159 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 16:54:27 -0000 >> rick >> ps: It would be nice if someone with the right expertise could explore >> other things in TCP specifically for NFS. For example, I don't see >> why a retransmit timeout should go above about 100msec, since net >> delays are well below that level, even half way around the world >> these days. Having said that, I don't know enough about TCP retransmit >> to say that one second retry intervals aren't correct? > >Wouldn't this be a problem for a server under high disk load? If the >disks are very very busy, and clients are requesting stat's on files, >etc, then the server would be waiting on disk, and the time could be way >more than 100ms, even more than 1s. Of course, this would be a slow >server because of the load, however it does occur, and so lowering it to >100msec might be too aggresive. If you have many many clients, all >attempting lots of NFS activity, during times of load you could make the >server even more overloaded with all the retransmits, right? It is a concern. If the previously sent request is still in the server's TCP socket receive queue, then TCP will throw away the retransmit. If the request is in progress via an nfsd thread, then the recent request cache code should wait for the reply created from the first one and then both requests get copies of the reply. (This introduces overhead, but at least no additional disk I/O or risk of repeating a non-idempotent request.) nb: My current server cache code does this, but I don't believe the one currently in FreeBSD does? The trick is to not have the nfsd threads remove a request from the socket receive queue until the disk subsystem isn't backlogged. Since delayed ACK is disabled for NFS over TCP, the server will then throw away the retransitted request and generate an ACK to the client right away, so the TCP layer in the client won't retransmit it again. The problem is "how do you make sure the nfsd threads don't start a request if the disk I/O subsystem is backlogged". An interesting question and I'd appreciate hearing suggestions. Part of the problem is that many requests can be satified out of caches in the server (such as the vnode/inode in memory, for a Getattr) and the server doesn't know if a request will be doing disk I/O (it's hidden behind the VFS/Vnode layer). One possibility is for nfsd threads to time how long they take to do a request. When the thread sees that time increasing dramatically, it could assume a backlog in the disk I/O subsystem and sleep for a while before getting the next request off a socket receive queue. Sounds like something worth looking at. Unfortunately I think it will require a pretty high resolution time clock and I don't think I can count on that in FreeBSD unless the server has the right hardware? Any other ideas? rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:23:49 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C62A16A41F for ; Fri, 14 Oct 2005 17:23:49 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id D509043D58 for ; Fri, 14 Oct 2005 17:23:48 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dumaguete.citi.umich.edu [141.211.133.51]) by citi.umich.edu (Postfix) with ESMTP id 46B291BBE6; Fri, 14 Oct 2005 13:23:48 -0400 (EDT) To: rick@snowhite.cis.uoguelph.ca From: Jim Rees In-Reply-To: rick@snowhite.cis.uoguelph.ca, Fri, 14 Oct 2005 12:55:49 EDT Date: Fri, 14 Oct 2005 13:23:48 -0400 Sender: rees@citi.umich.edu Message-Id: <20051014172348.46B291BBE6@citi.umich.edu> Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:23:49 -0000 Tcp always throws away retransmissions. Doesn't matter whether the data is still in the receive socket queue or not. The nfs server will never see replayed requests as a result of tcp retransmission. The problem is "how do you make sure the nfsd threads don't start a request if the disk I/O subsystem is backlogged". Isn't this simply a matter of choosing the right number of nfsd threads? From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:39:35 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EE73316A420 for ; Fri, 14 Oct 2005 17:39:35 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3AF1043D5E for ; Fri, 14 Oct 2005 17:39:34 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j9EHdXpG011295; Fri, 14 Oct 2005 12:39:33 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FED3F.5060907@centtech.com> Date: Fri, 14 Oct 2005 12:39:11 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jim Rees References: <20051014172348.46B291BBE6@citi.umich.edu> In-Reply-To: <20051014172348.46B291BBE6@citi.umich.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: fs@freebsd.org, rick@snowhite.cis.uoguelph.ca Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:39:36 -0000 Jim Rees wrote: > Tcp always throws away retransmissions. Doesn't matter whether the data is > still in the receive socket queue or not. The nfs server will never see > replayed requests as a result of tcp retransmission. > > The problem is "how do you make sure the nfsd threads don't start a > request if the disk I/O subsystem is backlogged". > > Isn't this simply a matter of choosing the right number of nfsd threads? I don't think so - you could easily jam up 4 threads with a large number of machines, if traffic is right. Having a larger number of threads allows more clients in that case. If you have too many threads, then when the disks are busy, your load goes high (because of all the threads), but they wait until the disks are not busy. I'm not really sure what it would give you to have the nfsd's wait until disk is not as busy, as that is what it is doing now, right? Maybe you would smooth out your spikes a bit, but that's not saying it is running any better. There's no cure for slow block devices, except for more cache, or faster block devices. :) Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:40:32 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8193B16A41F for ; Fri, 14 Oct 2005 17:40:32 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 85C9543D5F for ; Fri, 14 Oct 2005 17:40:29 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9EHePU9021403 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2005 19:40:25 +0200 (CEST) Received: from corbeau.imag.fr ([129.88.43.162]) by mail-veri.imag.fr with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQTXd-0006JX-PZ for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 19:40:25 +0200 Received: from kowalski by corbeau.imag.fr with local (Exim 4.50) id 1EQTXd-0001xV-OM for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 19:40:25 +0200 To: freebsd-fs@FreeBSD.org References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> From: Nicolas KOWALSKI Date: Fri, 14 Oct 2005 19:40:25 +0200 In-Reply-To: <434FD761.3050506@centtech.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Nicolas Kowalski X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 19:40:25 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:40:32 -0000 Eric Anderson writes: > Nicolas KOWALSKI wrote: >> Mike Silbersack writes: >> >>>Actually, there may be a quick fix for this specific machine. If >>>you set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz is), >>>that'll cause keepalive packets to be sent every minute to an idle >>>connection, rather than every 2 hours. That would kill the stuck >>>connections much quicker. >> Unfortunately, this does not work as expected. I just tested with >> my workstation (Linux 2.6), with NFS filesystems mounted with TCP; >> when the station rebooted abruptely, mounting the same NFS >> filesystems hung more than 1 minute (15 minutes just now). During >> this hang, I saw on the server, using netstat, the nfsd process >> related to my workstation in ESTABLISHED state. > > > Man fixmount? This is a FreeBSD-only command apparently. I did not find it on Linux or Solaris. It could have been useful, by calling it before NFS filesystems are mounted on clients, yes. Thanks anyway, -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:41:37 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8733C16A44E for ; Fri, 14 Oct 2005 17:41:37 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from dargo.cs.uoguelph.ca (dargo.cs.uoguelph.ca [131.104.96.159]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3199043D45 for ; Fri, 14 Oct 2005 17:41:36 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by dargo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EHfYeU013258 for ; Fri, 14 Oct 2005 13:41:36 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id NAA23853 for fs@freebsd.org; Fri, 14 Oct 2005 13:42:58 -0400 (EDT) Date: Fri, 14 Oct 2005 13:42:58 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141742.NAA23853@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.159 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:41:37 -0000 > Tcp always throws away retransmissions. Doesn't matter whether the data is > still in the receive socket queue or not. The nfs server will never see > replayed requests as a result of tcp retransmission. 1) Yes, of course. (Brain fart on my part. I mentioned I wasn't a TCP wizzard, didn't I?:-) There is still the problem that not all clients obey the "only retry an RPC after a disconnect/reconnect" rule, but that should be fixed for NFSv4. > The problem is "how do you make sure the nfsd threads don't start a > request if the disk I/O subsystem is backlogged". > > Isn't this simply a matter of choosing the right number of nfsd threads? That's the only mechanism available to-day (at least for my server and, I think, the current FreeBSD one). Unfortunately load is pretty dynamic and it might be nice if "the number of active nfsd threads" changed to reflect that, without manual sysadmin intervention. However, given 1) and clients that only retry RPCs after a reconnect, I don't think it will be all that critical. The clients will see the slow response to RPCs and new requests will be limited by the number of threads doing I/O on the client. Other opinions? rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:41:49 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DAC6816A423 for ; Fri, 14 Oct 2005 17:41:49 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E94243D49 for ; Fri, 14 Oct 2005 17:41:48 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j9EHfm49091944; Fri, 14 Oct 2005 12:41:48 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FEDC6.4040405@centtech.com> Date: Fri, 14 Oct 2005 12:41:26 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nicolas KOWALSKI References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:41:50 -0000 Nicolas KOWALSKI wrote: > Eric Anderson writes: > > >>Nicolas KOWALSKI wrote: >> >>>Mike Silbersack writes: >>> >>> >>>>Actually, there may be a quick fix for this specific machine. If >>>>you set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz is), >>>>that'll cause keepalive packets to be sent every minute to an idle >>>>connection, rather than every 2 hours. That would kill the stuck >>>>connections much quicker. > > >>>Unfortunately, this does not work as expected. I just tested with >>>my workstation (Linux 2.6), with NFS filesystems mounted with TCP; >>>when the station rebooted abruptely, mounting the same NFS >>>filesystems hung more than 1 minute (15 minutes just now). During >>>this hang, I saw on the server, using netstat, the nfsd process >>>related to my workstation in ESTABLISHED state. >> >> >>Man fixmount? > > > This is a FreeBSD-only command apparently. I did not find it on Linux > or Solaris. It could have been useful, by calling it before NFS > filesystems are mounted on clients, yes. It's available on Fedora Core 2 and 3 at least. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:47:32 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1DCE116A420 for ; Fri, 14 Oct 2005 17:47:32 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4661943D5E for ; Fri, 14 Oct 2005 17:47:31 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9EHlRNJ022957 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2005 19:47:27 +0200 (CEST) Received: from corbeau.imag.fr ([129.88.43.162]) by mail-veri.imag.fr with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQTeR-0006VU-17 for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 19:47:27 +0200 Received: from kowalski by corbeau.imag.fr with local (Exim 4.50) id 1EQTeQ-0001xy-VS for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 19:47:26 +0200 To: freebsd-fs@FreeBSD.org References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> <434FEDC6.4040405@centtech.com> From: Nicolas KOWALSKI Date: Fri, 14 Oct 2005 19:47:26 +0200 In-Reply-To: <434FEDC6.4040405@centtech.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Nicolas Kowalski X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 19:47:27 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:47:33 -0000 Eric Anderson writes: > Nicolas KOWALSKI wrote: >> Eric Anderson writes: >> >>>Nicolas KOWALSKI wrote: >>> >>>>Mike Silbersack writes: >>>> >>>> >>>>>Actually, there may be a quick fix for this specific machine. If >>>>>you set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz >>>>>is), that'll cause keepalive packets to be sent every minute to >>>>>an idle connection, rather than every 2 hours. That would kill >>>>>the stuck connections much quicker. >> >>>>Unfortunately, this does not work as expected. I just tested with >>>>my workstation (Linux 2.6), with NFS filesystems mounted with TCP; >>>>when the station rebooted abruptely, mounting the same NFS >>>>filesystems hung more than 1 minute (15 minutes just now). During >>>>this hang, I saw on the server, using netstat, the nfsd process >>>>related to my workstation in ESTABLISHED state. >>> >>> >>>Man fixmount? >> This is a FreeBSD-only command apparently. I did not find it on >> Linux or Solaris. It could have been useful, by calling it before >> NFS filesystems are mounted on clients, yes. > > It's available on Fedora Core 2 and 3 at least. So, its a non-option, because we are only using Debian Sarge and Solaris 9 UNIX workstations. :-( Thanks for your advice, -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:49:28 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 72CAF16A420 for ; Fri, 14 Oct 2005 17:49:28 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from dargo.cs.uoguelph.ca (dargo.cs.uoguelph.ca [131.104.96.159]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5F3EA43D53 for ; Fri, 14 Oct 2005 17:49:26 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by dargo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EHnPYZ016672 for ; Fri, 14 Oct 2005 13:49:25 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id NAA24051 for fs@freebsd.org; Fri, 14 Oct 2005 13:50:49 -0400 (EDT) Date: Fri, 14 Oct 2005 13:50:49 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141750.NAA24051@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.159 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:49:28 -0000 [good stuff snipped] > threads), but they wait until the disks are not busy. I'm not really > sure what it would give you to have the nfsd's wait until disk is not as > busy, as that is what it is doing now, right? Maybe you would smooth The difference is that, if the nfsd threads don't take the request off the socket receive queue, then the TCP send window slows/stops new requests being sent from the client at some point. Currently, the nfsd thread takes the request off the TCP sockets receive queue and then sleeps inside the VFS/VnodeOp calls. (As noted in the previous email, I think what happens now is good enough for most cases, so long as the client doesn't retry the RPCs. I had forgotten that the TCP seq# would cause TCP level retransmits to be discarded, even after the request is removed from the receive queue.) Fun stuff, rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:49:50 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D003516A420 for ; Fri, 14 Oct 2005 17:49:50 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4277743D5E for ; Fri, 14 Oct 2005 17:49:49 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j9EHnmx7011538; Fri, 14 Oct 2005 12:49:48 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FEFA6.8050102@centtech.com> Date: Fri, 14 Oct 2005 12:49:26 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Nicolas KOWALSKI References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> <434FEDC6.4040405@centtech.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:49:51 -0000 Nicolas KOWALSKI wrote: > Eric Anderson writes: > > >>Nicolas KOWALSKI wrote: >> >>>Eric Anderson writes: >>> >>> >>>>Nicolas KOWALSKI wrote: >>>> >>>> >>>>>Mike Silbersack writes: >>>>> >>>>> >>>>> >>>>>>Actually, there may be a quick fix for this specific machine. If >>>>>>you set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz >>>>>>is), that'll cause keepalive packets to be sent every minute to >>>>>>an idle connection, rather than every 2 hours. That would kill >>>>>>the stuck connections much quicker. >>> >>>>>Unfortunately, this does not work as expected. I just tested with >>>>>my workstation (Linux 2.6), with NFS filesystems mounted with TCP; >>>>>when the station rebooted abruptely, mounting the same NFS >>>>>filesystems hung more than 1 minute (15 minutes just now). During >>>>>this hang, I saw on the server, using netstat, the nfsd process >>>>>related to my workstation in ESTABLISHED state. >>>> >>>> >>>>Man fixmount? >>> >>>This is a FreeBSD-only command apparently. I did not find it on >>>Linux or Solaris. It could have been useful, by calling it before >>>NFS filesystems are mounted on clients, yes. >> >>It's available on Fedora Core 2 and 3 at least. > > > So, its a non-option, because we are only using Debian Sarge and > Solaris 9 UNIX workstations. :-( I'm sure it isn't magic - you could probably easily port it to those. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:50:33 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 43FCD16A41F for ; Fri, 14 Oct 2005 17:50:33 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8CDC643D83 for ; Fri, 14 Oct 2005 17:50:25 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dumaguete.citi.umich.edu [141.211.133.51]) by citi.umich.edu (Postfix) with ESMTP id 749E01BBE6 for ; Fri, 14 Oct 2005 13:50:25 -0400 (EDT) To: freebsd-fs@FreeBSD.org From: Jim Rees In-Reply-To: Nicolas KOWALSKI, Fri, 14 Oct 2005 19:40:25 +0200 Date: Fri, 14 Oct 2005 13:50:25 -0400 Sender: rees@citi.umich.edu Message-Id: <20051014175025.749E01BBE6@citi.umich.edu> Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:50:33 -0000 The tcp spec says that after a reboot an implementation shouldn't send any packets until at least the maximum segment lifetime has elapsed since the crash. So technically the nfs client machine is at fault. But that doesn't really help you. Having sobind return a random port instead of always counting up from 798 (or whatever) might be the right fix here. Later versions of FreeBSD might even do this already. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:52:52 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F3D6816A41F for ; Fri, 14 Oct 2005 17:52:51 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28AF043D69 for ; Fri, 14 Oct 2005 17:52:47 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id j9EHqkZT092134 for ; Fri, 14 Oct 2005 12:52:46 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <434FF058.1010800@centtech.com> Date: Fri, 14 Oct 2005 12:52:24 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.11) Gecko/20050914 X-Accept-Language: en-us, en MIME-Version: 1.0 To: fs@freebsd.org References: <200510141750.NAA24051@snowhite.cis.uoguelph.ca> In-Reply-To: <200510141750.NAA24051@snowhite.cis.uoguelph.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1134/Fri Oct 14 03:07:44 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:52:52 -0000 rick@snowhite.cis.uoguelph.ca wrote: > [good stuff snipped] > >>threads), but they wait until the disks are not busy. I'm not really >>sure what it would give you to have the nfsd's wait until disk is not as >>busy, as that is what it is doing now, right? Maybe you would smooth > > > The difference is that, if the nfsd threads don't take the request off > the socket receive queue, then the TCP send window slows/stops new > requests being sent from the client at some point. > > Currently, the nfsd thread takes the request off the TCP sockets receive > queue and then sleeps inside the VFS/VnodeOp calls. > (As noted in the previous email, I think what happens now is good enough > for most cases, so long as the client doesn't retry the RPCs. I had > forgotten that the TCP seq# would cause TCP level retransmits to be > discarded, even after the request is removed from the receive queue.) Then what happens when the socket receive queue fills up? (Sorry if this is obvious). Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 17:55:31 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2AB6316A41F for ; Fri, 14 Oct 2005 17:55:31 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B2FD43D5D for ; Fri, 14 Oct 2005 17:55:30 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dumaguete.citi.umich.edu [141.211.133.51]) by citi.umich.edu (Postfix) with ESMTP id 28D651BBE4 for ; Fri, 14 Oct 2005 13:55:30 -0400 (EDT) To: fs@freebsd.org From: Jim Rees In-Reply-To: rick@snowhite.cis.uoguelph.ca, Fri, 14 Oct 2005 13:42:58 EDT Date: Fri, 14 Oct 2005 13:55:30 -0400 Sender: rees@citi.umich.edu Message-Id: <20051014175530.28D651BBE4@citi.umich.edu> Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 17:55:31 -0000 There was a bug in the FreeBSD v3 client that it didn't retransmit at all, even after a reconnect. I fixed that in rev 1.125 of nfs_socket.c. It should now retransmit exactly once. That could of course result in a replay at the server, depending on what caused the reconnect. In the case where the server drops the connection due to inactivity, I think there is no replay. From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 18:35:39 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BF4DF16A41F for ; Fri, 14 Oct 2005 18:35:39 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from antsrv1.ant.uni-bremen.de (antsrv1.ant.uni-bremen.de [134.102.176.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 521D243D58 for ; Fri, 14 Oct 2005 18:35:38 +0000 (GMT) (envelope-from rebehn@ant.uni-bremen.de) Received: from 213-172-120-154.dsl.aktivanet.de ([213.172.120.154]) by antsrv1.ant.uni-bremen.de with esmtpa (Exim 4.54 (FreeBSD)) id 1EQUP1-000Hw7-BZ; Fri, 14 Oct 2005 20:35:35 +0200 Message-ID: <434FFAD6.6000002@ant.uni-bremen.de> Date: Fri, 14 Oct 2005 20:37:10 +0200 From: Heinrich Rebehn User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050831 Debian/1.7.8-1sarge2 X-Accept-Language: en MIME-Version: 1.0 To: Nicolas KOWALSKI References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> <434FEDC6.4040405@centtech.com> In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 18:35:39 -0000 Nicolas KOWALSKI wrote: > Eric Anderson writes: > > >>Nicolas KOWALSKI wrote: >> >>>Eric Anderson writes: >>> >>> >>>>Nicolas KOWALSKI wrote: >>>> >>>> >>>>>Mike Silbersack writes: >>>>> >>>>> >>>>> >>>>>>Actually, there may be a quick fix for this specific machine. If >>>>>>you set net.inet.tcp.keepidle to 1 minute (60*whatever kern.hz >>>>>>is), that'll cause keepalive packets to be sent every minute to >>>>>>an idle connection, rather than every 2 hours. That would kill >>>>>>the stuck connections much quicker. >>> >>>>>Unfortunately, this does not work as expected. I just tested with >>>>>my workstation (Linux 2.6), with NFS filesystems mounted with TCP; >>>>>when the station rebooted abruptely, mounting the same NFS >>>>>filesystems hung more than 1 minute (15 minutes just now). During >>>>>this hang, I saw on the server, using netstat, the nfsd process >>>>>related to my workstation in ESTABLISHED state. >>>> >>>> >>>>Man fixmount? >>> >>>This is a FreeBSD-only command apparently. I did not find it on >>>Linux or Solaris. It could have been useful, by calling it before >>>NFS filesystems are mounted on clients, yes. >> >>It's available on Fedora Core 2 and 3 at least. > > > So, its a non-option, because we are only using Debian Sarge and > Solaris 9 UNIX workstations. :-( > For Debian Sarge, it is in am-utils (amd automounter). Since we are bit by the same problem - we have diskless Linux clients that mount their root fs from a FreeBSD server, which sometimes takes some 15 minutes - i will try fixmount next week. --Heinrich From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 19:11:27 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 483BC16A41F for ; Fri, 14 Oct 2005 19:11:27 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from dargo.cs.uoguelph.ca (dargo.cs.uoguelph.ca [131.104.96.159]) by mx1.FreeBSD.org (Postfix) with ESMTP id ED71C43D45 for ; Fri, 14 Oct 2005 19:11:26 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by dargo.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EJBPdQ017358 for ; Fri, 14 Oct 2005 15:11:25 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id PAA25440 for fs@freebsd.org; Fri, 14 Oct 2005 15:12:48 -0400 (EDT) Date: Fri, 14 Oct 2005 15:12:48 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510141912.PAA25440@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.159 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 19:11:27 -0000 > Then what happens when the socket receive queue fills up? (Sorry if > this is obvious). Its been a long time since I went through the BSD TCP code, so if I get this wrong, hopefully someone will jump in and correct it. Here goes... When the socket receive queue is full, the server will throw away further data that it receives, instead of queuing it. At the client end, there is a send window (#byte of unacknowledged data that can be sent) and once it sends "send window" bytes of data that gets thrown away at the server, it will hit the "send window" limit and stop sending more data. The NFS client will block in sosend() until the server moves the send window (which won't happen until the nfsd threads take data out of the socket receive queue on the server). Hope I got that right? rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 19:31:08 2005 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1E3CB16A448 for ; Fri, 14 Oct 2005 19:31:08 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCF6043D60 for ; Fri, 14 Oct 2005 19:31:06 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id F126046BDC; Fri, 14 Oct 2005 15:31:05 -0400 (EDT) Date: Fri, 14 Oct 2005 20:31:05 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Victor Sudakov In-Reply-To: <20051014134820.GA43849@admin.sibptus.tomsk.ru> Message-ID: <20051014203021.L66014@fledge.watson.org> References: <434F4FF8.9050903@ant.uni-bremen.de> <20051014064145.GA40856@admin.sibptus.tomsk.ru> <434F9DAE.6070607@ant.uni-bremen.de> <20051014134820.GA43849@admin.sibptus.tomsk.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Problem with default ACLs and mask X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 19:31:08 -0000 On Fri, 14 Oct 2005, Victor Sudakov wrote: > Heinrich Rebehn wrote: >> >> As you can see, it works for the dirs created by hand, but not for the >> dir created by tar. > > I think tar does a chmod on extracted files because it stores and > extracts permission information. I really see no way of working around > this. > > However, I think those people who designed POSIX ACLs might have had a > solution for this problem, it is too common. Our tar speaks ACLs, but I'm not sure what model it uses to decide what to do with the default ACL of the directory where the tar is extracted. It could well be that tar specifically restores ACLs, overriding the default ACL where the files are untar'd. Robert N M Watson From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 19:55:01 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 65AF916A41F for ; Fri, 14 Oct 2005 19:55:01 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id F141343D46 for ; Fri, 14 Oct 2005 19:55:00 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from [10.58.52.227] (nat-198-95-226-230.netapp.com [198.95.226.230]) by citi.umich.edu (Postfix) with ESMTP id 0FCDC1BBDC; Fri, 14 Oct 2005 15:54:59 -0400 (EDT) Message-ID: <43500D13.1030702@citi.umich.edu> Date: Fri, 14 Oct 2005 15:54:59 -0400 From: Chuck Lever Organization: Network Appliance, Inc. User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: rick@snowhite.cis.uoguelph.ca References: <200510141742.NAA23853@snowhite.cis.uoguelph.ca> In-Reply-To: <200510141742.NAA23853@snowhite.cis.uoguelph.ca> Content-Type: multipart/mixed; boundary="------------070508070106000201080603" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cel@citi.umich.edu List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 19:55:01 -0000 This is a multi-part message in MIME format. --------------070508070106000201080603 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit rick@snowhite.cis.uoguelph.ca wrote: >>Tcp always throws away retransmissions. Doesn't matter whether the data is >>still in the receive socket queue or not. The nfs server will never see >>replayed requests as a result of tcp retransmission. > > > 1) Yes, of course. (Brain fart on my part. I mentioned I wasn't a TCP wizzard, > didn't I?:-) > There is still the problem that not all > clients obey the "only retry an RPC after a disconnect/reconnect" rule, where is that rule stated? most NFS clients i am aware of retransmit an RPC after 60 seconds over TCP. > but that should be fixed for NFSv4. agreed. >> The problem is "how do you make sure the nfsd threads don't start a >> request if the disk I/O subsystem is backlogged". >> >>Isn't this simply a matter of choosing the right number of nfsd threads? > > > That's the only mechanism available to-day (at least for my server and, > I think, the current FreeBSD one). Unfortunately load is pretty dynamic > and it might be nice if "the number of active nfsd threads" changed to > reflect that, without manual sysadmin intervention. > > However, given 1) and clients that only retry RPCs after a reconnect, I > don't think it will be all that critical. The clients will see the slow > response to RPCs and new requests will be limited by the number of threads > doing I/O on the client. that assumes the client uses a threaded implementation too, doesn't it? --------------070508070106000201080603-- From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 19:59:26 2005 Return-Path: X-Original-To: freebsd-fs@FreeBSD.org Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5818F16A41F for ; Fri, 14 Oct 2005 19:59:26 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from imag.imag.fr (imag.imag.fr [129.88.30.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2D66A43D69 for ; Fri, 14 Oct 2005 19:59:20 +0000 (GMT) (envelope-from Nicolas.Kowalski@imag.fr) Received: from mail-veri.imag.fr (mail-veri.imag.fr [129.88.43.52]) by imag.imag.fr (8.13.0/8.13.0) with ESMTP id j9EJxDJ7017523 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Fri, 14 Oct 2005 21:59:13 +0200 (CEST) Received: from corbeau.imag.fr ([129.88.43.162]) by mail-veri.imag.fr with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA:32) (Exim 4.50) id 1EQVhx-0000RO-7B for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 21:59:13 +0200 Received: from kowalski by corbeau.imag.fr with local (Exim 4.50) id 1EQVhx-0002C6-4W for freebsd-fs@FreeBSD.org; Fri, 14 Oct 2005 21:59:13 +0200 To: freebsd-fs@FreeBSD.org References: <20051014160128.hev160v52ossokg0@wwws.cs.ait.ac.th> <20051014045824.V5343@odysseus.silby.com> <434FD761.3050506@centtech.com> <434FEDC6.4040405@centtech.com> <434FFAD6.6000002@ant.uni-bremen.de> From: Nicolas KOWALSKI Date: Fri, 14 Oct 2005 21:59:13 +0200 In-Reply-To: <434FFAD6.6000002@ant.uni-bremen.de> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: Nicolas Kowalski X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-1.6 (imag.imag.fr [129.88.30.1]); Fri, 14 Oct 2005 21:59:13 +0200 (CEST) X-IMAG-MailScanner: Found to be clean X-IMAG-MailScanner-Information: Please contact the ISP for more information Cc: Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 19:59:26 -0000 Heinrich Rebehn writes: > Nicolas KOWALSKI wrote: >> Eric Anderson writes: >> >>>Nicolas KOWALSKI wrote: >>> >>>>Eric Anderson writes: >>>> >>>>>Nicolas KOWALSKI wrote: >>>>> >>>>>>Mike Silbersack writes: >>>>>> >>>>>>>Actually, there may be a quick fix for this specific machine. >>>>>>>If you set net.inet.tcp.keepidle to 1 minute (60*whatever >>>>>>>kern.hz is), that'll cause keepalive packets to be sent every >>>>>>>minute to an idle connection, rather than every 2 hours. That >>>>>>>would kill the stuck connections much quicker. >>>> >>>>>>Unfortunately, this does not work as expected. I just tested >>>>>>with my workstation (Linux 2.6), with NFS filesystems mounted >>>>>>with TCP; when the station rebooted abruptely, mounting the same >>>>>>NFS filesystems hung more than 1 minute (15 minutes just >>>>>>now). During this hang, I saw on the server, using netstat, the >>>>>>nfsd process related to my workstation in ESTABLISHED state. >>>>> >>>>>Man fixmount? >>>> >>>>This is a FreeBSD-only command apparently. I did not find it on >>>>Linux or Solaris. It could have been useful, by calling it before >>>>NFS filesystems are mounted on clients, yes. >>> >>>It's available on Fedora Core 2 and 3 at least. >> So, its a non-option, because we are only using Debian Sarge and >> Solaris 9 UNIX workstations. :-( > > For Debian Sarge, it is in am-utils (amd automounter). Oh, thanks for the information... > Since we are bit by the same problem - we have diskless Linux > clients that mount their root fs from a FreeBSD server, which > sometimes takes some 15 minutes - i will try fixmount next week. I just tried it without success. I call it just before /etc/rcS.d/S45mountnfs, and it does not help. It looks "normal" for me, because the hang (visible on the etherreal trace) does not happen at mount call, but when requesting the SYN to the nfsd port on the server. -- Nicolas From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 20:19:29 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F369016A41F for ; Fri, 14 Oct 2005 20:19:28 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from mailhub.cs.uoguelph.ca (mailhub.cs.uoguelph.ca [131.104.96.75]) by mx1.FreeBSD.org (Postfix) with ESMTP id A37FF43D49 for ; Fri, 14 Oct 2005 20:19:28 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by mailhub.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EKJO9A015793; Fri, 14 Oct 2005 16:19:24 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id QAA26662; Fri, 14 Oct 2005 16:20:48 -0400 (EDT) Date: Fri, 14 Oct 2005 16:20:48 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510142020.QAA26662@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.75 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 20:19:29 -0000 > where is that rule stated? most NFS clients i am aware of retransmit an > RPC after 60 seconds over TCP. For NFSv4, it's in RFC3530, Sec. 3.1.1 (actually applies to RPCs other than NULL). For NFSv2,3 it was never required by the RFCs, so it is questionable what the correct behaviour is. Being the first to do NFS over TCP, I only did retransmits after reconnect. I think I described it that way in the ancient Usenix paper. (http://snowhite.cis.uoguelph.ca/nfsv4, then click on it) When Sun first did NFS over TCP, I believe they did do retries (using a conservative timeout). I think I eventually convinced Sun that it wasn't a good idea and I think that Solaris no longer does them, but I'm not sure. (For this to work correctly, a server is required to disconnect whenever it can't generate a reply to an RPC over TCP for any reason.) So, for NFSv2,3 I don't know of a stated "rule". I don't think it is covered in the NFS interoperability RFC that appeared a while back, but can't remember for sure. The reason I didn't do retries over TCP in the original work are basically what has already been discussed in this thread. I intended that the TCP virtual circuit would apply back pressure to the client to slow/stop sending requests to a congested server. rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 20:26:53 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7B50F16A41F for ; Fri, 14 Oct 2005 20:26:53 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from ccshst09.cs.uoguelph.ca (ccshst09.cs.uoguelph.ca [131.104.96.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1960043D45 for ; Fri, 14 Oct 2005 20:26:52 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by ccshst09.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9EKQpJt012921; Fri, 14 Oct 2005 16:26:51 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id QAA26807; Fri, 14 Oct 2005 16:28:15 -0400 (EDT) Date: Fri, 14 Oct 2005 16:28:15 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510142028.QAA26807@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.18 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 20:26:53 -0000 > that assumes the client uses a threaded implementation too, doesn't it? I used the term "thread" loosely. Anything that might do NFS RPCs concurrently, such as different processes, biods (I called them nfsiods) that do read-ahead or write behind, etc. It wouldn't apply to a DOS client, rick From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 21:16:29 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F32B616A41F for ; Fri, 14 Oct 2005 21:16:28 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4E11643D6A for ; Fri, 14 Oct 2005 21:16:24 +0000 (GMT) (envelope-from rees@citi.umich.edu) Received: from citi.umich.edu (dumaguete.citi.umich.edu [141.211.133.51]) by citi.umich.edu (Postfix) with ESMTP id 6E4D61BB35; Fri, 14 Oct 2005 17:16:24 -0400 (EDT) To: cel@citi.umich.edu From: Jim Rees In-Reply-To: Chuck Lever, Fri, 14 Oct 2005 15:54:59 EDT Date: Fri, 14 Oct 2005 17:16:24 -0400 Sender: rees@citi.umich.edu Message-Id: <20051014211624.6E4D61BB35@citi.umich.edu> Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 21:16:29 -0000 where is that rule stated? most NFS clients i am aware of retransmit an RPC after 60 seconds over TCP. Why would they do that? I would consider that a bug. I am only familiar with the various bsd implementations, but none of them retransmit over tcp. Are there buggy servers that drop rpc requests over tcp? Why? From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 21:24:12 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1C6DA16A41F for ; Fri, 14 Oct 2005 21:24:12 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id D8D6E43D62 for ; Fri, 14 Oct 2005 21:24:10 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from [10.58.52.227] (nat-198-95-226-230.netapp.com [198.95.226.230]) by citi.umich.edu (Postfix) with ESMTP id 03B1B1BBDC; Fri, 14 Oct 2005 17:24:09 -0400 (EDT) Message-ID: <435021F9.5090302@citi.umich.edu> Date: Fri, 14 Oct 2005 17:24:09 -0400 From: Chuck Lever Organization: Network Appliance, Inc. User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Jim Rees References: <20051014211624.6E4D61BB35@citi.umich.edu> In-Reply-To: <20051014211624.6E4D61BB35@citi.umich.edu> Content-Type: multipart/mixed; boundary="------------030606080804020302010005" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cel@citi.umich.edu List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 21:24:12 -0000 This is a multi-part message in MIME format. --------------030606080804020302010005 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Jim Rees wrote: > where is that rule stated? most NFS clients i am aware of retransmit an > RPC after 60 seconds over TCP. > > Why would they do that? because NFS/TCP implementations are usually based on pre-existing NFS/UDP implementations, which already have support for retransmitting. also, see below: some clients might assume that a server could drop an RCP request coming over a TCP connection. > I would consider that a bug. I am only familiar > with the various bsd implementations, but none of them retransmit over tcp. Solaris and Linux both do it. that doesn't mean it's the right thing to do. however, there are reasons why that would make for some interoperability issues with servers that never drop. > Are there buggy servers that drop rpc requests over tcp? Why? i don't know of a specific implementation that drops RPC requests over TCP. however, i can think of some reasons (allowed by "holes" in the various RFCs) why a server implementor might think it's OK to do. --------------030606080804020302010005-- From owner-freebsd-fs@FreeBSD.ORG Fri Oct 14 21:45:03 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EF0B916A41F for ; Fri, 14 Oct 2005 21:45:02 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from citi.umich.edu (citi.umich.edu [141.211.133.111]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8F1D443D45 for ; Fri, 14 Oct 2005 21:45:02 +0000 (GMT) (envelope-from cel@citi.umich.edu) Received: from [10.58.52.227] (nat-198-95-226-230.netapp.com [198.95.226.230]) by citi.umich.edu (Postfix) with ESMTP id 70A151BB35; Fri, 14 Oct 2005 17:44:59 -0400 (EDT) Message-ID: <435026DA.5050101@citi.umich.edu> Date: Fri, 14 Oct 2005 17:44:58 -0400 From: Chuck Lever Organization: Network Appliance, Inc. User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: rick@snowhite.cis.uoguelph.ca References: <200510142020.QAA26662@snowhite.cis.uoguelph.ca> In-Reply-To: <200510142020.QAA26662@snowhite.cis.uoguelph.ca> Content-Type: multipart/mixed; boundary="------------090700060104010008020909" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: fs@freebsd.org Subject: Re: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: cel@citi.umich.edu List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Oct 2005 21:45:03 -0000 This is a multi-part message in MIME format. --------------090700060104010008020909 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit rick@snowhite.cis.uoguelph.ca wrote: >>where is that rule stated? most NFS clients i am aware of retransmit an >>RPC after 60 seconds over TCP. > > > For NFSv4, it's in RFC3530, Sec. 3.1.1 (actually applies to RPCs other > than NULL). i recently had a thorough discussion of this with the author of that section, Mike Eisler. > For NFSv2,3 it was never required by the RFCs, so it is > questionable what the correct behaviour is. Being the first to do NFS over > TCP, I only did retransmits after reconnect. I think I described it that > way in the ancient Usenix paper. (http://snowhite.cis.uoguelph.ca/nfsv4, > then click on it) i will try to grab that. > When Sun first did NFS over TCP, I believe they did > do retries (using a conservative timeout). I think I eventually convinced Sun > that it wasn't a good idea and I think that Solaris no longer > does them, but I'm not sure. (For this to work correctly, a server is required > to disconnect whenever it can't generate a reply to an RPC over TCP for any > reason.) yes, this is a difficult semantic. it means that there is now a race that allows a server to redo a non-idempotent request if the client reconnects on another port and sends a retransmit of a stuck request. i've seen this in practice, and for certain applications this will cause data corruption. most Linux NFS clients will not reconnect on the same port after the server disconnects (a bug i recently addressed). for servers with a duplicate reply cache, this means the client can retransmit non-idempotent requests and the DRC will not stop the requests from being reapplied. such servers are dependent on identifying RPC requests by the tuple of [ XID, source port, client IP ] -- if source port changes, then the DRC is rendered ineffective. servers that don't have a DRC for TCP are exposed to this problem. when they disconnect the TCP connection, they've lost all stream transport guarantees (no request reordering, no duplicate requests). on reconnect a client can retransmit any requests it hasn't received a reply for, which are then reapplied by the server. if the server doesn't guarantee that these retransmitted requests are applied in the same order that the original requests were applied, there is opportunity for data corruption. retransmitting an idempotent request will cause a connection drop, meaning any non-idempotents requests that were outstanding at the time will have to be retransmitted. this is load dependent behavior. when a server slows down, a client that retransmits on TCP is more likely to retransmit one or more non-idempotent requests. this means the server will disconnect, creating even more work for server, network, and client, and it means the likelihood of data corruption increases as load increases. if a client *doesn't* retransmit, is there any guarantee that a hard-mounted client can make forward progress? > So, for NFSv2,3 I don't know of a stated "rule". I don't think it is covered > in the NFS interoperability RFC that appeared a while back, but can't > remember for sure. we've been looking for a while, but haven't seen anything. --------------090700060104010008020909-- From owner-freebsd-fs@FreeBSD.ORG Sat Oct 15 18:41:44 2005 Return-Path: X-Original-To: fs@freebsd.org Delivered-To: freebsd-fs@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 887CA16A420 for ; Sat, 15 Oct 2005 18:41:44 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from ccshst09.cs.uoguelph.ca (ccshst09.cs.uoguelph.ca [131.104.96.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2CF1D43D4C for ; Sat, 15 Oct 2005 18:41:43 +0000 (GMT) (envelope-from rick@snowhite.cis.uoguelph.ca) Received: from snowhite.cis.uoguelph.ca (snowhite.cis.uoguelph.ca [131.104.48.1]) by ccshst09.cs.uoguelph.ca (8.13.1/8.13.1) with ESMTP id j9FIfcAc023938; Sat, 15 Oct 2005 14:41:38 -0400 Received: (from rick@localhost) by snowhite.cis.uoguelph.ca (8.9.3/8.9.3) id OAA37331; Sat, 15 Oct 2005 14:43:01 -0400 (EDT) Date: Sat, 15 Oct 2005 14:43:01 -0400 (EDT) From: rick@snowhite.cis.uoguelph.ca Message-Id: <200510151843.OAA37331@snowhite.cis.uoguelph.ca> To: fs@freebsd.org X-Scanned-By: MIMEDefang 2.52 on 131.104.96.18 Cc: Subject: FreeBSD NFS server not responding to TCP SYN packets from Linux/SunOS clients X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 15 Oct 2005 18:41:44 -0000 >> When Sun first did NFS over TCP, I believe they did >> do retries (using a conservative timeout). I think I eventually convinced Sun >> that it wasn't a good idea and I think that Solaris no longer >> does them, but I'm not sure. (For this to work correctly, a server is required >> to disconnect whenever it can't generate a reply to an RPC over TCP for any >> reason.) > >yes, this is a difficult semantic. For v3,4 it shouldn't be necessary, except in extreme circumstances, since the server can always just reply NFSERR_DELAY. For v2, I'd be tempted to discourage v2 over TCP, arguing that v2 is just there for old clients that can't do anything else and let them use UDP. In other words, NFSERR_DELAY is your friend:-) > it means that there is now a race that allows a server to redo a > non-idempotent request if the client reconnects on another port and > sends a retransmit of a stuck request. i've seen this in practice, and > for certain applications this will cause data corruption. > > most Linux NFS clients will not reconnect on the same port after the > server disconnects (a bug i recently addressed). for servers with a > duplicate reply cache, this means the client can retransmit > non-idempotent requests and the DRC will not stop the requests from > being reapplied. such servers are dependent on identifying RPC requests > by the tuple of [ XID, source port, client IP ] -- if source port > changes, then the DRC is rendered ineffective. I'd argue that the DRC shouldn't depend on the same port#. (It can even be argued that it shouldn't depend on same client host IP#, since they can change dynamically via dhcp, etc.) I think you'll find a very brief (and crappy) description of what I use for my current DRC on the ftp site (ftp.cis.uoguelph.ca/pub/nfsv4/server-cache.algorithm and some notes in ftp.cis.uoguelph.ca/pub/nfsv4/doc.tar.gz). Basically, it uses XID, plus a checksum of the first N bytes of the request and a few other checks. [good stuff snipped] > if a client *doesn't* retransmit, is there any guarantee that a > hard-mounted client can make forward progress? Probably not. But I don't think it has been a problem, in practice, for FreeBSD? (I suspect that servers only fail to reply to requests when they are "dead in the water".) The BSD server never drops a request in progress. It does MGET()s and MALLOC()s with M_WAITOK. The problem is that most BSDen are pretty well toast by the time this happens. I am thinking that I should change the server to use M_NOWAIT and then return NFSERR_DELAY when it gets a NULL ptr. (For v2, only allow UDP and drop the request.) But I haven't gotten around to coding it. (Lots of cases where NULL ptrs have to be checked for--> lots of work:-) rick