From owner-freebsd-fs@FreeBSD.ORG Tue Jan 8 15:05:11 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AA3155BB for ; Tue, 8 Jan 2013 15:05:11 +0000 (UTC) (envelope-from simon@comsys.ntu-kpi.kiev.ua) Received: from comsys.kpi.ua (comsys.kpi.ua [77.47.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id 65F617C7 for ; Tue, 8 Jan 2013 15:05:11 +0000 (UTC) Received: from pm513-1.comsys.kpi.ua ([10.18.52.101] helo=pm513-1.comsys.ntu-kpi.kiev.ua) by comsys.kpi.ua with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1Tsajl-0006NS-1L; Tue, 08 Jan 2013 17:05:09 +0200 Received: by pm513-1.comsys.ntu-kpi.kiev.ua (Postfix, from userid 1001) id 8600C1E08A; Tue, 8 Jan 2013 17:05:08 +0200 (EET) Date: Tue, 8 Jan 2013 17:05:08 +0200 From: Andrey Simonenko To: Tim Gustafson Subject: Re: Problems Re-Starting mountd Message-ID: <20130108150508.GA2248@pm513-1.comsys.ntu-kpi.kiev.ua> References: <20130103123730.GA19137@pm513-1.comsys.ntu-kpi.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Authenticated-User: simon@comsys.ntu-kpi.kiev.ua X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Exim-Version: 4.63 (build at 28-Apr-2011 07:11:12) X-Date: 2013-01-08 17:05:09 X-Connected-IP: 10.18.52.101:38576 X-Message-Linecount: 88 X-Body-Linecount: 71 X-Message-Size: 3324 X-Body-Size: 2496 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 15:05:11 -0000 On Fri, Jan 04, 2013 at 08:43:32AM -0800, Tim Gustafson wrote: > > Can you give example of two lines for two users (four lines in total). > > /export/home/abc -network=1.2.3.4/22 > /export/home/abc -network=5.6.7.8/23 > > /export/home/def -network=1.2.3.4/22 > /export/home/def -network=5.6.7.8/23 > > > How many file systems are mounted on your system? > > Around 1,400. > > > What are types of these file systems? > > All ZFS. As I understood each /export/home/* pathname from /etc/exports is a mount point for ZFS file system. > > > If NFS export settings on your system have -mapall or -maproot, > > then tell which type of database is used for users and groups names. > > They do not. > > > Give the content of /etc/nsswitch.conf. > > group: files ldap > passwd: files ldap > hosts: files dns > networks: files > shells: files > services: compat > services_compat: nis > protocols: files > rpc: files > I created 2000 file systems on ZFS file system backed by vnode md(4) device. The /etc/exports file contains 4000 entries like your example. On 9.1-STABLE mountd spends ~70 seconds in flushing current NFS exports in the NFS server, parsing data from /etc/exports and loading parsed data into the NFS server. ~70 seconds is not several minutes. Most of time mountd spends in nmount() system call in "zio->io_cv" lock. Can you show the output of "truss -fc -o /tmp/output.txt mountd" (wait wchan "select" state of mountd and terminate it by a signal). If everything is correct you should see N statfs() calls, N+M nmount() calls and something*N lstat() calls, where N is the number of /etc/exports lines, M is the number of mounted file systems. Number of lstat() calls depends on number of components in pathnames. Since truss does not support all needed system calls, I modified it (src/usr.bin/truss/): --- syscalls.c.orig 2012-12-10 13:54:44.000000000 +0200 +++ syscalls.c 2013-01-08 16:19:40.000000000 +0200 @@ -194,6 +194,10 @@ struct syscall syscalls[] = { .args = { { Int, 0 } } }, { .name = "nanosleep", .ret_type = 0, .nargs = 1, .args = { { Timespec, 0 } } }, + { .name = "nmount", .ret_type = 0, .nargs = 3, + .args = { { Ptr, 0 }, { Int, 1 }, { Int, 2 } } }, + { .name = "statfs", .ret_type = 0, .nargs = 2, + .args = { { Name | IN, 0 }, { Ptr, 1 } } }, { .name = "select", .ret_type = 1, .nargs = 5, .args = { { Int, 0 }, { Fd_set, 1 }, { Fd_set, 2 }, { Fd_set, 3 }, { Timeval, 4 } } }, { .name = "poll", .ret_type = 1, .nargs = 3,