From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 2 15:42:59 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C24981065675 for ; Tue, 2 Jun 2009 15:42:59 +0000 (UTC) (envelope-from simon@comsys.ntu-kpi.kiev.ua) Received: from comsys.ntu-kpi.kiev.ua (comsys.ntu-kpi.kiev.ua [77.47.192.42]) by mx1.freebsd.org (Postfix) with ESMTP id A08918FC18 for ; Tue, 2 Jun 2009 15:42:53 +0000 (UTC) (envelope-from simon@comsys.ntu-kpi.kiev.ua) Received: from pm513-1.comsys.ntu-kpi.kiev.ua (pm513-1.comsys.ntu-kpi.kiev.ua [10.18.52.101]) (authenticated bits=0) by comsys.ntu-kpi.kiev.ua (8.13.7/8.13.7) with ESMTP id n52FTw3j075251 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 2 Jun 2009 18:29:58 +0300 (EEST) Received: by pm513-1.comsys.ntu-kpi.kiev.ua (Postfix, from userid 1001) id 5365C1CC21; Tue, 2 Jun 2009 18:21:03 +0300 (EEST) Date: Tue, 2 Jun 2009 18:21:03 +0300 From: Andrey Simonenko To: freebsd-hackers@freebsd.org Message-ID: <20090602152103.GA81350@pm513-1.comsys.ntu-kpi.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.19 (2009-01-05) X-Virus-Scanned: ClamAV version 0.93.1, clamav-milter version 0.93.1 on comsys.ntu-kpi.kiev.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.4 X-Spam-Checker-Version: SpamAssassin 3.1.4 (2006-07-25) on comsys.ntu-kpi.kiev.ua Subject: NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2009 15:43:00 -0000 Hello, Here I want to describe changes that allow to make atomic updates of NFS exports lists, dynamic on-the-fly atomic updates of NFS exports lists and improve security of NFS exports. Solved tasks: ------------- 1. NFS export specifications (spec -- for short) updates are atomic. NFS server's users will not get EACCES for exported file systems or wrong access rights while exports list is reloaded. 2. The mountd utility has an option for testing configuration, now one can check and see real configuration not loading it into nfsserver. 3. The mountd utility does not load incomplete settings to nfsserver, wrong configuration will not allow denied exports. 4. Atomic on-the-fly modifications of NFS export specifications were implemented, it is possible to change exports settings dynamically. 5. If some file system is mounted or unmounted, then sighup signal sending to mountd is not required, this change removes several race conditions. 6. NFS exports related code is a part of nfsserver/ code, NFS export related data can be removed from directories not related to NFS. Which actions are atomic? ------------------------- 1. Loading export settings from exports(5) file into nfsserver is atomic for each exported file system and for all exported file systems. 2. Loading updates into nfsserver is atomic for each exported file system and for all exported file systems. 3. If a file system was mounted, then loading export settings for it into nfssever is atomic for this file system. Which actions are not atomic? ----------------------------- 1. Loading export specifications for WebNFS file system and specifying WebNFS related settings require more than one system call. 2. Since VFS events such as mounting and unmounting are asynchronous, events for all exported and not exported file systems are checked as separate system calls. Since nmount(2) is not used in this implementation and subdirectories exports are not allowed, it is unlikely that these changes will be accepted. If absence of insecure subdirectories exports is not a problem, then it it is possible to support both existent mountd and new API on 7-STABLE (see nfsserver/nfs_srvsubs.c:nfsrv_fhtovp() function patch). The mountd utility was completely rewritten, actually the better name for new utility with new properties would be "nfse". The single source file mountd.c was split into three .c and three .h files: mountd.c (previous code was rewritten and new code was added), mountd.h (new code), mountd_conf.c (new code), mountd_conf.h (new code), mountd_xdr.c (previous code was updated to support new data structure and options), mountd_xdr.h (previous code). The analog of kern/vfs_export.c was written from zero and now it is called nfssever/nfs_export.c. This version of mountd can be used on modified 7.2 system. It was tested (except WebNFS related settings) on amd64 and i386 arch. Support of 8.0 system is possible, it is necessary to modify the nfs_export.c:nfse_check() function (~60 lines) and add new NFSv4 related options, but since there are sys/nfssever/ and sys/fs/nfsserver/ that have functions that call VFS_CHECKEXP and there is activity in NFS development, it is unclear which arguments nfse_check() should accept. It is better to discuss arguments list and semantic of nfse_check() first. I think this is the only one function that does not allow to use these changes on 8.0 system. Available patches have only necessary changes to sys/nfsserver/, sys/kern/, sys/sys/ and sys/conf/. Complete changes require removing all NFS exports related data and code from directories not related to NFS. Also nfs_pub structure should be a part of the new nfsserver/nfs_export.c file. I did not make all these changes (all file systems, all NFS related flags in mount.h, etc.), just to show the main parts of changes. The following text contains more or less detail description of changes (definitely I forgot to mention something here). Major improvements: ------------------- * Now all export spec updates are atomic. mountd uses nfssvc(2) for this (the new nfssvc(NFSSVC_EXPORT) call is used). Now it is safe to reload exports files. * New nfs_export.c file was added to nfsserver/, all API details are located in nfs_export.h. All NFS related flags and structures are part of nfsserver. * Now mountd uses kernel event EVFILT_FS to see mount and umount VFS events. The mount(8) utility should not send sighup signal to mountd any more. New EVENTHANDLERs were declared: vfs_mount_event and vfs_unmount_event. Registered function for these handlers are invoked when a file system is mounted or unmounted respectively. The nfsserver uses these event handlers to synchronizes own data with available file systems. Memory leak was removed when an exported file system is unmounted. Now the nfssever understands covered file systems (file system mounted on mount point of another file system). * The mountd utility has a new option -c, that allows to modify export spec on-the-fly. One can clear, add, update, delete export spec. All updates are atomic. One commands set works like a transaction with changes, it is applied completely or is not applied at all. * Now mountd has the -t switch: parse configuration files or commands and output all settings to stdout. This option allows to check and see real configuration. Incompatible security changes: ------------------------------ * Now subdirectory export is disallowed. Subdirectory export does not improve security, instead it is the right way for misconfiguration (export settings for a subdirectory can be completely unrelated to this subdirectory and does not protect access to another parts of exported file system). The nfsserver exports file systems, not directories, looks like that subdirectory export for NFS is too complex or impossible to implement completely. Anyway there is nullfs. Having read RFCs, documentation for another NFS implementations and thoughts in user groups in Internet I think that this (radical) modification will improve security. This version of mountd has the same logic of export rules as nfsserver has. * Now mountd allows to mount file systems, subdirectories and regular files (if the -r flag is on) in exported file systems by default. The -alldirs option became obsolete. Compatible security changes: ---------------------------- * Ignoring exports files is not safe, since remote users can get wrong access rights. Alternative compatible solution: all exports file must be present, a user can specify directory/ and all regular files from the given directories will be loaded (any directory can be absent). * Now if mountd cannot correctly parse export specification for some file system, then it does not load anything to nfsserver for this file system. Ignoring something in exports file is not safe. * Now security flavors are per address specification settings in nfsserver and mountd. * In rare cases mountd completely ignore settings in exports file, and does not load anything into nfsserver (this can happen if mistake in configuration does not allow to finish file parsing). Updates for mountd: ------------------- * Now everywhere IPv4 and IPv6 addresses are used, since the kernel knows nothing about domain names, netgroups, etc. Now mountdtab file contains only address, MOUNT protocol's procedure EXPORT and DUMP output addresses. This removes problems with reverse name resolving, but sometimes entries are not removed on unmount (depends on used address). * Better output for MOUNT protocol's procedure EXPORT: host is an address, network is an address with prefix. * Now mountdtab is parsed more carefully. * Zone scope index checking was removed for IPv6 addresses, nfsserver does not check zone scope index. * Now mountdtab is saved only when mountd exists, no other program in the base system uses this file. The representation of mountdtab file's content in memory was optimized. * Do not leave PID file if some error occurred and mountd exited. * Allowed to use loopback addresses in the -h option. (I do not like design idea of -h, -p and similar options.) * Corrected incorrect binding when -p option is not used (nobody saw this because this can happen very seldom, but I could reproduce this error). * Wrong implementation of mask creation when prefix length is given as /prefixlength was corrected. Updates for nfsserver: ---------------------- * Previous nfsserver could access released memory returned by VFS_CHECKEXP. New code does have this problem. Updates for exports(5): ----------------------- * Added new option -host to allow to use host names and the same netgroups names at once. * Added new option -rw: read-write access. * Added flag `!' for hosts and networks, this flag means "deny access". * Added new line "options: ...", right now it is used for global -sec, -no_mntproc_dump and -no_mntproc_export options, later it can be used for NFSv4. * Added new option -nospec, that means "this line does not have any address specifications". * Added new option -no_mntproc_dump to disable MOUNT protocol's procedure DUMP. * Added new option -no_mntproc_export to disable MOUNT protocol's procedure EXPORT. * exports(5) says that -o is the only one compatible option. Actually there are others: -root and -r for -maproot, -m for -mask and -n for -network. Now mountd logs a warning message if an obsolete option is used. * Do not allow to use any option between -network and -mask. * Now #-comment can be anywhere in a line. * \xxx octal number can be used in directories names and option's arguments for representing an arbitrary character. * Now it is possible to mix hosts and netgroups with networks: "host1 -network=somenetwork host2". * Now it is possible to change options for particular host/network in one line: "-ro -mapall=user1 host1 -mapall=user2 host2" (host2 will inherit previous option -ro, but will get new -mapall option). Since previously exports(5) says that options must be given before hosts and networks, this change is backward compatible and allows to represent all settings for one file system in one logical line. * Content of exports(5) was simplified and updated. Open questions and tasks: ------------------------- * There must be a global solution to check whether it is possible to unload a KLD module when no process currently is working with its syscalls. * Looks like that first argument for nfssvc(2) is not a set of flags any more (according to STABLE, CURRENT and NFSv4 implementation). May be there is a sense to make it a value, not flags. * WebNFS related data in nfsserver is protected when export settings are set (in vfs_export.c and new nfs_export.c), but when other parts of nfsserver access WebNFS related data no synchronization is performed. * If a file system cannot be exported in NFS, then there must be some flag to indicate this (MNT_NFSEXPORTABLE or something more general, see fs/msdosfs/msdosfs_vfsops.c for example). * Should signals be checked more often in mountd? Right now signals are checked when mountd is waiting for RPC request, or if nfssvc's commands transaction timeout occurred. Previous code has race conditions with signals and does too many things disallowed by SUSv3 in signal handlers. * Should be there any limitation in nfsserver on number of export specifications and number of command transactions? * May be mountd should be renamed to something another, eg. nfse. NFSv4 does not use MNT procedure, but still needs utility for configuring access rights as I understand. Also nfse command name is more obvious for -c commands, eg. "nfse -c 'add /fs -ro'". As well /etc/exports can be renamed to /etc/nfs.exports, /var/db/mountdtab -> /var/db/nfse.mounts. Also such renaming will allow to use mountd and new nfse in 7-STABLE at the same time and mount(8) from 7-STABLE will not send SIGHUP to nfse, since its PID will be saved in the nfse.pid file. * netgroup.5 can be moved to src/lib/libc/gen/ or src/share/man/man5, this documentation is not part of mountd. Examples: --------- 1. Correct file: exports file: options: -no_mntproc_dump /fs -host 1.1.1.1 -ro 2.2.2.2 /fs -network 10.20.30.40 /home -mapall nobody -network 10/8 -mapall operator -network 20.1/8 "mountd -t" output: configure: reading file exports Global options: -no_mntproc_dump Directory /fs Export specifications: -ro -sec sys -maproot=-2:-2 -host 2.2.2.2 -rw -sec sys -maproot=-2:-2 -host 1.1.1.1 -rw -sec sys -maproot=-2:-2 -network 10.0.0.0/8 Directory /home (mount point) Export specifications: -rw -sec sys -mapall 2:5 -network 20.0.0.0/8 -rw -sec sys -mapall 65534:65534 -network 10.0.0.0/8 2. Wrong file: exports: /fs -ro 1.1.1.1 /fs -network 10/8 -host 1.1.1.1 /home -quiet -ro "mountd -t" output: configure: reading file exports parsing error: exports:2: duplicated address specification 1.1.1.1 was found in this line parsing error: exports:2: -host option's argument parsing failed Directory /fs Wrong configuration Directory /home (mount point) File system options: -quiet Export specifications: -ro -sec sys -maproot=-2:-2 (default) 3. Commands testing: mountd -t -c 'add /fs -ro -mapall nobody -host 1.1.1.1 -network !10/8' \ -c 'flush /home' -c 'update /usr -ro -mapall operator' configure: parsing -c commands Directory /fs Commands: -c add -ro -sec sys -mapall 65534:65534 -host 1.1.1.1 -c add -ro -sec sys -mapall 65534:65534 -network !10.0.0.0/8 Directory /home (mount point) Commands: -c flush Directory /usr (mount point) Commands: -c update -ro -sec sys -mapall 2:5 (default) 4. Specifying exports(5) files mountd /etc/exports /etc/local.exports /usr/local/nfs-export/ Files /etc/exports and /etc/local.exports must be present. The nfs-export directory can be absent, if it present or will be present, then all regular files from it are read. Sources ------- http://comsys.ntu-kpi.kiev.ua/~simon/nfse/ MD5 (nfse-20090602.tar.bz2) = 8670b95fcfb7a80433afa4db43143418