From owner-freebsd-hackers@freebsd.org Mon Nov 16 15:15:32 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63D05A305B8 for ; Mon, 16 Nov 2015 15:15:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4547F12CD for ; Mon, 16 Nov 2015 15:15:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 439BAA305B7; Mon, 16 Nov 2015 15:15:32 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 432CAA305B6 for ; Mon, 16 Nov 2015 15:15:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C3E9B12CC for ; Mon, 16 Nov 2015 15:15:31 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id tAGFFDP9078596 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 16 Nov 2015 17:15:13 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua tAGFFDP9078596 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id tAGFFDd0078586; Mon, 16 Nov 2015 17:15:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 16 Nov 2015 17:15:13 +0200 From: Konstantin Belousov To: Rick Macklem Cc: Daniel Braniss , hackers@freebsd.org Subject: Re: kqueue of a nfs mounted file not working Message-ID: <20151116151513.GF5854@kib.kiev.ua> References: <9BC3EFA2-945F-4C86-89F6-778873B58469@cs.huji.ac.il> <20151115152635.GB5854@kib.kiev.ua> <3AEC67FD-2E67-4EF9-9D46-818ABF3D8118@cs.huji.ac.il> <661673285.88370232.1447682409478.JavaMail.zimbra@uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <661673285.88370232.1447682409478.JavaMail.zimbra@uoguelph.ca> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 15:15:32 -0000 On Mon, Nov 16, 2015 at 09:00:09AM -0500, Rick Macklem wrote: > Daniel Braniss wrote: > > > > > On 15 Nov 2015, at 17:26, Konstantin Belousov wrote: > > > > > > On Sun, Nov 15, 2015 at 11:22:55AM +0200, Daniel Braniss wrote: > > >> HI, > > >> I???m writing a program to monitor a file using kqueue(2), if the file is > > >> local > > >> all is OK, but if the file is via a nfs mounted fs, it only works once. > > >> stat shows the file growing, but kevent is not triggered. > > > > > > Does file grow due to local changes on the nfs client, or some other > > > client changes the file, while your client tries to get kevent > > > notifications ? > > > > it gets updated by a host which has the file as local, so yes, it gets > > updated > > by another client/host. > > > Hmm, I am not surprised that this doesn't work. The only indication to the > client that the file has changed on the server is a change in the file's > attributes when they're acquired (via a Getattr RPC or similar) from the server. > > There is a vfs operation called VFS_SYSCTL(). This isn't implemented on > the current NFS client. It was implemented on the old one, but only for > NFS locking events and I didn't understand what needed to be done, so I > didn't do it. You probably mean VOP_KQFILTER, not VFS_SYSCTL(). BTW, the later is only used by nfs and I do not quite see why its functionality not subsumed by the mount options. WRT VOP_KQFILTER, the default implementation is adequate. What is missed for NFS is the knote activation when the code notes that the cached metadata is invalidated by server. > Kostik, do you know if there is a VFS_SYSCTL() call done when the kevent > stuff is probing for a file size change? (Or does it not probe and events > get triggered via the write syscall or ???) I took a quick look at the kevent > stuff, but admit I got lost and couldn't figure out what triggered events > being logged? > > Also, is the event for "file growing" or "file changed"? > If it is the latter, all the NFS client can do is look for a change in > the file's modify time and this is often at a resolution of 1sec., which > implies that a change within the same second as the previous one may not > be noticed. (NFSv4 has a Change attribute that is always guaranteed to > change, but that is only NFSv4.) Also, you see metadata changes as well > as data changes, at least for the NFSv4 attribute. Please look at the sys/kern/vfs_subr.c lines 4296-4419. There is a bunch of the post-VOP hooks which are executed after the filesystem VOP method was executed. You would see the complete list of the notifications which are sent, and the way they are sent, by calling VFS_KNOTE{_LOCKED}(vp, NOTE_XXX). Similar calls should be spread over the nfs client code when the attribute cache entries are deleted, possibly comparing old and new values to select proper notification. But you are of course right that client cannot fully implement the notifications without the server notifying it, so whatever efforts are done for NFSv3, there are definitely will be cases which cannot work. I do not know NFSv4 enough to make similar statement, but I think something would prevent the complete implementation, e.g. for rename. This makes me wonder should we need to add the calls to VFS_KNOTE() into the nfs client, at all. It is similar to lockd/nolockd options, in that the client might be not interested in the server or other clients updates, only in the local notifications. This plus the fact that the feature cannot have complete implementation, raises the question.