From owner-freebsd-stable@freebsd.org Mon Jun 12 18:48:17 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86288C0A12D for ; Mon, 12 Jun 2017 18:48:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 71CEE796A1 for ; Mon, 12 Jun 2017 18:48:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6E0C0C0A12B; Mon, 12 Jun 2017 18:48:17 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69D68C0A12A; Mon, 12 Jun 2017 18:48:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 202E5796A0; Mon, 12 Jun 2017 18:48:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by mail.baldwin.cx (Postfix) with ESMTPSA id 0EAAE10AF09; Mon, 12 Jun 2017 14:47:59 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org, stable@freebsd.org Subject: Re: post ino64: lockd no runs? Date: Mon, 12 Jun 2017 10:14:27 -0700 Message-ID: <2474735.4VjKMe5DLv@ralph.baldwin.cx> User-Agent: KMail/4.14.10 (FreeBSD/11.0-STABLE; KDE/4.14.10; amd64; ; ) In-Reply-To: <20170611172022.GA3184@albert.catwhisker.org> References: <20170611172022.GA3184@albert.catwhisker.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Mon, 12 Jun 2017 14:47:59 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jun 2017 18:48:17 -0000 On Sunday, June 11, 2017 11:12:25 AM David Wolfskill wrote: > On Sun, Jun 04, 2017 at 08:57:44AM -0400, Michael Butler wrote: > > It seems that {rpc.}lockd no longer runs after the ino64 changes on any > > of my systems after a full rebuild of src and ports. No log entries > > offer any insight as to why :-( > > > > imb > > I don't tend to use NFS on my systems that are running head, so I > haven't had occasion to test this as stated. > > However, I just completed my weekly update of the "prooduction" systems > here at home, running stable/11. And I find that lockd seems to be ... > claiming that all is well, but declining to run (for long). > > To the best of my knowledge, that was not the case until this last > update, which was from: > > FreeBSD albert.catwhisker.org 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #316 r319566M/319569:1100514: Sun Jun 4 03:54:41 PDT 2017 root@freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT amd64 > > to > > FreeBSD albert.catwhisker.org 11.1-BETA1 FreeBSD 11.1-BETA1 #322 r319823M/319823:1100514: Sun Jun 11 03:56:10 PDT 2017 root@freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT amd64 > > The "glaringly obvious" symptom in my case is that I am now unable > to (directly) save an email message from within mutt(1) by appending > it to an NFS-resident file. (Saving it to a local file, then using > cat(1) to append that to the NFS- resident file & removing the local > copy works....) > > After a few variations on a theme of: > > albert(11.1)[5] sudo service lockd restart > lockd not running? > Starting lockd. > albert(11.1)[6] echo $? > 0 > albert(11.1)[7] service lockd status > lockd is not running. > > I finally(!) thought to ask ktrace what's going on (as tailing > /var/log/messages was completely unproductive, even after enabling > rc_debug). > > So I tried: "sudo ktrace -di service lockd restart"; upon exanimation of > the output of kdump(1), I see that the trace ends with: > > ... > 2811 rpc.lockd NAMI "/var/run/logpriv" > 2786 sh CALL read(0xa,0x627fc0,0x400) > 2786 sh GIO fd 10 read 0 bytes > "" > 2811 rpc.lockd RET connect 0 > 2786 sh RET read 0 > 2811 rpc.lockd CALL sendto(0x3,0x7fffffffe2c0,0x27,0,0,0) > 2786 sh CALL exit(0) > 2811 rpc.lockd GIO fd 3 wrote 39 bytes > "<30>Jun 11 15:43:10 rpc.lockd: Starting" > 2811 rpc.lockd RET sendto 39/0x27 > 2811 rpc.lockd CALL sigaction(SIGALRM,0x7fffffffec20,0) > 2811 rpc.lockd RET sigaction 0 > 2811 rpc.lockd CALL nlm_syscall(0,0x1e,0x4,0x801015040) > 2811 rpc.lockd RET nlm_syscall -1 errno 14 Bad address This is a really good clue. nlm_syscall is dying with EFAULT. The last argument is a pointer to an array of char * pointers, and the only way I can see it dying is if it fails to copyin() one of the strings pointed to by those pointers. You could try running rpc.lockd under gdb from ports and setting a breakpoint on 'nlm_syscall' and then printing out 'addr_count' and 'p addrs@(addr_count * 2)'. Unfortunately I'm not able to reproduce the failure on a test machine I have running head post-ino64. -- John Baldwin