Date: Mon, 12 Jun 2017 12:28:55 -0700 From: Xin LI <delphij@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: FreeBSD Current <freebsd-current@freebsd.org>, stable@freebsd.org Subject: Re: post ino64: lockd no runs? Message-ID: <CAGMYy3tVZ6CNOfKsTv0uXKV%2BFyUGC0Fw-mN_CHsMEf1FRM-VAg@mail.gmail.com> In-Reply-To: <2474735.4VjKMe5DLv@ralph.baldwin.cx> References: <20170611172022.GA3184@albert.catwhisker.org> <2474735.4VjKMe5DLv@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Jun 12, 2017 at 10:14 AM, John Baldwin <jhb@freebsd.org> wrote:
> On Sunday, June 11, 2017 11:12:25 AM David Wolfskill wrote:
>> On Sun, Jun 04, 2017 at 08:57:44AM -0400, Michael Butler wrote:
>> > It seems that {rpc.}lockd no longer runs after the ino64 changes on any
>> > of my systems after a full rebuild of src and ports. No log entries
>> > offer any insight as to why :-(
>> >
>> > imb
>>
>> I don't tend to use NFS on my systems that are running head, so I
>> haven't had occasion to test this as stated.
>>
>> However, I just completed my weekly update of the "prooduction" systems
>> here at home, running stable/11. And I find that lockd seems to be ...
>> claiming that all is well, but declining to run (for long).
>>
>> To the best of my knowledge, that was not the case until this last
>> update, which was from:
>>
>> FreeBSD albert.catwhisker.org 11.1-PRERELEASE FreeBSD 11.1-PRERELEASE #316 r319566M/319569:1100514: Sun Jun 4 03:54:41 PDT 2017 root@freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT amd64
>>
>> to
>>
>> FreeBSD albert.catwhisker.org 11.1-BETA1 FreeBSD 11.1-BETA1 #322 r319823M/319823:1100514: Sun Jun 11 03:56:10 PDT 2017 root@freebeast.catwhisker.org:/common/S1/obj/usr/src/sys/ALBERT amd64
>>
>> The "glaringly obvious" symptom in my case is that I am now unable
>> to (directly) save an email message from within mutt(1) by appending
>> it to an NFS-resident file. (Saving it to a local file, then using
>> cat(1) to append that to the NFS- resident file & removing the local
>> copy works....)
>>
>> After a few variations on a theme of:
>>
>> albert(11.1)[5] sudo service lockd restart
>> lockd not running?
>> Starting lockd.
>> albert(11.1)[6] echo $?
>> 0
>> albert(11.1)[7] service lockd status
>> lockd is not running.
>>
>> I finally(!) thought to ask ktrace what's going on (as tailing
>> /var/log/messages was completely unproductive, even after enabling
>> rc_debug).
>>
>> So I tried: "sudo ktrace -di service lockd restart"; upon exanimation of
>> the output of kdump(1), I see that the trace ends with:
>>
>> ...
>> 2811 rpc.lockd NAMI "/var/run/logpriv"
>> 2786 sh CALL read(0xa,0x627fc0,0x400)
>> 2786 sh GIO fd 10 read 0 bytes
>> ""
>> 2811 rpc.lockd RET connect 0
>> 2786 sh RET read 0
>> 2811 rpc.lockd CALL sendto(0x3,0x7fffffffe2c0,0x27,0,0,0)
>> 2786 sh CALL exit(0)
>> 2811 rpc.lockd GIO fd 3 wrote 39 bytes
>> "<30>Jun 11 15:43:10 rpc.lockd: Starting"
>> 2811 rpc.lockd RET sendto 39/0x27
>> 2811 rpc.lockd CALL sigaction(SIGALRM,0x7fffffffec20,0)
>> 2811 rpc.lockd RET sigaction 0
>> 2811 rpc.lockd CALL nlm_syscall(0,0x1e,0x4,0x801015040)
>> 2811 rpc.lockd RET nlm_syscall -1 errno 14 Bad address
>
> This is a really good clue. nlm_syscall is dying with EFAULT. The last
> argument is a pointer to an array of char * pointers, and the only way
> I can see it dying is if it fails to copyin() one of the strings pointed
> to by those pointers. You could try running rpc.lockd under gdb from
> ports and setting a breakpoint on 'nlm_syscall' and then printing out
> 'addr_count' and 'p addrs@(addr_count * 2)'.
Yes, I found that the kernel was trying to copyin() from NULL, and
then found that corresponds to 'uaddr'. After some tracing I found
that the tightened condition for taddr2uaddr have enforced (correctly)
buffer length passed from caller, which was not set correctly since ~9
years ago (r177633, which sets the size to sizeof(pointer)) but never
gets noticed because there is no check on that, so the solution seems
to be to correctly set the length values to (allocated size), and that
have fixed the issue for me.
The code could use some cleanups and I plan to do it at some later time.
> Unfortunately I'm not able to reproduce the failure on a test machine
> I have running head post-ino64.
This should have been fixed by r319852 in -HEAD (
https://svnweb.freebsd.org/base?view=revision&revision=319852 ), and
I'll MFC the change after 3 days' settle assuming there is no
objections, as this is a regression.
Cheers,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGMYy3tVZ6CNOfKsTv0uXKV%2BFyUGC0Fw-mN_CHsMEf1FRM-VAg>