Date: Wed, 15 Sep 2010 11:38:53 -0400 From: Steve Polyack <korvus@comcast.net> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Eric Crist <ecrist@secure-computing.net>, freebsd-fs@freebsd.org, Thomas Johnson <tom@claimlynx.com> Subject: Re: NFS nfs_getpages errors Message-ID: <4C90E88D.9050608@comcast.net> In-Reply-To: <1260697257.960376.1284564539991.JavaMail.root@erie.cs.uoguelph.ca> References: <1260697257.960376.1284564539991.JavaMail.root@erie.cs.uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
On 09/15/10 11:28, Rick Macklem wrote:
>> Hey folks,
>>
>> We've got 4 servers running FreeBSD 8.1-RELEASE which PXE boot with
>> NFS root. On these machines, we run proftpd and apache 2.2. Over the
>> past couple weeks, we've seen a ton of errors as follows:
>>
>> Sep 14 20:28:59 lion-3 proftpd[31761]: 0.0.0.0
>> (folsom-1-red.claimlynx.com[216.17.68.130]) - ProFTPD terminating
>> (signal 11)
>> Sep 14 20:28:59 lion-3 kernel: nfs_getpages: error 1046353552
>> Sep 14 20:28:59 lion-3 kernel: vm_fault: pager read error, pid 31761
>> (proftpd)
>> Sep 14 20:28:59 lion-3 kernel: Sep 14 20:28:59 lion-3 proftpd[31761]:
>> 0.0.0.0 (folsom-1-red.claimlynx.com[216.17.68.130]) - ProFTPD
>> terminating (signal 11)
>> Sep 14 20:28:59 lion-3 kernel: nfs_getpages: error 1046353552
>> Sep 14 20:28:59 lion-3 kernel: vm_fault: pager read error, pid 31761
>> (proftpd)
>> Sep 14 20:28:59 lion-3 kernel: pid 31761 (proftpd), uid 0: exited on
>> signal 11
>>
>> These, in this case, occurred on three of the four machines until
>> midnight after which all three of the machines had proftpd exit on
>> signal 11. The message above was for child processes. At midnight, the
>> logfile rotated, and newsyslog sent singal 1 to the parent process,
>> which I think finally finished it off. The fourth machine remained
>> running and did not display these messages.
>>
>> The number following 'nfs_getpages: error' changes for each cycle and
>> I'm not certain if any of them repeat.
>>
> Well, at a quick glance, those errors seem to be coming from the NFS
> server in a read reply. Also, the error values seem bogus, since they
> should be small positive numbers (1<->70 + a few just above 10000).
We see these errors on some 8.1 clients as well:
nfs_getpages: error 1110586608
nfs_getpages: error 1108948624
vm_fault: pager read error, pid 56216 (php)
nfs_getpages: error 1114969744
vm_fault: pager read error, pid 54770 (php)
nfs_getpages: error 1137006224
vm_fault: pager read error, pid 50578 (php)
They do not show up often, so we haven't spent much time looking into it
(no tcpdumps yet). Our NFS server is a 8-STABLE system backed by ZFS,
so maybe its related to that (again :) ).
Eric, is your NFS server backed by ZFS as well?
The NFS server doesn't seem to be logging any errors, but the ret-failed
count is always increasing:
Server Info:
Getattr Setattr Lookup Readlink Read Write
Create Remove
543523097 14397049 1949982185 6380 17587820 14002952 8980955
8070238
Rename Link Symlink Mkdir Rmdir Readdir
RdirPlus Access
6966495 9 1668 1117125 904969 5567689 22307
184929325
Mknod Fsstat Fsinfo PathConf Commit
0 338500745 57 0 7129262
Server Ret-Failed
29089796
Server Faults
0
Server Cache Stats:
Inprog Idem Non-idem Misses
0 0 0 0
Server Write Gathering:
WriteOps WriteRPC Opsaved
14001235 14002952 1717
> Could you possibly get a packet capture when one of these happens?
> ("tcpdump -s -0 -w xxx host<nfs-server>" would suffice, but you need to
> have it running when the error occurs. If you can reproduce it by
> talking to the proftpd server, so the tcpdump doesn't run for too
> long, that would be best.)
>
> You can look in the tcpdump via wireshark and see what it being returned
> for the Read RPCs at that time. (You can email me the "xxx" packet trace
> as an attachment and I can look at it, if you get that far.)
>
> rick
> ps: Otherwise, I'd go look at your NFS server and see if it's logging
> errors or if there are indications of problems.
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C90E88D.9050608>