Date: Fri, 19 Mar 2010 11:05:21 -0400 From: Steve Polyack <korvus@comcast.net> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-fs@freebsd.org, User Questions <freebsd-questions@freebsd.org>, bseklecki@noc.cfi.pgh.pa.us Subject: Re: FreeBSD NFS client goes into infinite retry loop Message-ID: <4BA392B1.4050107@comcast.net> In-Reply-To: <4BA37AE9.4060806@comcast.net> References: <4BA3613F.4070606@comcast.net> <201003190831.00950.jhb@freebsd.org> <4BA37AE9.4060806@comcast.net>
index | next in thread | previous in thread | raw e-mail
On 03/19/10 09:23, Steve Polyack wrote:
> On 03/19/10 08:31, John Baldwin wrote:
>> On Friday 19 March 2010 7:34:23 am Steve Polyack wrote:
>>> Hi, we use a FreeBSD 8-STABLE (from shortly after release) system as an
>>> NFS server to provide user home directories which get mounted across a
>>> few machines (all 6.3-RELEASE). For the past few weeks we have been
>>> running into problems where one particular client will go into an
>>> infinite loop where it is repeatedly trying to write data which causes
>>> the NFS server to return "reply ok 40 write ERROR: Input/output error
>>> PRE: POST:". This retry loop can cause between 20mbps and 500mbps of
>>> constant traffic on our network, depending on the size of the data
>>> associated with the failed write.
>>>
>>> We spent some time on the issue and determined that something on one of
>>> the clients is deleting a file as it is being written to by another NFS
>>> client. We were able to enable the NFS lockmgr and use lockf(1) to fix
>>> most of these conditions, and the frequency of this problem has dropped
>>> from once a night to once a week. However, it's still a problem and we
>>> can't necessarily force all of our users to "play nice" and use
>>> lockf/flock.
>>>
>>> Has anyone seen this before? No errors are being logged on the NFS
>>> server itself, but the "Server Ret-Failed" counter begins to increase
>>> rapidly whenever a client gets stuck in this infinite retry loop:
>>> Server Ret-Failed
>>> 224768961
>>>
>>> I have a feeling that using NFS in such a matter may simply be prone to
>>> such problems, but what confuses me is why the NFS client system is
>>> infinitely retrying the write operation and causing itself so much
>>> grief.
>> Yes, your feeling is correct. This sort of race is inherent to NFS
>> if you do
>> not use some sort of locking protocol to resolve the race. The infinite
>> retries sound like a client-side issue. Have you been able to try a
>> newer OS
>> version on a client to see if it still causes the same behavior?
>>
> I can't try a newer FBSD version on the client where we are seeing the
> problems, but I can recreate the problem fairly easily. Perhaps I'll
> try it with an 8.0 client. If I remember correctly, one of the
> strange things is that it doesn't seem to hit "critical mass" until a
> few hours after the operation first fails. I may be wrong, but I'll
> double check that when I check vs. 8.0-release.
>
> I forgot to add this in the first post, but these are all TCP NFS v3
> mounts.
>
> Thanks for the response.
Ok, so I'm still able to trigger what appears to be the same retry loop
with an 8.0-RELEASE nfsv3 client (going on 1.5 hours now):
$ cat nfs.sh
client#!/usr/local/bin/bash
for a in {1..15} ; do
sleep 1;
echo "$a$a$";
done
client$ ./nfs.sh >~/output
the on the server while the above is running:
server$ rm ~/output
What happens is that you will see 3-4 of the same write attempts happen
per minute via tcpdump. Our previous logs show that this is how it
starts, and then ~4 hours later it begins to spiral out of control,
throwing out up to 3,000 of the same failed write requests per second.
home |
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4BA392B1.4050107>