From owner-freebsd-stable@FreeBSD.ORG Wed Apr 23 09:52:11 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D66E41065673 for ; Wed, 23 Apr 2008 09:52:11 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.188]) by mx1.freebsd.org (Postfix) with ESMTP id E777B8FC1B for ; Wed, 23 Apr 2008 09:52:10 +0000 (UTC) (envelope-from pluknet@gmail.com) Received: by nf-out-0910.google.com with SMTP id b2so1062158nfb.33 for ; Wed, 23 Apr 2008 02:52:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=Mxxgc4jiI5Cz5UEBYPoVi9Mht0h8gGeKUDTqJWGHoGk=; b=hnjIsFJcw8pN6zvjxg/XxWbPok00HFiobCHWoZKfRdMul27S0Oo6pQUgZoCE+SRtSNnprNg0P9XWtvvBDOzqG8lsmtJpj/ITQIKprPEdaHGSW8bBZWnTX0VW9Wwop1xvSCRQwuehIJGSPFmVxczLF3Lc07O9352X7bi9pX+9alo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ZTx+2jhxoapLdcFOxSh/d14xVqddOptycu/oZ+ULNtEUui+HioG4YtJ6D/f/WJ3s7G2bpsUo3kykVU0OOUPiktVx2yZ10MK7vf44/6H5mLqExmodBxZzk6j9MT8S7R7+GTf0Lm3wvdJgI14+lAq2/ZlPcU06RwOUmUS3lIbSyLE= Received: by 10.78.197.9 with SMTP id u9mr703129huf.56.1208944328486; Wed, 23 Apr 2008 02:52:08 -0700 (PDT) Received: by 10.78.16.10 with HTTP; Wed, 23 Apr 2008 02:52:08 -0700 (PDT) Message-ID: Date: Wed, 23 Apr 2008 13:52:08 +0400 From: pluknet To: pyunyh@gmail.com In-Reply-To: <20080423045347.GE54715@cdnetworks.co.kr> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20080421094718.GY25623@hub.freebsd.org> <200804211537.m3LFbaZA086977@lava.sentex.ca> <200804221501.m3MF1guW092221@lava.sentex.ca> <200804221741.m3MHfYjO092795@lava.sentex.ca> <200804221807.m3MI73bN092981@lava.sentex.ca> <20080423045347.GE54715@cdnetworks.co.kr> Cc: stable@freebsd.org Subject: Re: nfs-server silent data corruption X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Apr 2008 09:52:11 -0000 2008/4/23 Pyun YongHyeon : > > On Wed, Apr 23, 2008 at 12:13:44AM +0400, pluknet wrote: > > On 22/04/2008, Mike Tancsa wrote: > > > At 02:00 PM 4/22/2008, Arno J. Klaassen wrote: > > > > > > > > > > > > > Are you using the latest RELENG_7, or at least the latest version of > > > > > nfe thats in RELENG_7 ? > > > > > > > > > > > > Think so : > > > > > > > > > > OK, and it is the latest RELENG_7 ? Or just the if_nfe.c file has been > > > manually updated ? Also, you are using ULE or the 4BSD scheduler ? I still > > > have 4BSD on the box I am testing on. > > > > Hi, I have the same problem with data corruption (with nfe on nfs server side), > > particularly when transferring large files. > > Maybe this is somehow associated with the topic. > > > > My simple test case: > > truncate -s 1000m bigfile > > ^^ here I get zero-filed file > > cp bigfile /nfs/mounted > > ^^ here I get not-at-all-zero-filed file, after uploading to nfs server > > > > I looked at the corrupted file. It contains a few ranges, filed with > > non-zero bytes: > > equal to zero? real 4-byte value offset > > ====================================== > > not equal 1200355616 at pos=38797316 > > ... <-- this range contains per-4bytes garbage, omit > > not equal 3879749905 at pos=38813696 > > > > not equal 161160732 at pos=45613060 > > ... <-- ditto > > not equal 575257183 at pos=45629440 > > > > not equal 1943682165 at pos=59768836 > > ... <-- ditto > > not equal 2843639625 at pos=59785216 > > > > not equal 2653910121 at pos=60293124 > > ... <-- ditto > > not equal 3462830780 at pos=60309504 > > > > Some info: > > > > nfs server on 8-CURRENT as of Apr 17 > > nfs client on 7.0-STABLE as of Apr 12 > > > > dmesg | grep nfe > > nfe0: port 0xe000-0xe007 mem > > 0xe2001000-0xe2001fff irq 20 at device 4.0 on pci0 > > miibus0: on nfe0 > > nfe0: Ethernet address: 00:04:61:6c:76:b1 > > nfe0: [FILTER] > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > nfe0: tx v1 error 0x6001 > > ^^^ > > I'm not sure it's related with data corruption issue but 0x6001 > would mean Tx underflow error. I recall these Tx errors were seen > on nfe(4) if negotiated speed/duplex does not match with link > partner or MACs. > Does link partner also agree on speed/duplex settings of nfe(4)? One unmanaged 10/100 switch is between them (which are both 100baseTX), so I cannot say exactly :( Though I can achieve speed upto 100mbps. I can test later directly on demand. > What PHY driver nfe(4) use? > $ kldload if_nfe nfe0: port 0xe000-0xe007 mem 0xe2001000-0xe2001fff irq 20 at device 4.0 on pci0 nfe0: Ethernet address: 00:04:61:6c:76:b1 nfe0: [FILTER] miibus0: on nfe0 rlphy0: PHY 1 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto nfe0: link state changed to DOWN nfe0: link state changed to UP So, it seems to be rlphy. > > > This appears while cp'ing file to server. > > (btw they do not appear with disabled polling, probably it's an another issue) > > > > vmstat -i | grep nfe > > irq20: nfe0 ohci0 1 0 > > > > nfe0: flags=8843 metric 0 mtu 1500 > > options=48 > > ether 00:04:61:6c:76:b1 > > inet 192.168.200.137 netmask 0xffffff00 broadcast 192.168.200.255 > > media: Ethernet autoselect (100baseTX ) > > status: active > > I can reproduce it regardless polling presence. > > > > nfe0@pci0:0:4:0: class=0x020000 card=0x10001695 chip=0x006610de > > rev=0xa1 hdr=0x00 > > wbr, pluknet