From owner-freebsd-stable@FreeBSD.ORG Mon Jan 20 07:02:55 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0970EA7; Mon, 20 Jan 2014 07:02:55 +0000 (UTC) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.12]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7D3F317C2; Mon, 20 Jan 2014 07:02:55 +0000 (UTC) Received: from th-04.cs.huji.ac.il ([132.65.80.125]) by kabab.cs.huji.ac.il with esmtp id 1W58sg-000Mfc-3u; Mon, 20 Jan 2014 09:02:46 +0200 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: on 9.2-stable nfs/zfs and 10g hang From: Daniel Braniss In-Reply-To: Date: Mon, 20 Jan 2014 09:01:35 +0200 Message-Id: References: <588564685.11730322.1389970076386.JavaMail.root@uoguelph.ca> <2C287272-7B57-4AAD-B22F-6A65D9F8677B@cs.huji.ac.il> To: Adrian Chadd X-Mailer: Apple Mail (2.1827) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: Rick Macklem , FreeBSD stable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jan 2014 07:02:56 -0000 On Jan 18, 2014, at 6:13 PM, Adrian Chadd wrote: > Hi! >=20 > Please try reducing the size down to 32k but leave TSO enabled. >=20 did so, it worked ok, but took longer: with TSO disabled: 14834.61 real 609.29 user 1996.90 = sys with TSO + 32k: 15714.46 real 639.98 user 1828.07 = sys > It's 9.2, so there may be some bugfixes that haven't been backported > from 10 or -HEAD. Would you be able to try a -HEAD snapshot here? >=20 ENOTIME :-). > What's the NFS server and hosts? I saw the core.txt.16 that says > "ix0/ix1" so I can glean the basic chipset family but which NIC in > particular is it? What would people need to try and reproduce it? >=20 The hosts involved are Dell 720/710 the 10G card are Intel=20 ix0@pci0:5:0:0: class=3D0x020000 card=3D0x7a118086 chip=3D0x10fb8086 = rev=3D0x01 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82599EB 10-Gigabit SFI/SFP+ Network Connection' class =3D network subclass =3D ethernet the server is exporting a big ZFS file system, which is served via 2 = raid controllers: mfi1@pci0:65:0:0: class=3D0x010400 card=3D0x1f2d1028 = chip=3D0x005b1000 rev=3D0x05 hdr=3D0x00 vendor =3D 'LSI Logic / Symbios Logic' device =3D 'MegaRAID SAS 2208 [Thunderbolt]' class =3D mass storage subclass =3D RAID mfi2@pci0:66:0:0: class=3D0x010400 card=3D0x1f151028 = chip=3D0x00791000 rev=3D0x05 hdr=3D0x00 vendor =3D 'LSI Logic / Symbios Logic' device =3D 'MegaRAID SAS 2108 [Liberator]' class =3D mass storage subclass =3D RAID - just had the driver card lying around- I will try a divergent client, which has a Broadcom Nic later. Q: is the TSO bug in the NIC/driver or in the kernel or both? cheers danny >=20 > -a >=20 >=20 > On 18 January 2014 03:24, Daniel Braniss wrote: >>=20 >> On Jan 17, 2014, at 4:47 PM, Rick Macklem = wrote: >>=20 >>> Daniel Braniss wrote: >>>> hi all, >>>>=20 >>>> All was going ok till I decided to connect this host via a 10g nic >>>> and very soon it started >>>> to hang. Running multiple make buildworlds from other hosts = connected >>>> via 10g and >>>> using both src and obj on the server via tcp/nfs did ok. but = running >>>> find =85 -exec md5 {} + (the find finds over 6M files) >>>> from another host (at 10g) will hang it very quickly. >>>>=20 >>>> If I wait a while (can=92t be more specific) it sometimes recovers = - >>>> but my users are not very >>>> patient :-) >>>>=20 >>> This suggests that an RPC request/reply gets dropped in a way that = TCP >>> doesn't recover. Eventually (after up to about 15min, I think?) the = TCP >>> connection will be shut down and a new TCP connection started, with = a >>> retry of outstanding RPCs. >>>=20 >>>> I will soon try the same experiment using the old 1G nic, but in = the >>>> meantime, if someone >>>> could shed some light would be very helpful >>>>=20 >>>> I=92m attaching core.txt, but if it doesn=92t make it, it=92s also >>>> available at: >>>> ftp://ftp.cs.huji.ac.il/users/danny/freebsd/core.txt.16 >>>>=20 >>> You might try disabling TSO on the net interface. There are been = issues >>> with TSO for segments around 64K in the past (or use = rsize=3D32768,wsize=3D32768 >>> options on the client mount, to avoid RPCs over about 32K in size). >>>=20 >> BINGO! disabling tso did it. I=92ll try reducing the packet size = later. >> some numbers: >> there where some 7*10^6 files >> doing it locally (the find + md5) took about 3hs, >> via nfs at 1g took 11 hrs. >> at 10g it took 4 hrs. >>=20 >> thanks! >> danny >>=20 >>=20 >>> Beyond that, capturing a packet trace for the case that hangs easily = and >>> looking at what goes on near the end of it in wireshark might give = you >>> a hint about what is going on. >>>=20 >>> rick >>>=20 >>>> thanks, >>>> danny >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to >>>> "freebsd-stable-unsubscribe@freebsd.org" >>>>=20 >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"