From owner-freebsd-net@FreeBSD.ORG Sat Mar 1 08:55:13 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EB60BEF5 for ; Sat, 1 Mar 2014 08:55:13 +0000 (UTC) Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com [209.85.220.47]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BD4E6185D for ; Sat, 1 Mar 2014 08:55:13 +0000 (UTC) Received: by mail-pa0-f47.google.com with SMTP id lj1so1786187pab.6 for ; Sat, 01 Mar 2014 00:55:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=sft76OCpZjVyQ6NO+tqIw13YTYaItBSaASVdz/nxyng=; b=FSmPg+7IuIoP/Gd6+0pUGVnEmzC5ZSmebw5YmoOoGmewOh6rbmYayMO2i6NaEon8VQ CgXmxm+jxp7T0oQnwsALbGNTmhbnT8SGSGZzICZA//AhuK2gUGOTmN2I+UAALaetLqWm foVTMPPQvj95tPBuIIKLP3NKJCbW5Y5ru9q3kdg3PQrEezr13V8szUzau5KcpOmECMB/ ZkBrP6II+pGEXAcSgjOynzrM52UQBLz5LLikLKET4pmJnjRT/e3NdSZf6baEecLt38L2 IZt4aR3Dy/obdunYZ9LTwnTLZKxn2rPVkTVR+EBUADi/uHd46IVbAX3K77gCcczNnvRI MgOg== X-Gm-Message-State: ALoCoQmjOBGj2QiaYAj+v7DpnAfQIimlkW8Nwcdb4b8Dk63djofbx5eShox6lHQplf09UcDwnQ1Q MIME-Version: 1.0 X-Received: by 10.68.130.202 with SMTP id og10mr8471319pbb.133.1393664106988; Sat, 01 Mar 2014 00:55:06 -0800 (PST) Received: by 10.68.111.37 with HTTP; Sat, 1 Mar 2014 00:55:06 -0800 (PST) In-Reply-To: <2FDC6123-5891-4DDA-AC41-FE4B639C0042@hostpoint.ch> References: <1673358278.14528789.1393542781747.JavaMail.root@uoguelph.ca> <2FDC6123-5891-4DDA-AC41-FE4B639C0042@hostpoint.ch> Date: Sat, 1 Mar 2014 09:55:06 +0100 Message-ID: Subject: Re: Network loss From: Johan Kooijman To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-net@freebsd.org, Rick Macklem , Jack Vogel , John Baldwin X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Mar 2014 08:55:14 -0000 I think NFS is part of the issue here. Everybody that seems to have this issue is running NFS. On top of that: the setup I'm having issues with, didn't have any issues for months when we were running istgt instead of NFS. On Fri, Feb 28, 2014 at 6:08 PM, Markus Gebert wrote: > > On 28.02.2014, at 00:13, Rick Macklem wrote: > > > Markus Gebert wrote: > >> > >> On 27.02.2014, at 02:00, Rick Macklem wrote: > >> > >>> John Baldwin wrote: > >>>> On Tuesday, February 25, 2014 2:19:01 am Johan Kooijman wrote: > >>>>> Hi all, > >>>>> > >>>>> I have a weird situation here where I can't get my head around. > >>>>> > >>>>> One FreeBSD 9.2-STABLE ZFS/NFS box, multiple Linux clients. Once > >>>>> in > >>>>> a while > >>>>> the Linux clients loose their NFS connection: > >>>>> > >>>>> Feb 25 06:24:09 hv3 kernel: nfs: server 10.0.24.1 not responding, > >>>>> timed out > >>>>> > >>>>> Not all boxes, just one out of the cluster. The weird part is > >>>>> that > >>>>> when I > >>>>> try to ping a Linux client from the FreeBSD box, I have between > >>>>> 10 > >>>>> and 30% > >>>>> packetloss - all day long, no specific timeframe. If I ping the > >>>>> Linux > >>>>> clients - no loss. If I ping back from the Linux clients to FBSD > >>>>> box - no > >>>>> loss. > >>>>> > >>>>> The errors I get when pinging a Linux client is this one: > >>>>> ping: sendto: File too large > >> > >> We were facing similar problems when upgrading to 9.2 and have stayed > >> with 9.1 on affected systems for now. We've seen this on HP G8 > >> blades with 82599EB controllers: > >> > >> ix0@pci0:4:0:0: class=0x020000 card=0x18d0103c chip=0x10f88086 > >> rev=0x01 hdr=0x00 > >> vendor = 'Intel Corporation' > >> device = '82599EB 10 Gigabit Dual Port Backplane Connection' > >> class = network > >> subclass = ethernet > >> > >> We didn't find a way to trigger the problem reliably. But when it > >> occurs, it usually affects only one interface. Symptoms include: > >> > >> - socket functions return the 'File too large' error mentioned by > >> Johan > >> - socket functions return 'No buffer space' available > >> - heavy to full packet loss on the affected interface > >> - "stuck" TCP connection, i.e. ESTABLISHED TCP connections that > >> should have timed out stick around forever (socket on the other side > >> could have been closed ours ago) > >> - userland programs using the corresponding sockets usually got stuck > >> too (can't find kernel traces right now, but always in network > >> related syscalls) > >> > >> Network is only lightly loaded on the affected systems (usually 5-20 > >> mbit, capped at 200 mbit, per server), and netstat never showed any > >> indication of ressource shortage (like mbufs). > >> > >> What made the problem go away temporariliy was to ifconfig down/up > >> the affected interface. > >> > >> We tested a 9.2 kernel with the 9.1 ixgbe driver, which was not > >> really stable. Also, we tested a few revisions between 9.1 and 9.2 > >> to find out when the problem started. Unfortunately, the ixgbe > >> driver turned out to be mostly unstable on our systems between these > >> releases, worse than on 9.2. The instability was introduced shortly > >> after to 9.1 and fixed only very shortly before 9.2 release. So no > >> luck there. We ended up using 9.1 with backports of 9.2 features we > >> really need. > >> > >> What we can't tell is wether it's the 9.2 kernel or the 9.2 ixgbe > >> driver or a combination of both that causes these problems. > >> Unfortunately we ran out of time (and ideas). > >> > >> > >>>> EFBIG is sometimes used for drivers when a packet takes too many > >>>> scatter/gather entries. Since you mentioned NFS, one thing you > >>>> can > >>>> try is to > >>>> disable TSO on the intertface you are using for NFS to see if that > >>>> "fixes" it. > >>>> > >>> And please email if you try it and let us know if it helps. > >>> > >>> I've think I've figured out how 64K NFS read replies can do this, > >>> but I'll admit "ping" is a mystery? (Doesn't it just send a single > >>> packet that would be in a single mbuf?) > >>> > >>> I think the EFBIG is replied by bus_dmamap_load_mbuf_sg(), but I > >>> don't know if it can happen for an mbuf chain with < 32 entries? > >> > >> We don't use the nfs server on our systems, but they're > >> (new)nfsclients. So I don't think our problem is nfs related, unless > >> the default rsize/wsize for client mounts is not 8K, which I thought > >> it was. Can you confirm this, Rick? > >> > > Well, if you don't specify any mount options, it will be > > min(64K, what-the-server-specifies). > > > > "nfsstat -m" should show you what it actually is using, for 9.2 or > > later. > > Thanks for your answer and the command. I knew there is an new option to > nfsstat, but I couldn't find it in the 9.1 man page, of course ;-). > > > > 8K would be used if you specified "udp". > > I see. We're using tcp, so that would not be relevant then. Guess my look > at the mount vfsop was too quick... > > > > For the client, it would be write requests that could be 64K. > > You could try "wsize=32768,rsize=32768" (it is actually the > > wsize that matters for this case, but you might as well set > > rsize at the same time). With these options specified, you > > know what the maximum value is (it will still be reduced for > > udp or if the server wants it smaller). > > Ok. I checked this on a system that is still running under 9.2 (we had to > downgrade most production systems), and bingo, 64K wsize. So our nfs server > (a NetApp Cluster) seems to prefer 64K. > > This means NFS still is a potential trigger of the problem. Let's pretend > it is, despite I think I already tried disabling TSO. Would this explain > all the symptoms we were seeing? Why would ifconfig down/up help? Do you > have a theory on that? > > > Markus > > > -- Met vriendelijke groeten / With kind regards, Johan Kooijman T +31(0) 6 43 44 45 27 F +31(0) 162 82 00 01 E mail@johankooijman.com