From owner-freebsd-current@freebsd.org Tue May 16 19:52:26 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EFB24D7093D for ; Tue, 16 May 2017 19:52:26 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk0-x22c.google.com (mail-qk0-x22c.google.com [IPv6:2607:f8b0:400d:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A88BC145 for ; Tue, 16 May 2017 19:52:26 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk0-x22c.google.com with SMTP id u75so139099884qka.3 for ; Tue, 16 May 2017 12:52:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Dxjmrl1iIxeRQLT3VsuzUu2w0o/0uJcGgAbnJgu+U+E=; b=mqS0zPAOSvxM3UUZzoR0QsCYmd6x1gjK6eT0AzffEz+zBrcDUyFHwXxXbhn5TVYQQc LDC3c/TMgnUrzcoTpXTdd7JgwKvGZ51iMXVE/fwKSGk98uX+S7+QI0RbL7kiADd+KOoJ A7hTPL48wc9YxpOJ64QwsgreiBUUMiYhl39qCGGFMjVclQrvOTbBrXs65ovuvJQM4j8D HNs1U5mslq995GjMOyrqTxTMANSo0i0cPVTg7DotsyYMcY9ph5SQ5fOMxJ8Tv1LESROa 3957fyD0wNsCnqPkppb0EISbSt1yCctIAL6mIGYYm/Gn2yaN+loOxTOkDAMHMUHhR+c5 ocLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=Dxjmrl1iIxeRQLT3VsuzUu2w0o/0uJcGgAbnJgu+U+E=; b=elAlKExIzJzcBMTZWiIcLjfJCvSamFTzsymazqp+lokVpNOGl3SI8AzGbdZzz/zPgh 88G58c4VeWvVtrhON7tmJoVhuove9PtbAYmHMVu/Km5Zg3y9OV4RlLu17ntTzbfQCL/x lK6XYsUxWZS+DdEl94EYhGPlaAL0XcKpIChz2mJFOswSkLWYQuFQd+sg6TtrxFy4gKX7 KuzRdRnZYv23d13/liCDqlR9882eEc9/xQJBaFtgGHR1wAyFWoNDfAPLHsg3Xc6uK6QK BYpBkROiXsBkO8VBL/bolbmoFvrNpMpcU8zM83oZBjq2jM58cX9NFHUdzpf1qxtAmp0q cR3w== X-Gm-Message-State: AODbwcDUAp+hguK5fG7Ns5L0/WN1iBD6L1wDq4KSEm4SHuPeo1Hwsqyl ieXYoX0PsP3v9l/J X-Received: by 10.55.163.20 with SMTP id m20mr13194735qke.265.1494964345728; Tue, 16 May 2017 12:52:25 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-76-104-201-218.hsd1.wa.comcast.net. [76.104.201.218]) by smtp.gmail.com with ESMTPSA id b23sm184990qkb.31.2017.05.16.12.52.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 16 May 2017 12:52:25 -0700 (PDT) Sender: Mark Johnston Date: Tue, 16 May 2017 12:52:20 -0700 From: Mark Johnston To: "O. Hartmann" Cc: freebsd-current Subject: Re: IFLIB: em0/igb0 broken: No buffer space available/TX(0) desc avail = 1024, pidx = 0 Message-ID: <20170516195220.GA38599@wkstn-mjohnston.west.isilon.com> References: <20170516065623.6e2de036@freyja.zeit4.iv.bundesimmobilien.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170516065623.6e2de036@freyja.zeit4.iv.bundesimmobilien.de> User-Agent: Mutt/1.8.2 (2017-04-18) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 May 2017 19:52:27 -0000 On Tue, May 16, 2017 at 06:56:23AM +0200, O. Hartmann wrote: > Since the introduction of IFLIB, I have big trouble with especially a certain > type of NIC, namely formerly known igb and em. > > The worst device is an Intel NIC known as i217-LM > > em0@pci0:0:25:0: class=0x020000 card=0x11ed1734 chip=0x153a8086 rev=0x05 > hdr=0x00 vendor = 'Intel Corporation' > device = 'Ethernet Connection I217-LM' > class = network > subclass = ethernet > bar [10] = type Memory, range 32, base 0xfb300000, size 131072, enabled > bar [14] = type Memory, range 32, base 0xfb339000, size 4096, enabled > bar [18] = type I/O Port, range 32, base 0xf020, size 32, enabled > > This NIC is widely used by Fujitsu's workstations CELSIUS M740 and the fate > would have it, that I have to use one of these. > > When syncing data over the network from the workstation to an older C2D/bce > based server via NFSv4, since introduction of IFLIB the connection to the NFS > get stuck and I receive on the console messages like > > em0: TX(0) desc avail = 1024, pidx = 0 > em0: TX(0) desc avail = 42, pidx = 985 > > Hitting "Ctrl-T" on the terminal doing the sync via "rsync", I see then this > message: > > load: 0.01 cmd: rsync 68868 [nfsaio] 395.68r 4.68u 4.64s 0% 3288k (just for > the record) > > Server and client(s) are on 12-CURRENT: ~ FreeBSD 12.0-CURRENT #38 r318285: Mon > May 15 12:27:29 CEST 2017 amd64, customised kernels and "netmap" enabled (just > for the record if that matters). > > In the past, I was able to revive the connection by simply putting the NIC down > and then up again and while I had running a ping as a trace indication of the > state of the NIC, I got very often > > ping: sendto: No buffer space available > > Well, today I checked via dmesg the output to gather again those messages and > realised that the dmesg is garbled: > > [...] > nfs nfs servnnfs servefs r server19 2.19162n.fs snerver fs1 s9nfs s2er.nfs > server er192.168.0.31:/pool/packages: not responding v > er 192.168.0.31ver :/po1ol/packages9: 2.168.0.31:/pool/packagesn: noot > responding t > <6>n fs serverespondinngf > s > server 192.168.1rn nfs server 192.168.0.31:/pool/packages: not1 responding > 9 > 2.168.1f7s 0.31:/pool/packagenfs sesrver 19serv2er .168.0.31:/poo: not > respolnding / > packages: not responding > nfs server 19192.168.0.31:/pool/pa2c.k168.0.31:a/gpserver > ne1s92.168.0.31:/pool/pac: knot respaof1s68 gs.e17rve8r.2 > 3192.168.0.31:/pool/packa1:/pool/packages: not responding o goes: nl/packages: > not responding o > t responding > nfs server 192.168.0.31:/poes: ol/packages: nfns server > 192.168.0.31:/pool/paot responding c > kages: not respondinnfs server n192.1f68.0.31:/pool/packagess: ndi server > 192.168.0.31:/pool/packages: not responding > [...] > > Earlier this year after introduction of IFLIB, I checked out servers equipted > with Intels very popular i350T2v2 NIC and I had similar problems when dd'ing > large files over NFSv4 (ZFS backed) from a client (em0, a client/consumer grade > older NIC from 2010, forgot its ID, towards server with i350, but the server > side got stuck with the messages seen similar to those reported with the > i217-LM). Since my department uses lots of those server grade NICs, I will swap > the i217 with a i350T2 and check again. > > Nevertheless, the situation is very uncomfortable! I'm able to reproduce this (watchdog timeouts followed by a link bounce, garbled dmesg) on my workstation by dd'ing to a file on an NFSv3 mount of a local file server. This is with: em0@pci0:0:25:0: class=0x020000 card=0x102717aa chip=0x15028086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82579LM Gigabit Network Connection' class = network subclass = ethernet