From owner-freebsd-current@freebsd.org Tue May 16 04:56:38 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F5D5D6E73D for ; Tue, 16 May 2017 04:56:38 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F331BCD2 for ; Tue, 16 May 2017 04:56:37 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx003 [212.227.17.190]) with ESMTPSA (Nemesis) id 0MBnSJ-1dK92A2bm3-00Ak45 for ; Tue, 16 May 2017 06:56:29 +0200 Date: Tue, 16 May 2017 06:56:23 +0200 From: "O. Hartmann" To: freebsd-current Subject: IFLIB: em0/igb0 broken: No buffer space available/TX(0) desc avail = 1024, pidx = 0 Message-ID: <20170516065623.6e2de036@freyja.zeit4.iv.bundesimmobilien.de> Organization: Walstatt X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:Bj4OyTOIZQIKyQEpSn2Yc4eLCy9eBIH7oBGCoUOvjugt7simTnz xY4TtTFrPMbHwFyms4rcJ0XT0aK0IVg8+gqL1CLOwXsGyMZcfRPIb8YZxo14mIl6greItk0 CVGY8SUoo25yIsNQcrwqRe5M3UJ83sTRlZDWH3GBirgqQVk9DLldXldg5Bqf3s3qUJey1ai AUpGRUoIAbf2nPn7X6piA== X-UI-Out-Filterresults: notjunk:1;V01:K0:ZtgsmVS1v+o=:eWR0XXMlowEa1vkR8dnaMM JFkC6EMWZZtDZrRLdZloaEyUC7K1eOk9MwK4uXRsPcBxr4YNWtmM95xN0fLng9fhknKTmknnV 82QgNwXyqNNwx72v1/2i/2w5aCTZ2N1b4oS5/b0EvsMCcZw67TDoCT96GyhSsTfDTTfJ2clSh qiqDjlf3bxtZEVjBqE8Czf/IjdUb27pTfNY7reqSK9ISQ0BlgwoGK6QHWCGuvp9KXkBr4vKn7 9s1ETs9pgmQTR6UbFjM+4sUwwz0PGZVkVS8slw6SagVIev+tSalzZ/VqkhSvRitqQAeb7IsPF 9P2n9ngO6+iBc66lHqT2CRktkPPIGKi44335/tj9FjdwOH1eJG3fiaZOQ/RRfuEXFFblZPoPZ CP5Rn49f75pfZbE5aahRHC3UAzjbutgR7fA9LhKkfhX3J+UhVZc3ryXIJbPCwRtFGXtW8ccS4 irhzptFm6NC8wdHdv6eeEwTKrT55cNdrk6CdfSSAc/YK/G67kKoXz2nNOc1XY634LK5MntAhE 6kbzdIVc4HaQZ9EfMEHzE8XouZI2eKwysH/+xdPEzpXEA6njgEmvs4dUTLFlxTrE13jMPEI8U 66efNEyxRNgAAI/E3mjgkyPq1RZYfq0R8O3bwY7n6lnbrK9XCLkwhVjVCazc59chIAi3UqS1a dj5OWyS9yzC4PqrTCgVfvM4JnB40vvP1qH01Onb6s3UweRReMYT1Y67SI/DfPMOh+XRNtrTnp dvhMy2hazXAeNaDrqB9UWwBI8MsiCwL6KqitE1DJyc9mqNGC/lhf0+CTob93JR2/FaMgJvtDC hHmBAKK X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 May 2017 04:56:38 -0000 Since the introduction of IFLIB, I have big trouble with especially a certain type of NIC, namely formerly known igb and em. The worst device is an Intel NIC known as i217-LM em0@pci0:0:25:0: class=0x020000 card=0x11ed1734 chip=0x153a8086 rev=0x05 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-LM' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xfb300000, size 131072, enabled bar [14] = type Memory, range 32, base 0xfb339000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf020, size 32, enabled This NIC is widely used by Fujitsu's workstations CELSIUS M740 and the fate would have it, that I have to use one of these. When syncing data over the network from the workstation to an older C2D/bce based server via NFSv4, since introduction of IFLIB the connection to the NFS get stuck and I receive on the console messages like em0: TX(0) desc avail = 1024, pidx = 0 em0: TX(0) desc avail = 42, pidx = 985 Hitting "Ctrl-T" on the terminal doing the sync via "rsync", I see then this message: load: 0.01 cmd: rsync 68868 [nfsaio] 395.68r 4.68u 4.64s 0% 3288k (just for the record) Server and client(s) are on 12-CURRENT: ~ FreeBSD 12.0-CURRENT #38 r318285: Mon May 15 12:27:29 CEST 2017 amd64, customised kernels and "netmap" enabled (just for the record if that matters). In the past, I was able to revive the connection by simply putting the NIC down and then up again and while I had running a ping as a trace indication of the state of the NIC, I got very often ping: sendto: No buffer space available Well, today I checked via dmesg the output to gather again those messages and realised that the dmesg is garbled: [...] nfs nfs servnnfs servefs r server19 2.19162n.fs snerver fs1 s9nfs s2er.nfs server er192.168.0.31:/pool/packages: not responding v er 192.168.0.31ver :/po1ol/packages9: 2.168.0.31:/pool/packagesn: noot responding t <6>n fs serverespondinngf s server 192.168.1rn nfs server 192.168.0.31:/pool/packages: not1 responding 9 2.168.1f7s 0.31:/pool/packagenfs sesrver 19serv2er .168.0.31:/poo: not respolnding / packages: not responding nfs server 19192.168.0.31:/pool/pa2c.k168.0.31:a/gpserver ne1s92.168.0.31:/pool/pac: knot respaof1s68 gs.e17rve8r.2 3192.168.0.31:/pool/packa1:/pool/packages: not responding o goes: nl/packages: not responding o t responding nfs server 192.168.0.31:/poes: ol/packages: nfns server 192.168.0.31:/pool/paot responding c kages: not respondinnfs server n192.1f68.0.31:/pool/packagess: ndi server 192.168.0.31:/pool/packages: not responding [...] Earlier this year after introduction of IFLIB, I checked out servers equipted with Intels very popular i350T2v2 NIC and I had similar problems when dd'ing large files over NFSv4 (ZFS backed) from a client (em0, a client/consumer grade older NIC from 2010, forgot its ID, towards server with i350, but the server side got stuck with the messages seen similar to those reported with the i217-LM). Since my department uses lots of those server grade NICs, I will swap the i217 with a i350T2 and check again. Nevertheless, the situation is very uncomfortable! Kind regards, Oliver