From owner-freebsd-net@FreeBSD.ORG Tue May 1 16:21:33 2012 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54493106566B; Tue, 1 May 2012 16:21:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 960718FC0A; Tue, 1 May 2012 16:21:32 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q41GLMnn066883; Tue, 1 May 2012 19:21:22 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q41GLLSn048871; Tue, 1 May 2012 19:21:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q41GLLwr048870; Tue, 1 May 2012 19:21:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 1 May 2012 19:21:21 +0300 From: Konstantin Belousov To: net@freebsd.org Message-ID: <20120501162121.GV2358@deviant.kiev.zoral.com.ua> References: <20120407133715.GU2358@deviant.kiev.zoral.com.ua> <20120408051125.GA2358@deviant.kiev.zoral.com.ua> <201204091219.39580.jhb@freebsd.org> <20120412183849.GA2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qBJg7ibC5PRC7M9v" Content-Disposition: inline In-Reply-To: <20120412183849.GA2358@deviant.kiev.zoral.com.ua> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: jfv@freebsd.org, Jack Vogel , John Baldwin Subject: Re: 82574L hangs (with r233708 e1000 driver). X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 May 2012 16:21:33 -0000 --qBJg7ibC5PRC7M9v Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 12, 2012 at 09:38:49PM +0300, Konstantin Belousov wrote: > On Mon, Apr 09, 2012 at 12:19:39PM -0400, John Baldwin wrote: > > On Sunday, April 08, 2012 1:11:25 am Konstantin Belousov wrote: > > > On Sat, Apr 07, 2012 at 04:22:07PM -0700, Jack Vogel wrote: > > > > Make sure you have any firmware up to the latest available, if that= doesn't > > > > help > > > > let me know and I'll check internally to see if there are any outst= anding > > > > issues > > > > in shared code, that will be after the weekend. > > >=20 > > > I had BIOS rev. 151, after you hint I found rev. 154 on the site. > > > Now BIOS reports itself as MTCDT10N.86A.0154.2012.0323.1601, > > > March 23. > > >=20 > > > Unfortunately, upgrade did not changed anything in regard of hanging > > > interface. > >=20 > > Does reverting 233708 make any difference? Have you tried futzing arou= nd with > > kgdb when it is hung to see what state the device is in (software state= at > > least)? > It does, in a sense that without r233708 the interface becomes stuck > almost immediately. I just upgraded to the e1000@r234154, which does not > change much. >=20 > I fiddled with the adapter state after the hang in kgdb more, and I > noted something interesting. Apparently, tx works. When I ping the remote > host from my suffering atom machine, remote host sees the packet. Also > remote machine sees some udp traffic originating from the tom, like > ntp queries. >=20 > And, on receive, the atom board does receive interrupts, em0:rx 0 counter > in vmstat -i increases. Even more fun, the sysctl dev.em.0.debug > shows increasing hw rdh (as I understand, this is hardware 'last > received' packet pointer for rx ring). So I looked at the packet > descriptor at hw rdt index, and there I see > (kgdb) p/x ((struct adapter *)0xffffff80010e4000)->rx_rings->rx_base[78] > $11 =3D {buffer_addr =3D 0x12a128800, length =3D 0x5ea, csum =3D 0x3c2b, = status =3D 0x0,=20 > errors =3D 0x0, special =3D 0x0} >=20 > Apparently, the Descriptor Done bit is clear, so the em_rxeof() function > breaks from the loop, not consuming the current packet. Also, it returns > false due to DD bit clear. This prevents em_msix_rx() from scheduling > taskqueue for processing. So apparent cause for the hang is missing > DD bit in descriptor. >=20 > I am not sure isn't all this is obvious for anybody who knows em > internals, and were to go from there. Ok, nobody cares. Below is the workaround I use to prevent the interface wedging. It seems that the sole PCI register read (namely, the rx ring head read) and consequent recheck of the descriptor status greatly reduce the likelihood of the issue. Unfortunately, the read does not eliminate the hang completely. So it is not some PCIe coherency problem. With the patch applied, I am able to copy around blu-ray images, while previously the interface hang in 20-30 seconds of 100Mbit/s traffic. Sometimes the messages are printed: em0: Workaround: head 1018 tail 1002 cur 1010 em0: Workaround: head 976 tail 973 cur 974 em0: Workaround: head 950 tail 939 cur 946 em0: Workaround: head 435 tail 419 cur 426 Machine is still dead due to random memory corruption which I see, in particular, pmap sometimes read garbage from PTEs. I have no idea is it related to em0 rx descriptor missed writes, or is a different issue. diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index 9c31dad..b2e76cc 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -335,6 +335,11 @@ MODULE_DEPEND(em, ether, 1, 1, 1); =20 static SYSCTL_NODE(_hw, OID_AUTO, em, CTLFLAG_RD, 0, "EM driver parameters= "); =20 +static int em_rx_workaround; +TUNABLE_INT("hw.em.rx_workaround", &em_rx_workaround); +SYSCTL_INT(_hw_em, OID_AUTO, rx_workaround, CTLFLAG_RDTUN, &em_rx_workarou= nd, + 0, "Activate workaround for missed DD in rx ring"); + static int em_tx_int_delay_dflt =3D EM_TICKS_TO_USECS(EM_TIDV); static int em_rx_int_delay_dflt =3D EM_TICKS_TO_USECS(EM_RDTR); TUNABLE_INT("hw.em.tx_int_delay", &em_tx_int_delay_dflt); @@ -4422,14 +4427,60 @@ em_rxeof(struct rx_ring *rxr, int count, int *done) status =3D cur->status; mp =3D sendmp =3D NULL; =20 - if ((status & E1000_RXD_STAT_DD) =3D=3D 0) + if ((status & E1000_RXD_STAT_DD) =3D=3D 0) { + /* + * From PCI/PCI-X Family of Gigabit Ethernet + * Controllers Software Developer's Manual, + * rev. 2.5, p. 306. + * + * Reading the descriptor head to determine + * which buffers are finished is not reliable. + */ + if (em_rx_workaround) { + int head, next; + + head =3D E1000_READ_REG(&adapter->hw, + E1000_RDH(rxr->me)); + next =3D i + 1; + if (next =3D=3D adapter->num_rx_desc) + next =3D 0; + if (next =3D=3D head) + break; + /* + * Re-read the status for the typical + * case of head advanced due to + * received packet. + */ + status =3D cur->status; + if ((status & E1000_RXD_STAT_DD) !=3D 0) + continue; + /* + * Be extra-paranoid and only activate + * the workaround if the next + * descriptor in the rx ring has the + * Descriptor Done bit set, which + * clearly indicates missed write to + * the current descriptor status. + */ + if ((rxr->rx_base[next].status & + E1000_RXD_STAT_DD) =3D=3D 0) + break; + eop =3D 1; /* XXX */ + device_printf(adapter->dev, + "Workaround: head %d tail %d cur %d\n", + head, E1000_READ_REG(&adapter->hw, + E1000_RDT(rxr->me)), i); + goto boom; + } break; + } =20 len =3D le16toh(cur->length); eop =3D (status & E1000_RXD_STAT_EOP) !=3D 0; =20 if ((cur->errors & E1000_RXD_ERR_FRAME_ERR_MASK) || (rxr->discard =3D=3D TRUE)) { +boom: ifp->if_ierrors++; ++rxr->rx_discarded; if (!eop) /* Catch subsequent segs */ --qBJg7ibC5PRC7M9v Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+gDYEACgkQC3+MBN1Mb4hRNgCeL3KZ+GwLgQ6GFG6Cb6Du46bO JJEAmgNkddwterImVF2VjVwmBcqg9cpd =lTt9 -----END PGP SIGNATURE----- --qBJg7ibC5PRC7M9v--