From owner-freebsd-net@FreeBSD.ORG Thu Aug 1 21:24:07 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 6EA3D889; Thu, 1 Aug 2013 21:24:07 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BE4562B66; Thu, 1 Aug 2013 21:24:06 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id hq12so57074wib.8 for ; Thu, 01 Aug 2013 14:24:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Pyg8Hyw8UYVk48DVfJle02JsQJ3R/wjPSUpuPzIyNyo=; b=KG2gGLDxaEClC8dp1rw0FjWds26bq6q6GlGhBxnAx15DloVYRC8sX68PCpjZV87voc ZLSkNUEgGrBReKO1RVxRRlsOhBH90FzTKj0zzYCRdj7h0PhhmvezhnpSFDnqbYodcQZ8 fsFMR23eGBL7CmPo463hKSJbNDQAYz8DgSdM7wXHluq04ZMq9WNmUVJNmtxUkpCi7MD8 xlRdMXbKZTzzSGQKNnvOiuodTVcFro5rGuWs7q0twT3KjNoKMpmeYi55F9bPWK8Wk78l K0J5SmbR50K/S2iwD88R5Sp/SqmW+1ucZmcBZAS5e5R23R8oZLbbmbebvNXycukpAgXg hPtw== MIME-Version: 1.0 X-Received: by 10.180.211.206 with SMTP id ne14mr11967wic.30.1375392245073; Thu, 01 Aug 2013 14:24:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.94.132 with HTTP; Thu, 1 Aug 2013 14:24:04 -0700 (PDT) In-Reply-To: <201308012010.r71KAHUf024848@svn.freebsd.org> References: <201308012010.r71KAHUf024848@svn.freebsd.org> Date: Thu, 1 Aug 2013 14:24:04 -0700 X-Google-Sender-Auth: WBuccf36JNGJpgs56O5YdaHMpsk Message-ID: Subject: Re: svn commit: r253865 - head/sys/dev/ixgbe From: Adrian Chadd To: Jack F Vogel Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Aug 2013 21:24:07 -0000 Hiya, I've spent a bit of time over the last week helping chase this stuff down. A big thanks to Intel and Verisign for chasing down these bugs and getting a fix into the tree so quickly! If you use ixgbe on -HEAD or -9 I highly suggest you nab these changes and give them a good thrashing. -adrian On 1 August 2013 13:10, Jack F Vogel wrote: > Author: jfv > Date: Thu Aug 1 20:10:16 2013 > New Revision: 253865 > URL: http://svnweb.freebsd.org/changeset/base/253865 > > Log: > A number of important fixes: > - mbuf reused after an RX_COPY optimized operation can sometimes have > a bogus cached address, resulting in TCP hangs. Add critical save points > to the cached address. Thanks to Michael and the team at Verisign for > finding this problem. > - A couple more spots where the rxbuf->flags member should be cleared just > to be sure no incorrect RX_COPY state is left around. Thanks to Adrian > for tracking these down. > - Remove the rearm_queues function from the driver, this was found to be > responsible for some out-of-order packets by Verisign, and was always a > bandaid, with the other fixes in this delta the bandaid can finally be > removed. > - In the other/link interrupt handler the entire state of the EICS register > was being writen back into EICR (which clears causes and thus re-enables > those interrupts), this was wrong, so now mask off the queue portion of > the register value, so we only clear the other/link interrupt we intend. > Marc from Verisign found this. > - Make the SFP+ unsupported option tuneable now, by customer request. > - Finally, just a couple of minor DEBUG string fixes. > > I want to call out and thank all the participants in the 10G community/Intel > calls for helping track down these problems and make the driver better for > everyone! > > MFC after: 3 days, these are critical fixes for 9.2! > > Modified: > head/sys/dev/ixgbe/ixgbe.c > > Modified: head/sys/dev/ixgbe/ixgbe.c > ============================================================================== > --- head/sys/dev/ixgbe/ixgbe.c Thu Aug 1 19:37:11 2013 (r253864) > +++ head/sys/dev/ixgbe/ixgbe.c Thu Aug 1 20:10:16 2013 (r253865) > @@ -45,7 +45,7 @@ int ixgbe_display_debug_stat > /********************************************************************* > * Driver version > *********************************************************************/ > -char ixgbe_driver_version[] = "2.5.13"; > +char ixgbe_driver_version[] = "2.5.15"; > > /********************************************************************* > * PCI Device ID Table > @@ -297,6 +297,7 @@ TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd); > ** doing so you are on your own :) > */ > static int allow_unsupported_sfp = FALSE; > +TUNABLE_INT("hw.ixgbe.unsupported_sfp", &allow_unsupported_sfp); > > /* > ** HW RSC control: > @@ -1071,7 +1072,7 @@ ixgbe_init_locked(struct adapter *adapte > u32 rxdctl, rxctrl; > > mtx_assert(&adapter->core_mtx, MA_OWNED); > - INIT_DEBUGOUT("ixgbe_init: begin"); > + INIT_DEBUGOUT("ixgbe_init_locked: begin"); > hw->adapter_stopped = FALSE; > ixgbe_stop_adapter(hw); > callout_stop(&adapter->timer); > @@ -1382,23 +1383,6 @@ ixgbe_disable_queue(struct adapter *adap > } > } > > -static inline void > -ixgbe_rearm_queues(struct adapter *adapter, u64 queues) > -{ > - u32 mask; > - > - if (adapter->hw.mac.type == ixgbe_mac_82598EB) { > - mask = (IXGBE_EIMS_RTX_QUEUE & queues); > - IXGBE_WRITE_REG(&adapter->hw, IXGBE_EICS, mask); > - } else { > - mask = (queues & 0xFFFFFFFF); > - IXGBE_WRITE_REG(&adapter->hw, IXGBE_EICS_EX(0), mask); > - mask = (queues >> 32); > - IXGBE_WRITE_REG(&adapter->hw, IXGBE_EICS_EX(1), mask); > - } > -} > - > - > static void > ixgbe_handle_que(void *context, int pending) > { > @@ -1506,6 +1490,10 @@ ixgbe_msix_que(void *arg) > bool more; > u32 newitr = 0; > > + /* Protect against spurious interrupts */ > + if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) > + return; > + > ixgbe_disable_queue(adapter, que->msix); > ++que->irqs; > > @@ -1592,6 +1580,8 @@ ixgbe_msix_link(void *arg) > > /* First get the cause */ > reg_eicr = IXGBE_READ_REG(hw, IXGBE_EICS); > + /* Be sure the queue bits are not cleared */ > + reg_eicr = ~IXGBE_EICR_RTX_QUEUE; > /* Clear interrupt with write */ > IXGBE_WRITE_REG(hw, IXGBE_EICR, reg_eicr); > > @@ -2067,7 +2057,6 @@ ixgbe_local_timer(void *arg) > goto watchdog; > > out: > - ixgbe_rearm_queues(adapter, adapter->que_mask); > callout_reset(&adapter->timer, hz, ixgbe_local_timer, adapter); > return; > > @@ -3201,7 +3190,7 @@ ixgbe_free_transmit_buffers(struct tx_ri > struct ixgbe_tx_buf *tx_buffer; > int i; > > - INIT_DEBUGOUT("free_transmit_ring: begin"); > + INIT_DEBUGOUT("ixgbe_free_transmit_ring: begin"); > > if (txr->tx_buffers == NULL) > return; > @@ -4005,11 +3994,13 @@ ixgbe_setup_receive_ring(struct rx_ring > > addr = PNMB(slot + sj, &paddr); > netmap_load_map(rxr->ptag, rxbuf->pmap, addr); > - /* Update descriptor */ > + /* Update descriptor and the cached value */ > rxr->rx_base[j].read.pkt_addr = htole64(paddr); > + rxbuf->addr = htole64(paddr); > continue; > } > #endif /* DEV_NETMAP */ > + rxbuf->flags = 0; > rxbuf->buf = m_getjcl(M_NOWAIT, MT_DATA, > M_PKTHDR, adapter->rx_mbuf_sz); > if (rxbuf->buf == NULL) { > @@ -4026,8 +4017,9 @@ ixgbe_setup_receive_ring(struct rx_ring > goto fail; > bus_dmamap_sync(rxr->ptag, > rxbuf->pmap, BUS_DMASYNC_PREREAD); > - /* Update descriptor */ > + /* Update the descriptor and the cached value */ > rxr->rx_base[j].read.pkt_addr = htole64(seg[0].ds_addr); > + rxbuf->addr = htole64(seg[0].ds_addr); > } > > > @@ -4244,6 +4236,8 @@ ixgbe_free_receive_structures(struct ada > { > struct rx_ring *rxr = adapter->rx_rings; > > + INIT_DEBUGOUT("ixgbe_free_receive_structures: begin"); > + > for (int i = 0; i < adapter->num_queues; i++, rxr++) { > struct lro_ctrl *lro = &rxr->lro; > ixgbe_free_receive_buffers(rxr); > @@ -4268,7 +4262,7 @@ ixgbe_free_receive_buffers(struct rx_rin > struct adapter *adapter = rxr->adapter; > struct ixgbe_rx_buf *rxbuf; > > - INIT_DEBUGOUT("free_receive_structures: begin"); > + INIT_DEBUGOUT("ixgbe_free_receive_buffers: begin"); > > /* Cleanup any existing buffers */ > if (rxr->rx_buffers != NULL) { > @@ -4358,6 +4352,8 @@ ixgbe_rx_discard(struct rx_ring *rxr, in > m_free(rbuf->buf); > rbuf->buf = NULL; > } > + > + rbuf->flags = 0; > > return; > }